Two steps forward, one step back: effect size reporting in gifted education research from 1995-2000.

One of the most important recent advances in social science research is the increased emphasis on the calculation, interpretation, and reporting of statistical effect sizes. The current focus on effect sizes is due in part to the growing dissatisfaction with reliance on statistical significance testing as the sole strategy for interpreting quantitative research results.

Indeed, statistical significance testing has recently sparked numerous debates as to its usefulness in research, especially in the social sciences (see Harlow, Mulaik, & Steiger, 1997; Henson & Smith, 2000). Traditionally, research results have been reported in terms of whether or not the results were statistically significant at a prespecified probability level. However, this dichotomous decision of statistically significant versus not statistically significant tells a researcher nothing about the practical significance or importance of a particular finding (Chow, 1988; Falk, 1986; Kirk, 1996; Shaver, 1993). Whereas statistical significance tests provide information as to the existence of group differences or relationships, the magnitude of such differences or relationships is given by effect sizes (Plucker, 1997). As such, effect sizes provide very different information than probability levels (Cohen, 1997; Rosnow & Rosenthal, 1989).

A more promising strategy than a strict focus on significance testing is to help novice researchers understand the relationship among sample size, effect size, and statistical power. With the effect size held constant, an increase in sample size results in increased power. Holding sample size constant, increases in effect size also result in increased power. Holding statistical power constant, sample size and effect size have an inverse relationship--as one increases, the other must decrease to keep the statistical power constant. These three statistics are intimately related, and one cannot be interpreted without knowledge of the other two. Reporting power without consideration of effect size and sample size is pointless, as it provides no basis for interpreting the practical significance of a result. However, this practice still appears to be the norm for a majority of studies in the gifted education literature.

Despite the valuable information that effect sizes provide to researchers, effect sizes have rarely been reported in journal articles. This tendency not to report effect sizes may change, however, as a result of the recent recommendations made by the Task Force on Statistical Inference. The American Psychological Association (APA) formed the Task Force in order to examine current statistical practices, including statistical significance testing. Before the recent recommendations, researchers were merely "encouraged" to report effect sizes when presenting research results (American Psychological Association, 1994). The new recommendations, however, emphasize that effect sizes should always be reported (Wilkinson & Task Force on Statistical Inference, 1999).

A new wrinkle in the reporting of statistical results is the idea of including confidence intervals for effect size estimates (e.g., Fidler & Thompson, 2001; Smithson, 2001; Thompson, 2002). This recommendation is gaining acceptance and should be considered by researchers who include effect size estimates in their work.

Not only are effect sizes infrequently reported, there is rarely enough statistical information provided in research articles to allow the reader to calculate effect sizes. In addition, when effect sizes are provided, they are rarely interpreted as such (Henson & Smith, 2000; Kirk, 1996; Thompson, 1999; Thompson & Snyder, 1998). The new recommendation by the Task Force has resulted in changes in the author guidelines and policies of various journals. With the stronger language of the new recommendations, many journals (e.g., Journal of Counseling & Development, Exceptional Children) now require or strongly suggest that authors report effect sizes, often in addition to presenting results of statistical significance tests. Due to the new recommendations by the APA Task Force and the new journal policies that have appeared over the past few years, we note a need to investigate whether the practice of reporting effect sizes has improved.

Researchers may not report effect sizes for a variety of reasons. Researchers may be unsure as to what an effect size represents and how it differs from statistical significance. A second possibility is that researchers do not know what a "large" effect size is in general or specific to their research. Though general guidelines are available regarding what constitutes small and large effects sizes, the interpretation of an effect size often depends on the area of research (Thompson, 2002); a large effect size in one field (e.g., experimental psychology) may not necessarily be a large effect size in another field (e.g., social psychology; Cooper, 1981). Most probably, the combination of these potential causes results in a situation in which authors have difficulty conceptualizing and interpreting effect sizes and choose not to include them in published research reports.

The purpose of the present article is to investigate effect size reporting in gifted education research. This article elaborates on a similar investigation published by Plucker (1997) in which effect size reporting was examined in published gifted education research. Journals in the field of gifted education do not require authors to report effect sizes. However, in light of the recent focus on effect sizes, the recommendations by the APA Task Force, and the modifications of journal policies and author guidelines by some journals, the authors of the present article seek to examine whether the reporting practices regarding effect sizes have changed in gifted education journals over the past five years. Are we doing better about reporting effect sizes in our research on giftedness and gifted children?

Method

Because the current study serves as an update to the work of Plucker (1997), we used similar methodology to allow direct comparisons between the two sets of data. In the present study, a content analysis was conducted of the quarterly issues of Journal for the Education of the Gifted (JEG). Roeper Review, and Gifted Child Quarterly (GCQ) from 1995 to 2000 in order to identify the extent of effect size usage. Specifically, articles were examined from the following volumes of each journal: volume 28(2) to 34(4) of Journal for the Education of the Gifted, volume 17(3) to 23(2) of Roeper Review, and volume 39(2) to 45(3) of Gifted Child Quarterly. Articles were classified according to the dichotomous decision tree used by Plucker (see Appendix). Essentially, articles were classified as to whether or not they were a research study. Then, all of the articles that were identified as research studies were further classified as to whether they used quantitative analyses, qualitative analyses, or both. Articles that contained quantitative statistics were further examined in regard to whether effect size information was presented.

Research studies increasingly use both qualitative and quantitative research methods, and many quantitative studies employ combinations of univariate, bivariate, and multivariate statistics. In order to be consistent with Plucker's (1997) methodology, studies were classified into blocks based on the type of statistics used. Studies that included only descriptive statistics were coded as descriptive blocks. These blocks, then, did not include articles that used inferential statistics. However, studies that included both descriptive and inferential statistics were coded according to the level of inferential statistic employed (e.g., univariate or multivariate). Therefore, if an article contained a study that used a combination of inferential statistics, each level of statistic reported in the study would be represented by a separate block. Although according to the dichotomous tree, articles containing both univariate and multivariate statistics would be coded as containing "both" types of statistics, for purposes of analysis, the category of "both" was not used. Rather, the article was coded as having a univariate and a multivariate block. As a result, one study could contain multiple blocks. For example, if a study included a case study and percentages of responses, it was coded as having one qualitative research block and one descriptive block. If a study contained an ANOVA and a MANOVA, it was coded as having one univariate block and one multivariate block.

Results

The sample included 723 articles published in the time period of interest. Of the 723 articles, 175 came from JEG, 349 came from Roeper Review, and 199 came from GCQ. These articles were further classified as to the type of statistics and effect size estimates (if any) that were reported. The total number of qualitative and quantitative research blocks by journal is reported in Table 1.

Effect Sizes by Journal and Year

The total number of quantitative research blocks and those that include effect size estimates is reported in Figure 1 for each of the journals. In the current study, about 28.9% of the quantitative research blocks contained effect size estimates. A chi-square test of independence did not reveal any statistically significant differences between the journals on the reporting of effect size estimates ([chi square] [2, N = 280] = 2.199, p = 0.333, V = 0.089). Specifically, the percent of quantitative research blocks that contained effect size estimates were 24.2%, 29.0%, and 34. 1% for JEG, Roeper Review, and GCQ, respectively. Similar results were found by Plucker (1997), such that no differences were found between the journals in terms of effect size reporting. Figure 2 presents the total number of quantitative research blocks by journal by time period. The journals were categorized into three time periods: 1995-1996, 1997-1998, and 1999-2000. Chi-square tests for the entire sample indicated that there were not statistically significant differences in effect size reporting across the three time periods ([chi square] [2, N= 242] = 0.327, p = 0.849, V = 0.037). Additionally, chi-square tests for each individual journal did not reveal any statistically significant differences across time periods: JEG ([chi square] [2, n = 61] = 0.890, p = 0.641, V=0.121), Roeper Review ([chi square] [2, n = 93] = 1.238, p = 0.539, V = 0.115), and GCQ ([chi square] [2, n = 88] = 0.609, p = 0.737, V = 0.083). These results are similar to those found by Plucker (1997).

[FIGURE 1-2 OMITTED]

Univariate Versus Multivariate Blocks

Figure 3 reports the total number of univariate and multivariate research blocks and the effect sizes reported within these blocks by journal. Overall, 17.9% of the univariate research blocks reported effect size estimates. A chisquare test of independence revealed no statistically significant differences between journals on effect size reporting within univariate blocks ([chi square] [2, n = 190] = 5.007, p = 0.082) although the effect size estimate (V = 0.162) suggests the presence of a small, practically significant difference. Specifically, the percent of univariate research blocks by journal that reported effect sizes were 11.3%, 17.9%, and 26.9% for JEG, Roeper Review, and GCQ, respectively. Similar results were found for the multivariate research blocks, such that no significant differences were found between journals on the reporting of effect sizes for multivariate research blocks ([chi square] [2, n = 90] = 1.456, p = 0.483, V = 0.127). Overall, about 52.2% of multivariate research blocks included estimates of effect size, with 57.1%, 57.7%, and 44.4% of multivariate blocks reporting effect sizes in JEG, Roeper Review, and GCQ, respectively.

[FIGURE 3 OMITTED]

Though no differences were found between journals for the reporting of effect sizes for univariate and multivariate research blocks, a third analysis was conducted to determine whether univariate blocks and multivariate blocks contained the same percentage of effect size estimates. A chi-square test for independence indicated that multivariate blocks contained effect size estimates more often than did univariate research blocks ([chi square] [1, N = 280] = 35.003, p < 0.001, V= 0.354). About 52.2% of all multivariate research blocks contained effect size estimates whereas only 17.9% of univariate research blocks reported effect size estimates. Again, Plucker (1997) found similar results.

Interpretation of Effect Size Estimates

When effect size estimates were reported, the most frequent estimates were [R.sup.2]/percent variance explained, Cohen's d, and eta-squared. Although Plucker (1997) did not systematically examine researchers' interpretations of effect size estimates, in this study we recorded authors' interpretations of magnitude for these common effect size estimates. Table 2 contains the range of interpretations for each of the estimates when they were interpreted by the authors of the original studies (roughly half of the effect size estimates reported were not interpreted within their respective articles). Most estimates were identified by the authors as being of a small to medium value, though some were interpreted as large effects. The range of interpretations is considerable, especially for [R.sup.2] and eta-squared, for which evaluations of what constitutes a small or medium effect appear to vary widely. However, the opposite seems to be true for Cohen's d, as interpretations of this effect size estimate tended to be consistent across studies. This consistency of interpretations may be a result of the interpretation guidelines suggested by Cohen (i.e., .2 for small effects, .5 for medium effects, and .8 for large effects when interpreting estimates of standardized mean differences, such as Cohen's d or Hedges' g).

Discussion

Despite the recent focus on the reporting of effect sizes (e.g., the recommendations made by the APA Task Force and requirements by many journals), too few studies include effect size estimates, and fewer still include interpretations of calculated estimates. In fact, the similarity between the results of the present study and those reported by Plucker (1997) suggest that reporting of effect size information has not improved during the past few years.

Although the recommendations of the APA Task Force and new journal requirements represent progress in terms of encouraging authors to report effect sizes, the paucity of articles that actually include effect size estimates in the present study suggests that the field of gifted education has not yet modified its research reporting practices to reflect current recommendations and practice in other fields. However, we found signs of progress: All of the effect size estimates reported in these articles were specifically described by name, an improvement from Plucker's (1997) observations. And several authors attempted to interpret their effect size estimates, which was not evident in Plucker's earlier study.

For univariate research blocks, the most common types of effect size estimates reported were Cohen's d, percent of variance explained, [R.sup.2], and eta-squared. For multivariate research blocks, the most common types of effect size estimates were the percent of variance explained, [R.sup.2], change in [R.sup.2], and eta-squared. However, the majority of effect size estimates reported were biased estimators, with the exception of omega-squared (the latter was reported only in univariate research blocks). This is an improvement (albeit a rather small one) from the situation observed by Plucker (1997), who found that no univariate blocks reported unbiased effect size estimates.

Given considerable evidence that many effect size estimates are biased when used with small sample sizes, researchers should use unbiased estimates whenever possible. These corrected estimates are believed to represent population effect sizes more closely than biased estimators. For example, rather than Cohen's d, researchers could use Hedges' g or related estimates (see Hogarty & Kromrey, 2001). Rather than use eta-squared, researchers could use epsilon-squared or omega-squared (Olejnik & Algina, 2000).

It is tempting to suggest that interpretations of effect size estimates be standardized, allowing researchers, for example, to interpret an [R.sup.2] statistic as "moderate" if it falls into a certain range. However, interpretation of these statistics is best done on a case by case basis. The reliability of one's measures can impact the magnitude of effect sizes, as can the nature of the constructs being studied. Furthermore, although most estimates are scale free (i.e., allowing comparison of effects within a study), the two different classes of estimates have different comparative properties. Measures of association generally have a standard range of acceptable values, 0 to 1, and standardized mean differences do not, requiring considerable contextual interpretation in the analysis of the latter set of estimates. At the same time, standardized mean differences often can be compared to similar estimates from other studies and converted to a meaningful metric (e.g., a standardized mean difference of X represents a difference of Y points on the Iowa Test of Basic Skills), and measures of association usually cannot be compared across studies or converted to more meaningful metrics (Olejnik & Algina, 2000). However, regardless of these complexities, Cohen's (1988) guidelines for interpretation may be a good starting point--or standardized mean differences: .2 small, .5 moderate, .8 large; for eta-squared and related measures, .01 small, .06 moderate, .14 large.

At the least, researchers should report the basic statistical information that allows effect sizes to be calculated. Of course, in order to understand what information is needed, the researcher will need to know how to calculate effect sizes. Gifted education journals also should adopt strict policies about effect size reporting. As Thompson (2001) notes, mandating the inclusion of this information will benefit educational research as the quality of research reporting improves. Indeed, with the popularity of meta-analysis in education research, failure to include effect size estimates or related information may prevent studies from being included in meta-analyses, limiting their long-term impact.

Given the interrelatedness of sample size, statistical power, and effect size, researchers should consider the failure to report effect size estimates in the same light as failure to report p-values or sample size for statistical tests. The field would never allow the publication of research that did not include probability levels, yet it allows effect sizes not to be reported when, in many ways, effect sizes are more important pieces of information.

Table 1
Number of Blocks by Study Type,
Type of Statistics Used by Journal

                                           Journal

Study Type/Type of Statistics   JEG (a)   Roeper (b)   GCQ (c)

Qualitative                       40          79         41
Quantitative (d)
  Descriptives (e)                10          31         44
  Univariate blocks               36          67         52
  Multivariate blocks             23          26         36

(a) Includes 27 articles that contained qualitative analyses only, 37
articles that contained quantitative analyses only, and 37 articles
that contained both quantitative and qualitative analyses.

(b) Includes 43 articles that contained qualitative analyses only, 36
articles that contained quantitative analyses only, and 64 articles
that contained both quantitative and qualitative analyses.

(c) Includes 21 articles that contained qualitative analyses only, 19
articles that contained quantitative analyses only, and 48 articles
that contained both quantitative and qualitative analyses.

(d) A total of 50, 100, and 88 articles in JEG, Roeper Review, and GCQ,
respectively, included quantitative analyses.

(e) Includes "descriptive only" blocks, but does not include univariate
and multivariate blocks that included descriptive statistics in
addition to the respective inferential statistics.

Table 2
Effect Size Types and Magnitudes
as Identified by the Authors

Effect Size Estimate (a)   Range of Values   Magnitude of Estimate

[R.sup.2]                  0.008-0.10        Small
                           0.13-0.18         Medium
                           0.49              Moderately High

Eta-squared                0.016-0.022       Small
                           0.03-0.05         Small
                           0.126             Medium

Cohen's d                  0.01-0.49b        Small
                           0.23-0.48         Small to Medium
                           0.34-0.37         Approaching Medium
                           0.42-0.56         Moderate
                           0.52-0.76         Moderate
                           0.81-0.96         Large
                           0.99              High
                           1.01              Strong

(a) There are two main types of effect size estimate: difference
between means and measures of association. [R.sup.2] and eta-squared
are measures of association, and both are indicators of the amount of
variance explained, with the former being used in regression analyses
while the latter is primarily used in ANOVA designs. Cohen's d is a
standardized indicator of mean difference (Fern & Monroe, 1996).

(b) Though the author indicated that 0.49 "falls just short of
indicating a medium effect size" (Stumpf, 1998, p. 165).

REFERENCES

American Psychological Association (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.

Chow, S. L. (1988). Significance test or effect size? Psychological Bulletin, 103, 105-110.

Cohen. J. (1988). Statistical power analysis of the behavioral sciences (2nd ed.). New York: Academic Press.

Cohen, J. (1997). The earth is round (p<.05). In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no significance tests? (pp. 21-35). Mahwah. NJ: Lawrence Erlbaum.

Cooper, H. M. (1981). On the significance of effects and the effects of significance. Journal of Personality and Social Psychology, 41, 1013-1018.

Falk. R. (1986). Misconceptions of statistical significance. Journal of Structural Learning, 9, 83-96.

Fern, E. F., & Monroe, K. B. (1996). Effect-size estimates: issues and problems in interpretation. Journal of Consumer Research, 23. 89-105.

Fidler, F., & Thompson, B. (2001). Computing correct confidence intervals for ANOVA fixed-and random-effects effect sizes. Educational and Psychological Measurement, 61,575-604.

Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Mahwah, NJ: Lawrence Erlbaum.

Henson, R. K., & Smith, A. D. (2000). State of the art in statistical significance and effect size reporting: A review of the APA Task Force report and current trends. Journal of Research and Development in Education, 33, 285-296.

Hogarty, K. Y., & Kromrey, J. D. (2001, April). We're been reporting some effect sizes: Can you guess what they mean? Paper presented at the annual meeting of the American Educational Research Association. Seattle, WA.

Kirk, R. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.

Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241-286.

Plucker, J. A. (1997). Debunking the myth of the "highly significant" result: Effect sizes in gifted education research. Roeper Review, 20. 122-126.

Rosnow, R. L. & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.

Shaver. J. P. (1993). What statistical significance testing is, and what it is not. Journal of Experimental Education, 61,293-316.

Smithson, M. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement. 61. 605-632.

Stumpf. H. (1998). Gender-related differences in academically talented students' scores and use of time on tests of spatial ability. Gifted Child Quarterly. 42, 157-171.

Thompson, B. (1999). Improving research clarity and usefulness with effect size indices as supplements to statistical significance tests. Exceptional Children, 65. 329-337.

Thompson, B. (2001). Significance. effect sizes. stepwise methods, and other issues: Strong arguments move the field. Journal of Experimental Education, 70, 80-93.

Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 25-32.

Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent Journal of Counseling & Development research articles. Journal of Counseling & Development, 76, 436-441.

Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist. 54, 594-604.

Kelli M. Paul is a doctoral student at Indiana University and is studying educational psychology with an emphasis on inquiry methodology. She is a research assistant at the Center for Evaluation and Educational Policy. E-mail: [e-mail]

Jonathan A. Plucker is an associate professor of educational psychology and cognitive science at Indiana University, where he also directs the Center for Evaluation and Educational Policy. He conducts research on learning, creativity and intelligence, talent development, and education policy. E-mail: [e-mail]