Debunking the myth of the "highly significant" result: effect sizes in gifted education research.

The use of statistical effect sizes in educational research is frequently recommended, especially with the recent emphasis on meta-analysis of research results. The purpose of this article is to describe the utility of effect size reporting and interpretation and to investigate the use of statistical effect sizes in research on giftedness. Research articles published in a five-year span in Journal for the Education of the Gifted, Roeper Review, Gifted Child Quarterly, and a collection of journals not directly associated with gifted education were content analyzed with respect to the reporting of effect sizes. Results suggest that effect sizes generally are not included in research articles, with results consistent across journals.

Educators frequently hear or read that the results of a particular research project were "highly significant" or "approached significance" (Thompson, 1988). In fact, the significance level of a statistical test (the p value) is routinely interpreted as the only important statistic in educational research. Since the p level is directly influenced by the number of subjects in the sample (Snyder & Lawson, 1993; Thompson, 1988, 1989b), a study with many subjects may produce statistically significant results without the existence of a meaningful relationship between independent and dependent variables. Conversely, a study with a small sample may produce a relatively high p value in the presence of a large, meaningful relationship among variables. For example, remarking that a test result is "highly significant" implies that the difference between groups is large, when that may not be the case. Overreliance and misinterpretation of the complexities of statistical significance (Cohen, 1994; Huberty, 1987, 1993; Thompson, 1989a, 1989b) lead researchers into misinterpretation or under-interpretation of their results.

This situation can be conceptualized in non-research terms: Imagine that you are standing at the brink of a deep chasm. Since you need to cross the chasm in order to proceed on your journey, you would like to determine if the chasm is too wide for you to jump across. The fact that the chasm exists (i.e., the p level) is helpful, for without it you might not notice the hole in the ground. However, your primary concern is the size of the chasm (i.e., the effect size). Is it 10 inches wide or 10 feet wide? Either width is significant enough to notice, but attempting to cross one will probably be met with more success than the other. Determining the chasm's existence is important, but doing so does not provide any information about the size of the chasm.

In statistical terms, the calculation and reporting of effect sizes allow both researchers and research consumers to determine the practical significance of statistical tests (Harris, 1991). Effect sizes represent the size of group differences and/or relationships as opposed to the existence of such differences and relationships (i.e., the p level).

The effect size tells us something

very different from the p level. A

result that is statistically significant

is not necessarily practically

significant as judged by the magnitude

of the effect. Consequently,

highly significant p values should

not be interpreted as automatically

reflecting large effects. (Resnow

& Rosenthal, 1989, p. 1276)

The importance of effect size estimation and interpretation is enhanced by the current emphasis upon meta-analysis of educational research (e.g., Bus, Van ljzendoorn, & Pellegrini, 1995; Cameron & Pierce, 1994). Meta-analysis, which involves the combination of results across studies (e.g., Kulik & Kulik, 1982, 1992; Rogers, 1993), is often preferable to the traditional review of literature, which stresses the combination of conclusions across studies. For example, Winne (1979) -- via the traditional review of literature -- concluded that teachers' use of higher order questioning had little effect upon student achievement. In stark contrast, Redfield and Rousseau (1981) used meta-analysis to analyze effect sizes from various studies to investigate the same topic and found that teacher use of higher order questioning had a substantial impact upon student achievement. On average, students whose teachers asked higher order questions could be expected to score at the 77th percentile on an achievement test versus the 50th percentile if not subjected to higher order questions. By using effect sizes to analyze several studies simultaneously, Redfield and Rousseau (1981) convincingly refuted a review of literature with conclusions that could not have otherwise been contradicted.

To facilitate both recta-analysis and the interpretation of results, researchers in the social sciences are encouraged to provide either effect size information or test statistics that allow the calculation of effect sizes (APA, 1994; Asher, 1986; Cohen, 1994; Huberty, 1993; Rosnow & Rosenthal, 1989). The recommendation is also pertinent to research within the field of gifted education. Researchers and consumers of research need to be familiar with the situations in which specific estimates are appropriate, how the estimates can be calculated for common statistical tests, and how effect size estimates can be used during interpretation of statistical analyses.

However, considering that a comprehensive study of effect size usage in gifted education research has yet to be conducted, individuals planning to provide researchers with information about effect size estimates do not have a solid research base with which to focus their efforts. Because books can be (and have been) written on the calculation of effect size estimates (Cohen, 1988; Glass, McGaw, & Smith, 1981; Hedges & Olkin, 1985; Hedges, Shymansky, & Woodworth, 1989; Hunter & Schmidt, 1990; Rosenthal, 1984), detailed instructions for doing so are not included in this article. Rather, the primary purpose of this article is to investigate effect size usage in gifted education research in order to provide a focus for education efforts within gifted education research. In the remainder of this introduction, an attempt is made to familiarize researchers and research consumers with the basic types of effect size estimates and provide several examples of effect size estimates' usefulness when applied to research on giftedness.

Types of Effect Sizes

Researchers use several different terms when they refer to effect size estimates: effect size, percent of variance accounted for, strength of association, measure of association, and magnitude of effect. Some commonly used effect sizes are summarized in Table 1. Several estimates are statistically identical but conceptually are often taught as separate entities. For example, many researchers are quite familiar with correlation coefficients ([r.sup.2]), the variance accounted for in a regression analysis ([R.sup.2]), and the variance accounted for in an analysis of variance ([eta.sup.2]), yet many people do not realize that these estimates provide the exact same information in most cases. Effect size estimates can generally be classified as either measures of association or measures of mean difference(2).

Selected Effect Sizes and Uses

                       Type of
Estimate               Statistic

r                      parametric
[eta.sup.2]            parametric
[R.sup.2]              parametric
[R.sup.2] adjusted     parametric
[omega.sup.2]          parametric
[epsilon.sup.2]        parametric
phi                    nonparametric
contingency            nonparametric
coefficient
Cramer's V             nonparametric
BESD                   parametric
g                      parametric
d                      parametric
canonical              parametric
correlation

                       Nature of
Estimate               Statistic(a)   Bias(b)

r                      MA             B
[eta.sup.2]            MA             B
[R.sup.2]              MA             B
[R.sup.2] adjusted     MA             U
[omega.sup.2]          MA             U
[epsilon..sup.2]       MA             U
phi                    MA             U(c)
contingency            MA             B
coefficient
Cramer's V             MA             U
BESD                   MA             U
g                      MD             B
d                      MD             U
canonical              MA             B
correlation

Estimate               Common Use

r                      correlational studies
[eta.sup.2]            ANOVA, regression, MANOVA
[R.sup.2]              regression
[R.sup.2] adjusted     regression
[omega.sup.2]          ANOVA, regression
[epsilon..sup.2]       ANOVA, regression
phi                    chi square
contingency            chi square
coefficient
Cramer's V             chi square
BESD                   meta-analysis, experimental design
g                      meta-analysis
d                      meta-analysis
canonical              multivariate analyses
correlation

(a) measure of association (MA) or mean difference (MD)

(b) biased (B) or unbiased (U)

(c) unbiased only for 2 X 2 contingency tables 

Table 1

Measures of association involve the relationship between two variables and are usually expressed in terms of correlation or shared variance. Common measures include eta squared ([[Eta].sup.2]), R-squared ([R.sup.2]), adjusted R-squared ([R.sup.2]adj), and omega squared (Snyder & Lawson, 1993). Comparable estimates for use with chi square and other nonparametric tests include the phi coefficient, contingency coefficient, and Cramer's V (Gibbons, 1976, 1993; Gibbons & Chakraborti, 1992; Norusis, 1990; Siegel, 1956). In addition, various correlation coefficients (e.g., the Pearson correlation coefficient, Spearman rho) are occasionally interpreted as effect sizes. These estimates generally range between 0 and 1.

Estimates of mean differences provide an estimate of the difference between two groups on a particular test or measure and are commonly used in meta-analysis (Hedges & Olkin, 1985; Hedges, Shymansky, & Woodworth, 1989; Seifert, 1991) and power analysis (Cohen, 1988). The most well-known estimates of mean difference, which measure group differences in terms of standard deviation units, are g (Glass, McGaw, & Smith, 1981), the standardized mean difference, and d (see Cohen, 1988), the unbiased standardized mean difference. Unlike many other effect size estimates, if the difference between group means is large relative to their standard deviations, the effect size can exceed 1 or even 2. However, general guidelines for interpretation of effect sizes are still applicable.

Characteristics of Effect Sizes

Quantitative researchers recognize eta squared and R2 as representing the "percentage of variance accounted for" by a particular effect. However, these statistics capitalize on sampling error and as a result rarely equal zero in the absence of a relationship between the variables of interest (Huberty, 1994; Pedhazur, 1982; Snyder & Lawson, 1993). Other statistics correct for this bias and, therefore, are unbiased estimators of effect size (see Table 1). The unbiased estimators range from 0 (signifying a complete lack of relationship) to 1 (direct relationship).

Interpretation of effect size estimates is rather arbitrary, although the following guidelines are generally suggested: 0 to .2 representing a small effect, .3 to .4 representing a moderate effect, and .4 or larger signifying the presence of a large effect. Interpretation guidelines vary by discipline. For example, an effect size of .001 in a large medical study may represent the saving of one person's life, even though the effect is very small. However, conceiving of such an ethically and morally serious scenario in the social sciences is difficult. The best guide for interpretation of effect sizes is a review of similar studies to determine the range of effect sizes usually found.

Examples of Effect Size Usage

The following three examples illustrate the utility of effect size use and interpretation. In the first, an example of the recommended use and interpretation of nonparametric effect sizes is provided by Westberg, Archambault, Dobyns, and Salvin (1993), who reported

The obtained chi square statistic

indicates a significant difference;

namely, target gifted and talented

students were provided with less

wait time than target average students.

However, the phi coefficient

indicates that the strength of association

between the wait time and

target student variables is low.

(p. 135)

Without the use of an effect size estimate (i.e., phi), this particular result would have been reported much differently: Observed gifted students are given less wait time when questions are asked of them compared to their nongifted peers. The use of an effect size, however, provides evidence that this difference is predominantly statistical and not practically important.

The second example involves the calculation of effect sizes without proper interpretation. The researchers investigated differences among gifted and non-gifted students' social, emotional, and behavioral adjustment using a longitudinal data set. Effect sizes are reported for all chi square tests and ANOVAS, although the types of effect sizes are not mentioned. The authors report effect sizes ranging from .09 to .27 (with most below .20), indicating that most differences among groups lack practical significance. However, the authors still discuss implications for the reported group differences as though the effect sizes were quite large. The lack of practical importance is not revisited after the effect sizes are presented.

In the third example, researchers found an unexpected result and could have benefited from inclusion of effect size estimates. The researchers report 20 chi square tests of independence and 14 t-tests without effect sizes. Using the reported means and standard deviations, I calculated d as an effect size for the t-tests. The average d was .327 (ranging from .005 to .872), and the average d for the results reported as statistically significant at p [is less than or equal to] .01 was .437 (ranging from .166 to .872). These values could have aided the researchers in their interpretation of results. For example, at one point the authors remark that "surprisingly, [gifted] underachievers earned a higher score on [the] Natural Sciences" subtest of the ACT than gifted high achievers, reporting t(257, 30347) = 2.68, p [is less than or equal to] .01. The effect size for this test is d=.17, indicating that the observed statistical difference has little practical meaning. The "surprising" result actually lacks practical importance and is conceptually similar to the other results reported in the study.

Method

A content analysis of major journals in gifted education was conducted to identify the extent of effect size estimate usage. Analysis of the quarterly journals issues from the summer of 1992 to the most recent issue at the time of analysis: volume 36(3) to 39(1) of Gifted Child Quarterly; 15(4) to 18(2) of Journal for the Education of the Geed; and 14(4) to 17(3) of Roeper Review. Since gifted education research published outside of these major journals may be held to a different standard of quality, a comparison group was created. The 40 most recent research articles listed in the ERIC CD-ROM database with "gifted" in the title and not published in the three gifted education journals listed above were analyzed. In the process of analyzing the non-gifted education journals, a special issue of the Journal of Educational Psychology on the topic of giftedness was located. Given the reputation of this journal as a publisher of high quality educational research, all articles in this issue were also analyzed and added to the non-gifted education journal subsample (i.e., in order to create a consistently high standard to which gifted education journals could be compared). All studies were classified via the dichotomous key included in Appendix A. A list of the articles analyzed for this study is available from the author.

Studies increasingly contain a mix of qualitative and quantitative research, and many quantitative studies include reports of univariate, bivariate, and multivariate statistics. To simplify the content analysis, each research technique was classified as a block. Studies that included only descriptive statistics were coded as descriptive blocks, although reports of research containing both descriptive and inferential statistics were coded according to the level of statistics employed (i.e., univariate or multivariate). For example, if a study contained case studies, ANOVAS, and a discriminant analysis, the study was coded as having one qualitative research block, one univariate block, and one multivariate block. The author used the chi square test to analyze the data with Cramer's V reported as an effect size. Cramer's V was chosen because it is an unbiased estimator with values that range from 0 (no effect) to 1 (maximum possible effect). Values below .2 were interpreted as small, and values in excess of .2 as moderate.

The author analyzed differences in effect size reporting among journals to determine whether trends in effect size usage generalized across research on gifted education, while similar analyses by year and by type of statistical analysis (i.e., univariate vs. multivariate) allowed determination of trends in reporting over time and of use based on complexity of statistical analysis, respectively.

Results

The total number of quantitative research blocks and blocks containing effect size information is reported in Figure 1 for each journal. The lack of effect size information is consistent across journals, and a chi square test of independence produced no evidence of a significant difference among journals ([chi square] [3]=5.01, p=. 17, V=. 18). The percentage of effect size blocks in the gifted journals was similar to that in the nongifted journals ([chi square][1]=.823, p=.367, V=.07).

[Figure 1 ILLUSTRATION OMITTED]

Figure 2 contains a breakdown of Effect size reporting by years. For simplification purposes, the two time periods are 1992-1993 and 1994-1995. Due to the relative infrequency of articles dealing with gifted education in the non-gifted journals, a breakdown by year was not possible. Chi square tests for the entire sample ([chi square] [1]=1.63, p=.20, V=.12) and for Journal for the Education of the Gifted ([chi square] [1]= 1.29, p=.26, V=. 18) and Roeper Review ([chi square] [1]=.004, p=.95, V=.01) provide evidence that there is neither statistically nor practically significant differences between the time periods with respect to effect size reporting. However, chi square test results for Gifted Child Quarterly ([chi square] [1]=2.07, p=. 15, V=.25), with a moderate p level and moderate effect size, suggest that this particular test lacked statistical power. With a larger sample size, a small increase in the percentage of articles published with effect sizes in GCQ may be apparent over time.

[Figure 2 ILLUSTRATION OMITTED]

A third analysis was conducted to determine whether multivariate blocks contained effect size information in roughly the same percentage as univariate blocks (Figure 3). Results indicate that multivariate blocks included effect sizes information more frequently than univariate blocks ([chi square] [1]=20.92, p [is less than] .001, V=.365).

[Figure 3 ILLUSTRATION OMITTED]

This phenomenon could be due to a number of factors. With respect to the classification of research blocks, MANOVAS with multiple ANOVAS as a post hoc test were classified as having effect size information if effect sizes were reported for the ANOVAS. This classification increased the percentage of multivariate effect size blocks, in spite of the fact that the use of ANOVAS to analyze MANOVA results is strongly discouraged (Huberty & Morris, 1989) and that more appropriate multivariate effect sizes are available for MANOVA (e.g., multivariate etasquared, canonical correlation).

Another explanation is that the researchers most likely to use multivariate techniques are also most likely to know how to use, interpret, and report them as recommended in the literature. As a result, multivariate blocks include more statistical sophistication and more effect size information. However, in light of other investigations of the use of multivariate statistics in gifted education research (Plucker, 1995; Pyryt, 1995) and the possible influence of the present study's design, this hypothesis would not appear to be valid.

Discussion

Inclusion of effect size estimates in research articles is generally infrequent, with multivariate effect sizes reported more frequently than for univariate tests. The content analysis yields other findings pertaining to the use of effect sizes. First, most reports of effect sizes do not contain specifications of the type of estimate used. Considering the variety of available estimates and their strengths and weaknesses, this lack of information makes interpretation of results difficult. For example, one study stated that effect sizes ranged from approximately .30 to .60. Without knowing which type of effect size estimate was used (e.g., d, eta-squared, phi), interpreting the effect sizes is impossible.

Second, when researchers report a specific effect size, they usually employ a biased estimator (i.e., one which does not equal zero in the absence of an effect). In fact, not one univariate block in this study included an unbiased estimator, and only one multiple regression included a value for R-squared adjusted. The use of biased estimators was consistent across both gifted and non-gifted journals, as were all other trends observed in this study. These results are similar to those recently reported for research in the psychological sciences (e.g., Kirk, 1996).

The findings of this study hold implications primarily for reviewers of gifted education journals, instructors who are responsible for training future researchers in gifted education, and anyone who reads research about giftedness and gifted education. Instructors should include coverage of effect sizes in their students' quantitative coursework, emphasizing that a wide variety of effect size estimates is available, each estimate with its own strengths and weaknesses. Future researchers should be aware of the range of unbiased estimators that are applicable to the statistical situation in which they are involved and should interpret results accordingly.

Reviewers should make certain that when research results are reported by authors, the type of effect size is reported, the possible range of values for that effect size are described, and effect sizes are used in the interpretation of the statistical results. Doing so will increase both the accuracy, interpretation, and, therefore, the usability of research on giftedness and gifted education. In a related vein, people who read research studies will find the information to be much more valuable if they know how to interpret effect size estimates.

Analysis of effect sizes is often seen as a way to determine if a given piece of research is "bad." This misinterpretation is unfortunate. Rather than be viewed in such a negative light, effect size estimates are useful tools that aid both researchers and consumers of research in the interpretation of results produced by statistical analyses. Rogers (1989) recently noted that

researchers [concerned with gifted

education], except for a select

few .... have not kept current in

their selections of designs to

answer questions or test hypotheses

dealing with 1980's issues in

gifted education (p. 86)

In combination with the research of Rogers (1989), the results of this study suggest that gifted education researchers are moving in the right direction with respect to the sophistication of statistical analyses employed (e.g., more frequent use of multivariate designs), but that considerable effort is still necessary to expose researchers to the advantages of calculating and reporting effect sizes in their work.

Author Notes

This is one possible organization of effect size estimates. For example, Kirk (1996) adds a third category ("other") which includes odds ratios, success rates, and relative risk.

REFERENCES

American Psychological Association. (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.

Asher, W. (1986). Conducting research with meta-analysis: A new direction for gifted education. Gifted Child Quarterly, 30, 7-10.

Bus, A. G., Van Ijzendoom, M. H., & Pellegrini, A. D. (1995). Joint book reading makes for success in learning to read: A meta-analysis on intergenerational transmission of literacy. Review of Educational Research, 65, 1-21.

Cameron, J., & Pierce, W. D. (1994). Reinforcement, reward, and intrinsic motivation: A meta-analysis. Review of Educational Research, 64, 363-423.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Eftbaum.

Cohen, J. (1994). The earth is round (p [is less than] .05). American Psychologist, 49, 997-1003.

Gibbons, J. D. (1976). Nonparametric methods for quantitative analysis. New York: Holt, Rinehart, & Winston.

Gibbons, J. D. (1993). Nonparametric measures of association. Newbury Park, CA: Sage.

Gibbons, J. D., & Chakraborti, S. (1992). Nonparametric statistical inference (3rd. ed.). New York: Marcel Dekker.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Metaanalysis in social research. Beverly Hills, CA: Sage.

Harris, M. J. (1991). Significance tests are not enough: The role of effect-size estimation in theory corroboration. Theory and Psychology, 1, 375-382.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. New York: Academic Press.

Hedges, L. V., Shymansky, J. A., & Woodworth, G. (1989). Modern methods of meta-analysis. Washington, DC: National Science Teachers Association.

Huberty, C. J. (1987). On statistical testing. Educational Researcher, 16(8), 4-9.

Huberty, C. J. (1993). Historical origins of statistical testing practices: The treatment of Fisher versus Neyman-Pearson views in textbooks. Journal of Experimental Education, 61, 317-333.

Huberty, C. J. (1994). A note on interpreting an R2 value. Journal of Educational and Behavioral Statistics, 19, 351-356.

Huberty, C. J., & Morris, J. D. (1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302-308.

Hunter, J, E., & Schmidt, F. L. (1990). Methods of metaanalysis. Newbury Park, CA: Sage.

Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.

Kulik, C.-L. C., & Kulik, J. A. (1982). Effects of ability grouping on secondary school students: A rectaanalysis of evaluation findings. American Educational Research Journal, 19, 415-428.

Kulik, J. A., & Kulik, C.-L. C. (1992). Meta-analytic findings on grouping programs. Gifted Child Quarterly, 36, 73-77.

Norusis, M. J. (1990). SPSS/PC+ 4.0 base manual. Chicago: SPSS Inc.

Pedhazur, E. J. (1982). Multiple regression in behavioral research (2nd ed.). New York: Holt, Rinehart and Winston.

Plucker, J. A. (1995, April). Application of discriminant analysis in research with gifted students. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

Pyryt, M. (1995, April). MANOVA in research with gifted students. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

Redfield, D. L., & Rousseau, E. W. (1981). A recta-analysis of experimental research on teacher questioning behavior. Review of Educational Research, 51, 237-245.

Rogers, K. B. (1989). A content analysis of the literature on giftedness. Journal for the Education of the Gifted, 13, 78-88.

Rogers, K. B. (1993). Grouping the gifted and talented: Questions and answers. Roeper Review, 16, 8-12.

Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage.

Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.

Seifert, T. L. (1991). Determining effect sizes in various experimental designs. Educational and Psychological Measurement, 51, 341-347.

Siegel, S. (1956). Nonparametric statistics for the behavioral sciences. New York: McGraw-Hill.

Snyder, P., & Lawson, S. (1993). Evaluating results using corrected and uncorrected effect size estimates. Journal of Experimental Education, 61, 334-349.

Thompson, B. (1988). A note about significance testing. Measurement and Evaluation in Counseling and Development, 21, 146-148.

Thompson, B. (1989a). Asking "what if" questions about significance tests. Measurement and Evaluation in Counseling and Development, 22, 66-68.

Thompson, B. (1989b). Statistical significance, result importance, and result generalizability: Three noteworthy but somewhat different issues. Measurement and Evaluation in Counseling and Development, 22, 2-5.

Westberg, K. L., Archambault, F. X., Jr., Dobyns, S. M., & Salvin, T. J. (1993). The classroom practices observation study. Journal for the Education of the Gifted, 16, 120-146.

Winne, P. H. (1979). Experiments relating teachers' use of higher cognitive questions to student achievement. Review of Educational Research, 49, 13-50.

Jonathan A. Plucker is an assistant professor of learning, cognition, and instruction at Indiana University.

Manuscript submitted November, 1996. Revision accepted June, 1997.