Printer Friendly

Honorton the meta-analyst.

For anyone who has followed the history of parapsychology, there can be no doubt that the use of meta-analysis is an important part of that history. One of the principal arguments of the critics concerned the apparent lack of repeatable experiments. Meta-analytic techniques, along with attention to power and sample sizes, have revealed that the problem did not lie with the data; the problem was that repeatable was absurdly defined to mean that every experiment, at least a large percentage of them, must reach "statistical significance."

Even the early meta-analyses used in parapsychology relied on this "vote-counting" method of determining whether replication was evident. As more sophisticated uses of meta-analysis were introduced, however, it became clear that these techniques provided for a different definition of the term repeatable, namely, that effect sizes remain homogeneous from one similar experiment to another.

Meta-analytic techniques brought other welcome additions to investigating parapsychological data. First, they allowed for quantitative testing of the alleged problem that the more flawed experiments were the ones most likely to produce statistical significance. Second, they permitted the comparison of conditions both within and across experiments. Third, they provided a means by which power calculations could become an important part of designing a new experiment.

Certainly there were quantitative reviews of broad areas of parapsychology before the introduction of meta-analysis (see Schechter, 1987, for some references); but the widespread acceptance of meta-analysis as a way to consider controversial claims in medicine and the social sciences has made it ideal for examining claims in parapsychology as well.

Charles Honorton was one of the first to recognize the potential value of meta-analysis for parapsychologists. He continued to advocate the reporting of effect sizes and other information that would simplify the work of future meta-analysts. He was the first to publish a nominal meta-analysis in parapsychology (Honorton, 1985a), and he went on to publish several more. His sophistication in the use of meta-analytic techniques grew steadily over the years, and the way he wove them together with his other work provides valuable lessons for future researchers. We now turn to the history and magnitude of Chuck's contributions to the interface of parapsychology and meta-analysis.


In the second half of 1982, ten years before his death, two series of events converged in a way that would change the rest of Chuck's research career and the history of parapsychology itself. The first of these was that he and psychologist Ray Hyman had exchanged the first round of their now-famous debate (resulting in: Honorton, 1985a; Hyman, 1985a; and Hyman & Honorton, 1986) at the annual meeting of the Parapsychological Association, held in Cambridge, England, in August. Much of the debate centered around the question of whether any conclusions could be drawn from the existing body of ganzfeld studies. Hyman's approach to the database centered on identifying methodological flaws that he believed negated any statistically significant findings.

The second series of events started when Chuck and colleague Ephraim Schechter read a review by Karlis Osis (1982) describing a newly published book on meta-analysis by Glass, McGaw, and Smith (1981). Upon reading the book; they immediately realized that this new field of meta-analysis had relevance to parapsychology. In light of the status of the debate with Hyman, you can imagine Chuck's reaction when he read the following passage, which alone is marked in the margin of his copy of the book:

Respect for parsimony and good sense demands an acceptance of the

notion that imperfect studies can converge on a true conclusion. An important

part of every meta-analysis with which we have been associated

has been the recording of methodological weaknesses in the original

studies and the examination of their covariance on study findings. Thus,

the influence of "study quality" on findings has been regarded as an

empirical a posteriori question, not an a priori matter of opinion or

judgment used in excluding large numbers of studies from consideration.

(p. 222)

The emerging importance of meta-analysis in Chuck's research and thinking is evident from the progression of space devoted to its description in the annual reports of the Psychophysical Research Laboratories (PRL). By the time the 1982 report was written, crude steps had already been taken toward the meta-analysis that Honorton eventually published (1985a). The 1982 annual report spent one paragraph explaining what meta-analysis does and included descriptions of that meta-analysis, as well as the one eventually published by Schechter (1984) on hypnosis. The focus of both of these analyses as "vote-counting," that is, comparing significant with nonsigficant studies.

In contrast, the 1983 annual report of PRL devoted 10 pages to a general description of meta-analysis. These were followed by an additional 15 pages presenting more comprehensive results for the two areas of studies discussed in the 1982 report (ganzfeld and hypnosis). It is in this report that one begins to see Chuck's commitment to this emerging field, as we read: "Our feeling that the parapsychological research enterprise can benefit from meta-analysis is sufficiently strong that we are actively promoting the research" (p. 97).

Indeed, he did promote the research. On November 17, 1984, PRL hosted a one-day conference on meta-analysis, featuring Harvard psychologist Robert Rosenthal, one of the early proponents of meta-analysis, as the guest speaker. The conference was attended by five members of the PRL staff, as well as 12 other researchers, and is briefly summarized in the 1984 PRL annual report (section 5.0, pp. 96-97). In addition, PRL staff member Ephraim Schechter (1985b) organized a roundtable discussion at the 1984 Annual Convention of the Parapsychological Association, entitled "Meta-Analysis: Integrating Results Across Studies." Joining Schechter (1985b) and Honorton (1985b) as participants were Ray Hyman (1985b), John Palmer (1985), and Irvin Child (1985). The roundtable discussion was prefaced by the participants' prepared remarks, some of which will be examined in the next section of this paper.

During the next eight years, Chuck's sophistication with the use of meta-analytic techniques grew steadily. He collaborated with others to produce a few meta-analytic reviews (Honorton & Ferrari, 1989; Honorton et al., 1990; Honorton, Ferrari, & Bem, 1992), and his research reports all included effect sizes as well as sufficient details to be useful to future meta-analysts. He encouraged others to do the same. Let us now turn to a more in-depth study of this growing sophistication.

From Vote-Counter to Statistical Sophisticate

Chuck's use of meta-analysis in his research progressed through four stages, each identified by the use to which he put the analysis methods. He started with simple vote-counting, then progressed to the idea that meta-analysis could be used to identify flaws and correlate them with study outcome, thus helping to design better experiments for the future. The next stage was the realization that effect sizes could be estimated and used to plan future experiments by calculating power. In the final stage (e.g., Honorton et al., 1990), Chuck was performing a multifaceted analysis in which effect sizes were measured, tested for homogeneity, and compared across conditions to suggest more completely how to design future experiments for optimal results.

Chuck was actually performing meta-analysis in the form of vote-counting before the term had even been introduced (Honorton, 1977), and his first round of analysis of the ganzfeld data used this same technique. Vote-counting is the simple procedure of counting how many of a given group of studies achieve statistical significance at some prespecified level, usually .05. It has been shown that this procedure is actually more and more likely to lead one to the wrong conclusion as the number of studies increases (Hedges & Olkin, 1985). As late as 1989, in the meta-analysis with Ferrari (Honorton & Ferrari, 1989) vote-counting was still included as one of several measures examined, but the credence given to the procedure was so reduced over the years that it does not appear at all in the 1990 paper by Honorton et al.

The next stage incorporated the idea that meta-analysis could be used to identify and summarize flaws in previous studies so that information could be gained for designing better studies. As mentioned above, Chuck was first made aware of meta-analysis when he was in the middle of the debate with Hyman over the ganzfeld database. Because the focus of the debate was study quality and its relation to study outcome, it is not surprising that both Chuck and Ray Hyman picked up on that aspect of meta-analysis as being relevant to parapsychology. The degree to which their thinking was similar, and to which it presaged their joint communique (Hyman & Honorton, 1986) can be seen from their separate remarks at the 1984 roundtable on meta-analysis (Schechter, 1985b). They were both arguing that meta-analysis could be used to design better studies, which could then be used to confirm or reject the existence of the phenomena:

As applied to the search for psi, for example, no amount of meta-analysis,

no matter how well-conceived, can serve as proof for the existence

or nonexistence of psi or for patterns of correlation with psi. Like any

other output of exploratory search, the findings, at best, can serve to

guide the design and conduct of experiments that will carry confirmatory

weight. (Hyman, 1985b, p. 115)

By coding study procedures, meta-analysis helps define and clarify important

methodological issues.... Once these initial steps have been

taken, researchers and critics may be able to agree upon explicit criteria

for the design and evaluation of new sets of studies which will enable

sound conclusions to be drawn. The development of such criteria is, I

believe, a necessary prerequisite for resolution of the controversy over

the occurrence of psi phenomena. (Honorton, 1985b, p. 116)

The major change that occurred in the quantitative assessment of groups of studies as a result of the formal introduction of meta-analysis into parapsychology was the change in focus from vote-counting to measuring effect sizes. Unlike a significance level, the magnitude of an effect size does not depend on the sample size. Consequently, it is far better to define replication in terms of similar effect sizes than in terms of the number of "statistically significant" experiments.

The shift in Chuck's thinking along these lines can be seen by comparing the ganzfeld meta-analyses presented in the 1982 and 1983 PRL annual reports. In the 1982 report we read: "Meta-analysis techniques were used to compare statistically significant and nonsignificant studies to assess the effects of potential artifacts" (section III, p. 2). By 1983, however, the PRL annual report addressed the problems with vote-counting:

Last year, we presented a meta-analytic review of psi ganzfeld research

using the "vote counting" method.... The vote counting method has

low statistical power since it is based on dichotomized rather than continuous

scores. For this reason, we present here analysis using a more

powerful measure, the binomial z-score for direct hits. (section 4, p. 83)

The use of z scores for direct hits still carries the problem of being affected by the number of trials in each experiment, but the shift away from significance levels was a step in the right direction. The final step, that of using effect sizes like Cohen's h, was undoubtedly a result of Chuck's interactions with Robert Rosenthal. Rosenthal's (1986) commentary on the ganzfeld debate was based almost exclusively on analyzing these effect sizes. In any case, all of Chuck's meta-analyses subsequent to the ganzfeld debate relied on effect sizes as the primary measures of interest.

The final maturation in Chuck's use and understanding of meta-analysis can be seen in his treatment of the autoganzfeld database. The paper by Honorton et al. (1990) is technically not a standard meta-analysis because all of the experiments were performed in the same laboratory and no literature search was necessary. Nevertheless, the treatment of the data is a good example of how meta-analysis can be used to evaluate a group of experiments, and it illustrates the level of statistical sophistication Chuck had attained by that time.

In keeping with Chuck's spirit of educating the parapsychological community on meta-analysis, I will briefly review the attractive highlights of the work presented in that paper. The first good feature is that all procedures used in the laboratory and in the experiments are described in painstaking detail. This does not contribute to the meta-analysis of the data, but is necessary in all scientific reports if they are to be used for future meta-analyses. We cannot guess what aspect of an experiment might be important to some future analyst, so it is critical to include all possible details when reporting experimental work.

Turning to the numerical analyses, the main advancement over previous reviews is that this one does not focus at all on significance levels for individual studies. Instead, effect sizes are reported, along with confidence intervals for them. When z scores are reported, the focus is not on "significance," but rather on whether they were consistently in the predicted direction.

A further level of sophistication is an examination of the homogeneity of effect sizes. This is done with a chi-squared test, and the results should be used as a guideline for whether different experimenters and/or studies are getting similar results in terms of effect sizes.

Finally, this paper shows how the autoganzfeld work provided a reasonable replication of the effect sizes estimated in the earlier ganzfeld meta-analysis. It is this completion of the exploratory/confirmatory progression that illustrates one of the most useful applications of meta-analysis to parapsychology, and I suspect Chuck recognized that potential long before it became clear to most of us.

Scientific Progress

Although Chuck ventured into other psi regimes to perform meta-analyses, I believe the intertwining of the ganzfeld work with meta-analysis is an exemplary illustration of how science should proceed, and one of which he was most proud. Perhaps it is too early to know the ultimate impact of Chuck's contributions, but I believe he could rest easier knowing that the story was soon to be told in the Psychological Bulletin (Bem & Honorton, in press) and would thus reach a wide audience.

The general progression for which that story is such a superb illustration begins when someone, Chuck in this case, forms a theory, on the basis of previous related work, that a certain kind of experiment might succeed. In this case, Chuck noticed that attention to internal states was important in gaining information through psi and was thus led to the ganzfeld procedure (see Honorton, 1977). The next step in our ideal progression is that the initial experiment is successful enough that others might try to replicate, or more likely, improve it.

Ideally, and as happened with the ganzfeld, this initial phase of investigation includes experiments by people in different laboratories but using similar enough procedures so that the results can be used in a comprehensive review. The introduction of meta-analysis in the late 1970s and early 1980s provided the quantitative machinery necessary for a review that could most constructively lead to the next step.

It should be noted that Hyman was correct in stating that a meta-analysis of the type initially conducted on the ganzfeld data base is "Exploratory Data Analysis ... [and] we should not confuse the outcomes of such explorations with Confirmatory Data Analysis (1985b, p. 115)." However, the results of such analyses are exactly what are needed to plan the confirmatory studies, and that is precisely what happened in this case.

The debate and subsequent papers connected with the initial ganzfeld meta-analysis were useful in two ways. First, the analyses of flaws led Hyman and Honorton (1986) to provide explicit rules that a proper confirmatory study should follow-rules that were followed in the autoganzfeld work. Second, they provided effect-size measures that, if matched in a confirmation, would indicate the presence of a repeatable effect. The most useful of the papers in this regard was the contribution by Rosenthal (1986) in which he analyzed the database for its effect size by Cohen's h, and made an adjustment for the flaws that had been identified.

Continuing with our ideal progression, once flaws and effect sizes have been identified, new studies can be planned to avoid those flaws and to try to confirm the size of the effects. The autoganzfeld studies were designed to meet those goals, and seem to have successfully done so. Certainly they have yet to be found flawed in any way, and the overall effect sizes are just what were predicted from the earlier exploratory meta-analysis.
COPYRIGHT 1993 Parapsychology Press
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1993 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:Charles Honorton
Author:Utts, Jessica
Publication:The Journal of Parapsychology
Article Type:Biography
Date:Mar 1, 1993
Previous Article:The last days in Edinburgh.
Next Article:PK in a competitive computer game: a replication.

Terms of use | Copyright © 2017 Farlex, Inc. | Feedback | For webmasters