The statistical analysis of single-subject data: a comparative examination.

Researchers in physical therapy and rehabilitation have recently advocated and used single-subject research designs to examine the efficacy of intervention.[1-3] Single-subject designs represent alternatives to traditional group-comparison procedures used in the evaluation of clinical interventions. The traditional method of data analysis used in single-subject designs is visual inspection.[4] Kazdin defines visual analysis as "reaching a judgment about the reliability or consistency of intervention effects by visually examining graphed data."[5(p232)] Parsonson and Baer[6] have described several advantages of visual analysis in single-subject research. One advantage they identified is the insensitivity of visual analysis to weak treatment effects. Insensitivity to small or weak treatment effects is an advantage according to Parsonson and Baer because it helps ensure that only large treatment effects with obvious clinical significance will be allowed into the research literature.

A frequent criticism of visual analysis in single-subject designs is that there are no formal decision rules available to researchers or clinicians to interpret the data.[7,8] Some investigators[9-11] have questioned the accuracy and reliability of judgments based on visual analysis alone. Gibson and Ottenbacher[12] and Harbst et al[13] found poor interrater reliability (intraclass correlation coefficients [ICCs]=.52-.66) among rehabilitation professionals who visually analyzed sets of AB graphs of hypothetical data. Furlong and Wampold[14] also reported that visual analysis resulted in inconsistent and unreliable interpretations of time-series data. In a related study, Jones and colleagues[9] compared visual inference and time-series statistical analysis and found low agreement (r=.60) between visual and statistical methods. The inconsistency associated with visual inferences has led some investigators[7-9] to advocate using statistical methods in single-system research. Christensen, for example, observed that "as single-subject designs have become more popular in the applied research areas, there has been an increased emphasis on the need for statistical analysis of collected data.[15(p261)]

Advantages of statistical methods are (1) that they can be used with unstable baseline data; (2) that they have the ability to detect small, but consistent, treatment effects that might be ignored in visual analysis; (3) that they can be used when the data are serially dependent; and (4) that they will produce consistent results, independent of who performs the computations.[5,8,11] Some investigators,[6,15,16] however, disagree with the use of statistical methods in single-subject research and believe that such methods are harmful to the development of knowledge in a clinically based discipline. These authors argue that advocating the use of quantitative methods in single-subject research will produce confusion between statistical and clinical significance and reduce the usefulness of the designs.

The development of quantitative procedures specifically for use with single-subject research data is a relatively recent phenomenon.[5] There is no tradition of statistical analysis in single-subject research, and many questions remain regarding when to use a particular statistical technique, the equivalence of different statistical procedures, and the advantages and limitations of existing statistical methods.[4,5] The purpose of our study was to answer two questions regarding the use of statistical methods with single-subject data: (1) To what extent do the results from three statistical tests commonly advocated for use with single-subject data agree? and (2) Are there identifiable components of graphed data such as slope, level, or trend that influence agreement among these three statistical tests?

Method

Graphs

A set of 42 graphs was used in this study; 21 of the graphs were AB diagrams of hypothetical data used in a previous study.[12] Each graph contained data for the baseline period (A phase) and for the treatment period (B phase). The 21 AB graphs contained eight data points in each phase and varied with respect to six factors, believed to influence the visual analysis of single-system data.[12,13] These factors were (1) the degree of mean shift across phases, (2) variability across phases, (3) change in level, (4) change in slope across phases, (5) amount of overlap across phases, and (6) degree of serial dependency in the data set. These characteristics of graphed data have been extensively described in previous reports.[8,9,12,13]

The remaining 21 graphs were collected from different single-subject research investigations published during the past 10 years in Journal of Applied Behavioral Analysis, Physical Therapy, Journal of the Association for Persons With Severe Handicaps, and Journal of Behavior Therapy and Experimental Psychiatry. The charts were selected based on the following criteria: (1) that each graph contain two or more phases and (2) that each graph include at least eight data points in the baseline and treatment phases. In studies that used multiple-baseline designs, the data represented in each of the baselines was considered as a separate AB design (eg, a multiple-baseline design across three subjects was considered as three different AB charts). Twelve of the 42 AB graphs were obtained from multiple-baseline designs. In published graphs that contained more than one phase (ie, ABAB designs), only the first baseline and treatment phases were used. Examples of four different graphs included in the analysis are presented in Figure 1.

To obtain the published graphs, each of the four journals listed was reviewed beginning with the 1990 volume. Each journal was reviewed beginning with the 1990 volume until six graphs meeting the previously defined criteria were identified. This initial review was conducted by the first author (MRN). All twenty-four graphs identified using this procedure were then examined by a rater with more than a decade of experience in conducting research using single-subject designs (KJO). Three of the original graphs were judged not suitable because of ambiguous or confusing presentation of the data. These three graphs were eliminated from the study, leaving a total of 21 published single-subject graphs for use in the subsequent analysis. The entire packet of graphs may be obtained from the second author.

Methods of Statistical Analysis

The following statistical procedures were used to analyze each of the 42 charts.

Split-middle method of trend estimation. This method, developed by White and Haring,[17] is often called the "celeration-line" approach. The procedure is designed to demonstrate whether the data are displaying an accelerating, decelerating, or stationary trend. The objective is to apply a "best-fit" trend line to the data points within a phase. The trend line is computed using data in the baseline (A) phase. The line is then extended to the treatment (B) phase to evaluate the effect of intervention on the subject's performance. The effect of treatment is assessed by comparing the proportion of data points above and below the line across the two phases. If the treatment is not effective, the proportion of data points below (and above) the line should remain the same in both the baseline phase and the treatment phase.[9] In our study, the celeration line was computed using the procedure described by White and Haring.[17] The steps used in computing the celeration line are outlined in Table 1. Examples of how to compute the celeration line are included in several widely used texts and journal articles.[5,7,8,11] Statistical significance using the split-middle method of trend estimation is determined using a binomial test.[11,17] The binomial test statistically compares the proportion of data points above and below the trend line in both the baseline and treatment phases.

[TABULAR DATA OMITTED]

Two-standard deviation band method. The second statistical method used in this study was the two-standard deviation band, or Shewart chart, method.[7] The two-standard deviation hand method is based on the computation of the standard deviation for the baseline data. Once the standard deviation is computed for the baseline data, band are drawn on the graph that contain scores within [+ or -]2 standard deviations from the mean. This procedure has the advantage of being sensitive to changes in variability across the phases of a single-subject design.

Gottman and Leiblum[18] argue that if at least two consecutive data points in the treatment phase fall outside of the two-standard deviation range, then a significant change in performance has occurred across the two phases. This statistical significance is based on the assumption that the likelihood of such an event occurring is less than 5 in 100. This inference is also based on the assumption that the data are independent and normally distributed. The steps involved in computing the two-standard deviation band method are outlined in Table 1.

The C statistic. Tryon[19] described the C statistic as a simple method of time-series analysis that can be used with small data sets. The C statistic can be applied to a data series with as few as eight observations and can also be used with data that exhibit serial dependency.[19] The C statistic is used initially to evaluate baseline data. If the baseline data are found to contain a nonsignificant trend, the baseline and intervention data are combined and the C statistic is again computed to determine whether a statistically significant change has occurred. If the baseline data are found to contain a statistically significant trend, the C statistic can still be applied to the data by constructing a comparison data series, but its usefulness is more limited in this case.[20] The steps to compute the C statistic are outlined in Table 1. The C statistic produces a z value, which is interpreted using the normal probability table for z scores.

Data Analysis

All 42 graphs were individually analyzed using each of the statistical methods described and outlined in Table 1. Following the application of the three statistical tests to each of the 42 graphs, the degree of agreement among the tests was computed. Agreement among the three statistical tests was determined by dividing the number of cases in which all three tests produced similar results by the total number of the graphs, multiplied by 100. The mean shift difference, variability, level, slope, overlap, and autocorrelation for each graph were determined using standard procedures.[9,13] The impact each of these graphic characteristics had on agreement among the three statistical tests was examined using the SYSTAT computer package.[21] All analyses were conducted by a physical therapist with graduate training in traditional data analysis and single-subject research (MRN). The results for 15 randomly selected graphs were independently checked by a second rater with extensive experience in single-subject research (KOJ). The agreement between the two raters was .91 (ICC, type 2,1).

Results

The results revealed an overall low percentage of agreement (38%) among the three tests. The rate of agreement for any two statistical procedures ranged from 48% for the celeration line and two-standard deviation band methods to 71% for the C-statistic and two-standard deviation band methods. There was agreement between the C statistic and celeration line for 24 of the 42 graphs (57%). The agreement was higher for the journal graphs (52%) than for the graphs containing hypothetical data (24%).

Table 2 presents the numeric values for each of the graphic components used in traditional visual analysis of single-subject data. These components are mean shift difference across the phases, change in variability across the phases, change in level across the phases, amount of overlapping data between phases, change in slope across the phases, and degree of serial dependency or autocorrelation in the data. Numeric values for each of these graphic characteristics were computed using standard methods.[10-13]

[TABULAR DATA OMITTED]

In addition to the quantitative information for each of the graphic characteristics, the results of the comparisons for each of the three statistical tests are also included in Table 2. Information is provided in Table 2 indicating whether the result of a statistical test for a particular graph was statistically significant (indicated by S in the table) or nonsignificant (indicated by NS in the table). Agreement among all three statistical tests is indicated by the letter A in the last column of Table 2 (labeled "Agreement") Disagreement among the three tests is indicated by the letter D in the final column of Table 2. For instance, for graph 1, the

celeration line and C-statistic methods both produced statistically significant results (S). In contrast, the result for the two-standard deviation band method was nonsignificant (NS). Because all three tests did not agree for this graph, the final column in Table 2 is labeled D (disagreement).

Inspection of Table 2 reveals that there was agreement among the three statistical tests for 16 of the 42 graphs (38%). Of the 16 graphs demonstrating agreement among all three tests, 15 (94%) were agreements, indicating a statistically significant result. There was only 1 graph in which there was agreement for all three statistical tests concerning a statistically nonsignificant effect. The celeration line and C statistic each reported 31 (74%) of the statistical evaluations as statistically significant and 19 (26%) as nonsignificant. Twenty-seven (64%) of the tests conducted using the two-standard deviation band method were reported as statistically significant and 15 (36%) as nonsignificant.

The descriptive information in Table 2 suggests that the measure of percentage of overlap across design phases was different for graphs in which there was agreement across all three statistical tests (mean overlap = 29.62, SD = 31.30) versus those graphs in which there was disagreement (mean overlap = 59.44, SD = 38.13). This finding suggests that when there is substantial overlap in the distribution of data points across phases, there is a greater chance of disagreement among the three statistical tests.

To further explore the relationship of the various graphic characteristics to the amount of agreement (or disagreement) among the three statistical procedures, an exploratory logistic regression analysis was conducted. The logistic regression was selected as the more appropriate procedure to use with a dichotomous or binary outcome variable (agreement versus disagreement among the three statistical tests). The variables included as predictors in the regression model were mean shift, variability, level, overlap, slope, and autocorrelation. The logistic regression model revealed that the overlap score was the best predictor of whether the three statistical tests would agree or disagree ([beta] = .49, P< .03). Change in level and slope were the next best predictors of agreement (or disagreement) among the statistical tests. The standardized regression coefficients for level, slope, mean shift, variability, and autocorrelation were not statistically significant.

Discussion

Statistical methods of analyzing data collected from single-subject studies are being advocated and applied with increasing frequency.[22] A large number of methods are currently available, and new procedures, such as the C statistic, are being developed at a rapid rate.[23,24] Some of these tests are complex and statistically sophisticated, whereas others (eg, the celeration line) place only minimal demands on the statistical knowledge required by the user.

A frequently identified advantage of statistical methods is the consistency of the results.[4,5] Assuming that the calculations are properly done, statistical procedures will produce the same results for a given data set regardless of who performs the computations. As indicated previously, this same degree of consistency is not found when single-subject data are analyzed using visual inspection.[9-11] Although it is true that a particular statistical procedure will provide consistent results (when the calculations are done correctly), the results of this investigation indicate that there is limited consistency across different quantitative methods of analyzing single-subject data. The three statistical procedures examined in this study produced consistent results only 38% of the time. The amount of agreement between any two methods of statistical analysis collapsed across all 42 graphs ranged from 48% to 71%.

The results of this investigation also revealed that the three statistical tests are more likely to agree that there is a statistically significant difference across design phases than to agree that there is no statistically significant difference, Ninety-four percent of the instances in which there was agreement among all three statistical tests involved statistically significant results. For only 1 of the 42 graphs was there agreement among the three statistical procedures that no statistically significant difference existed across the design phases.

The three statistical procedures were more likely to agree if the graphs contained previously published data (52%) versus hypothetical data (24%). Visual inspection was the method of data analysis used in all of the published graphs. Parsonson and Baer[6] have argued that one of the advantages of visual analysis is the lack of sensitivity to small treatment effects. They contend that if a change across phases is large enough to be seen with the "naked eye," it is likely to be clinically significant. Thus, single-subject results published in the literature should include graphs that contain large treatment effects. There will generally be more agreement for graphs with large treatment effects than for charts with small changes across phases. In contrast to the published graphs, the charts of hypothetical data were constructed to emphasize specific visual characteristics (eg, slope, level, autocorrelation, and so on). The changes in these graphs represented a range of treatment effects that were frequently smaller than those included in the published single-subject graphs.

The largest discrepancy in statistical conclusions existed between the two-standard deviation band method and the celeration line approach (split-middle method of trend estimation). When collapsed across all 42 graphs, these two procedures agreed only 48% of the time. Figure 2 illustrates the results of a graph that produced disagreement between these two methods. The celeration line in Figure 2a indicates that there is no significant, change in performance across the two phases. The proportion of data points above and below the celeration line is essentially the same in both phases, and the statistical result is nonsignificant.

In contrast, the two-standard deviation band method (illustrated in Fig. 2b) indicates that there has been a significant change in performance across the two phases. There are two consecutive data points outside the two-standard deviation band range in the treatment (B) phase. Visual inspection of Figure 2 reveals that there is an accelerating trend in the data during the baseline and that this trend continues during the treatment phase. In this example, the celeration line approach appears to provide the correct statistical conclusion. The two-standard deviation band method does not provide an accurate result, due to the trend that exists in the baseline data. The two-standard deviation band method should only be used when the baseline data are stable without any obvious trend. Another limitation of the two-standard deviation band method not apparent in Figure 2 is that the procedure is very sensitive to extreme values. One outlier may adversely influence the variance in a small set of data and produce misleading results.

The importance of clearly identifying the assumptions associated with the use of statistical methods designed for single-subject data is illustrated in Figure 2. Unfortunately, these assumptions are frequently not identified or not followed by clinicians and researchers using single-subject methods. Bloom and Fisher[7] have accurately observed that many of the statistical procedures developed for single-subject research are relatively recent innovations and that the mathematical and statistical limitations of these procedures remain unexamined. The results of our investigation indicate that continued research is needed to identify the specific assumptions associated with the use of statistical methods such as the celeration line, two-standard deviation band method, and C statistic.[25-27] Research is also needed to identify, when a particular statistical procedure should be used. Selected statistical procedures may be most useful with single-subject graphed data that display certain characteristics (eg, changes in level) hut not other characteristics (eg, overlap). Recent research in visual analysis has indicated that specific characteristics of graphed data, for example, changes in slope, are associated with increased disagreement among raters.[28-30]

Criteria for comparing the advantages and limitations of different single-subject statistical methods do not currently exist. There is a clear need to establish a set of objectively based decision rules indicating when a particular statistical technique is most appropriate and when it should not be used. Many discussions of single-subject data analysis imply that various statistical procedures are uniformly useful for all data. For example, in introducing the C statistic, Tryon states that

...the C statistic is a simple, yet elegant, method for quantitatively evaluating the presence of changes due to treatment interventions in serially dependent time-series data. It is an omnibus test for abrupt changes in the level of a time series as well as gradual changes in its slope.[19(p428)]

Limitations

In considering the implications of this study, it is important to recognize certain limitations. One limitation is that only three statistical procedures were examined. Other statistical methods exist and could have been included in the analysis. The three statistical tests selected were chosen because they appear to be the most frequently used or advocated in the single-subject research literature.[5,7,8,11] Additional research is necessary to extend the findings of our study to other statistical procedures.

Another limitation is the use of only AB graphs as the unit of analysis. Bloom and Fisher[7] argue that the AB design is the fundamental framework in single-subject methodology. Other authorities,[6] however, have suggested that the simple AB design is not widely used as a single-subject research strategy and that different designs such as the ABAB or multiple-baseline should be used to examine the effectiveness of various analysis methods. The AB design has been used extensively in research examining the strengths and weaknesses of visual analysis.[7-10] The argument made in these previous investigations is that if consistency cannot be obtained across the two phases of the simple AB design, it will be even more difficult to obtain consistency in more complex designs.[7] For this reason, and for the sake of computational simplicity, the AB design was selected for use in this investigation.

Conclusions

Kazdin[5] has identified three situations in which statistical procedures may be of value in analyzing the data from a single-subject study. The first situation involves the failure to establish a stable baseline. It is not always possible to produce a stable baseline of patient performance prior to the introduction of treatment. In these instances, the patient's baseline data may demonstrate fluctuations or an accelerating or decelerating trend. In such a case, statistical methods sensitive to the detection of variability and trend may be useful as adjuncts to visual analysis of graphed data. The second situation identified by Kazdin in which statistical procedures may be useful occurs when a new treatment is being evaluated or when the expected treatment effects are not well understood by the investigator. Some interventions, particularly new interventions, may not produce immediate or dramatic treatment effects. Treatment effects that appear weak or ambiguous when evaluated using only visual analysis may, nevertheless, be repeatable and statistically significant.[11,22] In such cases, a new treatment may be refined and further developed and may eventually produce large clinical effects. The final reason identified by Kazdin[5] for using statistical procedures is the need to statistically control for extraneous factors in naturalistic environments. Clinical research studies occur in the natural environment, and it is often not possible to maintain or achieve the desired level of control over extraneous variables. Statistical procedures can assist the researcher in controlling extraneous variability and in assessing the impact and extent of patient change. Statistical control of potential confounding factors is a characteristic of both single-subject and traditional group-comparison research.

As noted previously, evidence of inconsistent visual analysis of single-subject data has been provided by numerous investigators.[8-11] This inconsistency has prompted some authorities[7,9,12] to advocate the use of statistical analysis in conjunction with visual inspection of single-subject data. The results of our investigation indicate that statistical analysis of single-subject data should not be viewed as a replacement for visual analysis. Kazdin[5] and others[9-11] have argued that single-subject data should always be graphed and visually analyzed. The results of this investigation support the need for multiple approaches to the analysis of single-subject data and the need for additional research to develop a set of statistical decision rules to assist investigators in making judgments regarding the effectiveness of therapeutic interventions.

References

[1] Ottenbacher KJ. Clinically relevant designs for rehabilitation research: the idiographic model. Am J Phys Med Rehabil, 1990;69:286-292. [2] Gonnella C. Single-subject experimental paradigm as a clinical decision tool. Phys Ther. 1989;69:601-609. [3] Harris SR, Riffle K. Effects of inhibitive ankle-foot orthoses on standing balance in a child with cerebral palsy: a single-subject design. Phys Ther. 1986;66:663-667. [4] Gingerich WJ. Significance testing in single case research. In: Rosenblatt A, Walbfogel D, eds, Handbook of Clinical Social Work. San Francisco, Calif. Jossey-Bass Inc Publishers; 1983:694-720. [5] Kazdin AE. Single-case Research Designs: Methods for Clinical and Applied Settings. New York, NY: Oxford University Press Inc; 1982. [6] Parsonson BS, Baer DM. The graphic analysis of data. In: Poling A, Fuqua RW, eds, Research Methods in Applied Behavior Analysis: Issues and Advances, New York, NY: Plenum Press; 1986:157-186. [7] Bloom M, Fisher J. Evaluating Practice: Guidelines for the Accountable Professional Englewood Cliffs, NJ: Prentice Hall; 1982. [8] Wolery M, Harris SR. Interpreting results of single-subject research designs. Phys Ther, 1982;62:445-452. [9] Jones RR, Weinrott MR, Vaught RS. Effects of serial dependency on the agreement between visual and statistical inferences. J Appl Behav Sci. 1978;11:277-283, [10] DeProspero A, Cohen S. Inconsistent visual analyses of intrasubject data. J Appl Behav Anal 1979;12:573-579. [11] Ottenbacher KJ. Evaluating Clinical Change: Strategies for Occupational and Physical Therapists. Baltimore, Md: Williams & Wilkins; 1986. [12] Gibson G, Ottenbacher KJ. Inconsistent visual analysis of intrasubject data: an empirical analysis. J Appl Behav Sci. 1988;24:298-314. [13] Harbst KB, Ottenbacher KJ, Harris SR. Interrater reliability of therapists judgments of graphed data. Phys Ther. 1991;71:107-115. [14] Furlong MJ, Wampold BE. Visual analysis of single-subject studies by school psychologists. Psychology in the Schools. 1981;18:80-86. [15] Christensen LB Experimental Methodology. 2nd ed. Boston, Mass: Allyn and Bacon; 1980. [16] Baer DM. Perhaps it would be better not to know everything. J Appl Behav Anal. 1977; 10:167-172. [17] White OR, Haring NG, Exceptional Teaching. 2nd ed. Columbus, Ohio: Charles E Merrill; 1980. [18] Gottman JM, Leiblum SR, How to Do Psychotherapy and How to Evaluate It. New York, NY: Holt, Rinehart and Winston; 1974. [19] Tryon WW. A simplified time-series analysis for evaluating treatment interventions. J Appl Behav Anal 1982;15:423-429. [20] Blumberg CJ. Comments on "Simplified time-series analysis for evaluating treatment interventions." J Appl Behav Anal 1984;17: 539-542. [21] Wilkinson L. SYSTAT The System for Statistics. Evanston, Ill: SYSTAT Inc; 1989, [22] Ottenbacher Kj. Analysis of data in idiographic research: issues and methods. Am J Phys,Med Rehabil. 1992;71:202-208. [23] Edington ES. Non-parametric tests for single-subject multiple schedule experiments. Behav, Assess. 1982;4:83-91. [24] Wolery M, Billingsley FF. An application of Revusky's Rn test to slope and level changes. Behav Assess, 1982;4:93-103. [25] Hojem MA, Ottenbacher KJ. Empirical investigation of visual inspection versus trendline analysis of single-subject data. Phys Ther. 1988;68:983-988. [26] Kratochwill TR, Levin JR. On the applicability of various data analysis procedures to the simultaneous and alternating treatment designs in behavior therapy research. Behav Assess. 1980;2:353-360. [27] Matyas TA, Greenwood KM. Visual analysis of single-case time-series: effects of variability, serial dependency and magnitude of intervention effects. J Appl Behav Anal. 1990;23:341-351. [28] Ottenbacher KJ, Cusick A. An empirical investigation of inter-rater agreement for single-subject data using graphs with and without trend lines. Journal of the Association for Persons With Severe Handicaps. 1991;16:48-55. [29] Ottenbacher KJ. Visual analysis of single-subject data: an empirical analysis. Ment Retard. 1990;28:283-290. [30] Johnson MB, Ottenbacher KJ. Trend line influence on visual analysis of single-subject data in rehabilitation research. Int Dis Studies. 1991;13:55-59