Use of the standard error as a reliability index of interest: an applied example using elbow flexor strength data.

A review of the physical examination literature reveals a plethora of competing tests for most components of the assessment.[1-5] For example, numerous devices, methods, and protocols exist for assessing range of motion, muscle strength, and joint stability.[1-5] The goal of many clinical measurement studies has been to aid in selecting the best measure for a specific purpose. One type of measurement study reported frequently is the reliability study, and numerous design choices are available to investigators who are interested in conducting such studies. Several designs reported often in the physical therapy literature include the simple replication design and the interrater design (equivalent to the intertrial or inter-occasion designs),(*) where raters, trials, or occasions have been considered to be either fixed or random.[6,7] A simple replication study exists when multiple measurements are taken on a sample of subjects and there is no structure linking the replicate measures to each other. Different raters assess different subjects, and there is no need for the number of replicate measurements to be the same for each subject. Presuming that raters are randomly allocated to subjects, within an analysis-of-variance (ANOVA) context, this represents a one-way random effects model.

A potential limitation of this design is that all components of error are included in a single error term. An equally popular and slightly more complex design is the interrater reliability study, where all raters evaluate all subjects (de, raters are crossed with subjects). Data from this design can be analyzed using a two-way ANOVA.[6,7] This approach allows partitioning of the variance terms into patients, raters, and error. With this design, the investigator must determine whether inferences are to be unique to only those raters taking part in the study or whether the raters are intended to represent a sample from a larger population of raters. In the first case, the rater effect is considered to be fixed, and in the second case, the rater effect is random.

Results from reliability studies, where the measure is quantified on an interval or ratio scale, are usually expressed by reporting the intraclass correlation coefficient (ICC) or the standard error of measurement (SEM).[1-5,8,9] The ICC provides information about a measure's ability to differentiate among patients, and it is obtained by dividing the true variance by the total variance.[10,11] When patients are the object of measurement, the true variance represents the variance among patients. The SEM expresses measurement error in the same units as the original measurement, and it is not influenced by variability among patients.[10,11]

Relationship Between the SEM and the ICC

The relationship between the SEM and ICC is defined by the following expression:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] represents the total variance and [Rho] is the ICC. Using a bit of algebra, it can be shown that:

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] is the error variance and equals the mean square error term from an ANOVA. Although the SEM and ICC are related, they do not convey the same information.[12] This point is highlighted in the results presented by Hayes and colleagues,[8] who studied the test-retest reliability of measurements of knee joint range of motion in patients with osteoarthritis. Tables 1 and 2 present the ANOVA results for extension and flexion measurements, respectively. The ICCs for extension and flexion are .71 and .96, respectively, and the SEMs for extension and flexion are 3.23 and 5.03 degrees, respectively. Notice that the ICC shows the flexion measurements to be more reliable, whereas the SEM indicates that the extension measurements are more reliable. The interpretation of this paradox is that, within the context of Hayes and colleagues' study design, knee flexion measurements are more useful for discriminating among patients, whereas the magnitude of error (expressed in degrees) is less for knee extension measurements. A second important point is that both the ICC and SEM represent point estimates of the population values for these variables. Although considerable attention has been provided to reporting ICCs (eg, stating the type and confidence interval are usual requirements),[13,14] the same thoughtfulness has not been afforded to reporting SEMs. The goals of this article are to illustrate a process for obtaining an interval estimate of the SEM, to show how this process can be used to estimate sample size, and to illustrate how error estimates can be compared statistically. To illustrate these analyses, we use actual strength assessment data.

Table 1.
Analysis-of-Variance Results for Knee Extension Range of Motion(a)

                                        Variance
Source           df   SS        MS      Estimate

Between people   21   1264.91   60.23   24.91
Within people    22   229.00    10.41   10.41
Total            43   1493.91           35.32

(a) Adapted from the work of Hayes et al[8] (Tab. 6, Examiner 1).

Table 2.
Analysis-of-Variance Results for Knee Flexion Range of Motion(a)

                                               Variance
Source            df    SS          MS         Estimate

Between people    21    28219.73    1343.80     659.11
Within people     22    563.00        25.59      25.29
Total             43    28782.73                684.40

(a)Adapted from the work of Hayes et al[8] (Tab. 7, Examiner 1).

Example Study

The data represent make and break tests performed on the right elbow flexors of 30 female volunteers. Two make tests and two break tests were performed on all subjects at the same test session by the same examiner. The order of testing methods was balanced across subjects. The data are shown in the Table 3, and summary ANOVA tables for the tests are presented in Tables 4 and 5. The error variances are equal to the mean square error (MSE) terms, and the SEMs are the square roots of these terms. Accordingly, the error variance is 0.76 (SEM=0.87 kgf) for the make test and 1.51 (SEM=1.23 kgf) for the break test. The difference scores for each method, shown in Table 3 and reproduced in Table 6, were obtained by subtracting the second trial score from the first trial score (eg, trial 1-trial 2). The correlation between the difference scores for the make and break tests is 0.21, suggesting a modest degree of dependency.

Table 3.
Summary Data for the Make and Break Tests (in Kilograms of Force)

         Make Test      Break Test

Subject                   Differ-                     Difference
No.        1      2       ence       1        2

1       19.1      16.2    2.9        25.4     22.3    3.1
2       18.2      18.3   -0.1        17.7     20.3   -2.6
3       19.5      19.2    0.3        19.6     18.8    0.8
4       23.7      24.5   -0.8        29.2     23.7    5.5
5       27.1      26.2    0.9        27.2     26.3    0.9
6       20.1      22.7   -2.6        23.0     23.9   -0.9
7       21.1      20.3    0.8        20.9     19.2    1.7
8       25.4      25.7   -0.3        26.2     27.6   -1.4
9       17.8      18.9-   1.1        21.5     21.9   -0.4
10      17.8      17.4    0.4        20.4     20.4    0.0
11      22.8      22.6    0.2        22.1     21.2   -1.1
12      16.7      15.2    1.5        19.5     18.3    1.2
13      20.3      19.6    0.7        20.1     21.2   -1.1
14      15.3      16.3   -1.0        19.6     16.6    3.0
15      23.0      21.8    1.2        23.5     19.3    4.2
16      14.8      15.7   -0.9        18.9     18.2    0.7
17      17.6      16.2    1.4        18.4     18.6   -0.2
18      19.1      17.2    1.9        19.2     20.3   -1.1
19      31.7      29.4    2.3        31.8     31.4    0.4
20      16.5      16.3    0.2        18.9     18.9    0.0
21      19.7      19.4    0.3        21.5     20.5    1.0
22      23.8      22.7    1.1        22.7     21.2    1.5
23      22.8      23.8   -1.0        23.1     22.8    0.3
24      19.3      18.1    1.21       21.4     20.5    0.9
25      15.1      15.2   -0.1        17.7     19.2   -1.5
26      16.2      16.8   -0.6        19.8     18.8    1.0
27      17.7      17.9   -0.2        17.5     16.9    0.6
28      16.7      17.6   -0.9        19.7     19.1    0.6
29      16.2      16.3   -0.1        15.8     15.0    0.8

30      16.4      18.5   -2.1        19.2     20.8   -1.6
X       19.72     19.53   0.18       21.38    20.83   0.56
SD      3.91       3.68   1.23        3.61     3.35   1.74
Table 4.
Analysis-of-Variance Results for Make Test

                                            Variance
Source             df     SS        MS      Estimate

Between subjects   29     814.31    28.08   13.66
Trials              1       0.50     0.50    0.00
Within subjects    29      22.06     0.76    0.76
Total              59     836.87            14.42
Table 5.
Analysis-of-Variance Results far Break Test

                                           Variance
Source              df   SS       MS       Estimate

Between subjects    29   659.22   22.731   0.61
Trials               1     4.65    4.65    0.10
Error               29    43.80    1.51    1.51
Total               59   707.67           12.12
Table 6.
Difference Scores for Make and Break Tests

Subject    Make Test    Break Test
No.        Difference   Difference      Sum     Difference
1           2.9          3.1            6.0     -0.2
2          -0.1         -2.6           -2.7      2.5
3           0.3          0.8            1.1     -0.5
4          -0.8          5.5            4.7     -6.3
5           0.9          0.9            1.8      0.0
6          -2.6         -0.9           -3.5     -1.7
7           0.8          1.7            2.5     -0.9
8          -0.3         -1.4           -1.7      1.1
9          -1.1         -0.4           -1.5     -0.7
10          0.4          0.0            0.4      0.4
11          0.2         -1.1           -0.9      1.3
12          1.5          1.2            2.7      0.3
13          0.7         -1.1           -0.4      1.8
14         -1.0          3.0            2.0     -4.0
15          1.2          4.2            5.4     -3.0
16         -0.9          0.7           -0.2     -1.6
17          1.4         -0.2            1.2      1.6
18          1.9         -1.1            0.8      3.0
19          2.3          0.4            2.7      1.9
20          0.2          0.0            0.2      0.2
21          0.3          1.0            1.3     -0.7
22          1.1          1.5            2.6     -0.4
23         -1.0          0.3           -0.7     -1.3
24          1.2          0.9            2.1      0.3
25         -0.1         -1.5           -1.6      1.4
26         -0.6          1.0            0.4     -1.6
27         -0.2          0.6            0.4     -0.8
28         -0.9          0.6           -0.3     -1.5
29         -0.1          0.8            0.7     -0.9
30         -2.1         -1.6           -3.7     -0.5

Estimating a Confidence Interval for the Standard Error of Measurement

Variances of independent samples drawn randomly from a population with a normal distribution have a chi-square distribution.[15] Specifically, the relationship is defined as

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] is the chi-square distribution on n-1 degrees of freedom, [s.sup.2] is the sample variance, [Sigma.sup.2] is the population variance, and the symbol ~ represents [Sigma.sup.2] is distributed as." Using the approach outlined by Armitage and Berry,(16)(PP221-223) an illustration of a confidence interval calculation based on the sample error variance for the make test, 0.76 from Table 4, is shown. Specifically, a two-sided 95VO confidence interval for the error variance associated with knee extensor measurements is defined as

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where SSE is the sum of squares error (22.06) in the ANOVA table, [MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII] represents the chi-square value for the probability level CY, and dfe is the degrees of freedom associated with SSE (ie, 29). Taking the square root of the confidence interval values for [Sigma.sup.2] yields a 95% confidence interval on the SEM of 0.69 to 1.17 degrees.

Sample Size Estimation

This section illustrates a method for determining sample size when an investigator is interested in designing a study to estimate the magnitude of the SEM. Consider the following scenario. A researcher is interested in conducting a study to determine the reliability of make tests for the elbow flexors in patients with osteoarthritis. Based on a pilot study on subjects with no known history of osteoarthritis, the researcher expects the subsequent investigation using an enhanced protocol on patients with osteoarthritis to yield a SEM of 0.71 kgf; however, the researcher believes that a SEM as high as 0.95 kgf would be acceptable. Rephrasing this last sentence yields the following hypotheses:

Null hypothesis: The measurement protocol of interest in

patients with osteoarthritis is reliable

(SEM [is greater than or equal to] 0.95 kgf.

Alternate hypothesis: The measurement protocol of interest

in patients with osteoarthritis is not reliable (SEM [is greater than] 0.95

kgf).

Moreover, the researcher wants to evaluate these hypotheses at a one-sided 100(1-[Alpha])% confidence level of 95% when the sample SEM is 0.71. The first step is to square the SEM values to produce variances. Thus, the researcher expects to find an error variance of 0.50 (ie, 0.712) with an upper acceptable variance limit of 0.90 (de, 0.952). Once again, we make use of the relationship

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

where [s.sup.2] is the expected study variance and [Sigma.sup.2] is the maximal acceptable value for the population variance (ie, the upper 95% confidence limit is set to this value).

Using the approach outlined by Armitage and Berry,(16)(PP221-223) n-1 is replaced by the degrees of freedom associated with the error variance (dfe) and this term is isolated

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

Using an iterative process, it is found that a chi-square value of 11.59 on 21 degrees of freedom produces the desired one-sided confidence level. Thus, 22 patients are required, provided two measurements are performed on each patient. Often, an investigator may want to obtain more than two measurements on each patient, and this will influence the number of patients required for the study. Table 7 supplies sample size estimates for various combinations of [Sigma.sup.2]/[s.sup.2] ratios and number of measurements per patient. For the example mentioned previously, [Sigma.sup.2]/[s.sup.2] is equal to 1.8 (0.90/0.50). By referring to Tables 4 and 5, for a [Sigma.sup.2]/[s.sup.2] ratio of 1.8, it is evident that 22 patients are required when two measurements are performed on each patient, 12 patients are needed when three measurements are performed on each patient, and so on.

Table 7. Approximate Sample Sizes for o One-Sided Upper 95% Confidence Limit for Population Variance

No of           Ratio of Population Variance to
Measurements    Sample Variance
per Subject     1.3   1.4   1.5   1.6   1.7   1.8   1.9   2.0

2               91    59    42    33    27    22    20    17
3               46    30    22    17    14    12    10     9
4               31    20    15    11    9      8     7     6
5               24    16    11     9    7      6     6     5
6               19    12     9     7    6      5     4     4

Comparing Error Estimates

Researchers and clinicians are constantly in search of strategies, techniques, and protocols that minimize measurement error. In pursuing this goal, it is necessary to compare the methods or measures of interest by estimating the error associated with the competing measures. Estimates of measurement error for the competing measures can be obtained on different samples or on the same sample. In the case where one group of subjects is used to estimate the error associated with method A and a second group of subjects is used to approximate the error for method B, the measurements are usually considered to be independent. When both methods are evaluated on the same group of subjects, however, the measurements are likely to be dependent. The intent of the following section is to show several statistical approaches for comparing error variances, and ultimately their square roots, the SEMs. The data for the make and break tests will be used to demonstrate the analyses.

Independent Samples: Variance Ratio Approach

Variance estimates from two independent samples can be compared using the ratio of the variance estimates.[16(PP115-117)] Variance ratios are distributed as an F distribution. Specifically, the larger variance is divided by the smaller variance, and this value is converted to a probability by referring to the summary table of F values for the appropriate degrees of freedom.

[MATHEMATICAL EXPRESSION NOT REPRODUCIBLE IN ASCII]

[F.sub.(29,29)] = 1.51/0.76

[F.sub.(29,29)] = 1.98

P = .035

This analysis suggests that the break test variance is significantly greater than the make test variance.

Dependent Samples: Paired Approach

This approach is discussed by Armitage and Berry[16] and Snedecor and Cochran.[17] It is appropriate when pairs of measurements are being compared. To use this technique to compare within-subject error variances, we have added an intermediate step. This step requires calculation of the difference between trials (ie, trial 1-trial 2) for each of the assessment methods. Thus, we are actually testing whether the variance of the difference scores for the two assessment methods (ie, make test difference and break test difference) differs, rather than whether the error variance associated with a single measurement differs. Given that the variance of a difference score is equal to the variance of a single measure times two, the statistical test provides valid information about the extent to which the error variances, and ultimately the SEMs of the make and break tests, differ. Table 6 provides the information necessary to formally evaluate the variances. The first step involves calculating the sum (make test+break test) and difference (make test--break test) for the difference scores shown in Table 6. The variances are compared by calculating the correlation between the sum and difference scores and testing whether the magnitude of the correlation coefficient differs from zero. In our example, the correlation between the sum and difference scores is 0.34 (F=3.58; df=1,28; P-.069), suggesting that there is a marginal difference associated with the variances of the make and break tests.

Dependent Samples: Bootstrap Approach

The paired method mentioned above is restricted to two observations (trials or judges) per subject for each measure being assessed. An alternate approach that avoids this shortcoming is the bootstrap method. Although reviewing this technique in detail is beyond the scope of this article, a brief explanation of its essential features will be offered. Specifically, the bootstrap method uses the sample data to estimate the distribution of the parent population.[17-19] This estimation is accomplished by drawing a relatively large number of random draws with replacement.[17-19] The random draws are used to construct the estimated population distribution, and the confidence interval of interest can be determined.

This approach was used to estimate the distribution of the difference in make and break test error variances. Specifically, 1,000 bootstrap samples, each with a sample size of 29 drawn with replacement, were obtained. This was accomplished by writing a macro program using MINITAB statistical software.[dagger] The one-tailed probability associated with the observed error variance difference of 0.75 (de, 1.51 - 0.76 from Tabs. 4 and 5) was P=.053. Again, the magnitude of this value suggests that the error variance for the break test may be marginally greater than for the make test.

Summary

This article shows that although the ICC and SEM are related, they convey different information about the reliability of a measure. Specifically, the ICC is influenced by multiple sources of variation (eg, subjects, raters, trials) and by error, whereas the SEM is affected by error variation only. A method for estimating a confidence interval for the SEM is illustrated, and how an a priori specification of confidence interval width can be used to estimate sample size is discussed. Finally, several approaches used to compare error variances are discussed.

(*) The repeated-measures factor could also represent trials or occasions; however, we will sue the word "raters" in the text.

[dagger] MINITAB Inc, 3081 Enterprise Dr, State College, PA 16801-3008.

References

[1] Bohannon RW. Make tests and break tests of elbow flexor muscle strength. Phys Ther. 1988;68:193-194.

[2] Malouin F, Boiteau M, Bonneau C, et al. Use of a hand-held dynamometer for the evaluation of spasticity in a clinical setting: a reliability study. Physiotherapy Canada. 1989;41:126-134.

[3] Helewa A, Goldsmith CH, Smythe A. The modified sphygmomanometer--an instrument to measure muscle strength: a validation study. J Chronic Dis. 1981;34:353-361.

[4] Stratford PW, Miseferi D, Ogilvie R, et al. Assessing the responsiveness of five KT1000 knee arthrometer measures used to evaluate anterior laxity at the knee joint. Clin J Sports Med. 1991;1:225-228.

[5] LaStayo PC, Wheeler DL. Reliability of passive wrist flexion and extension goniometric measurements: a multicenter study. Phys Ther. 1994;74:162-176.

[6] Fleiss JL. The Design and Analysis of Clinical Expenments. New York, NY: John Wiley & Sons Inc; 1986:1-32, 291-300.

[7] Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol BulL 1979;86:420-428.

[8] Hayes KW, Peterson C, Falconer J. An examination of Cyriax's passive motion tests with patients having osteoarthritis of the knee. Phys Ther. 1994;74:697-709.

[9] Eliasziw M, Young SL, Woodbury MG, Fryday-Field K Statistical methodology for the concurrent assessment of interrater and intrarater reliability: using goniometric measurements as an example. Phys Ther. 1994;74:777-788.

[10] Streiner DL, Norman GR. Health Measurement Scales: A Practical Guide to Their Development and Use. New York, NY: Oxford University Press; 1989:79-89.

[11] Domholdt E. Physical Therapy Research: Principles and Applications. Philadelphia, Pa: WB Saunders Co; 1993:153-157.

[12] Stratford PW. Reliability: consistency or differentiating among subjects? Phys Ther. 1989;69:299-300. Letter to the editor.

[13] Krebs DE. Declare your ICC type. Phys Ther. 1986;66:1431. Letter to the editor.

[14] Stratford PW. Confidence limits for your ICC. Phys Ther. 1989;69: 237-238. Letter to the editor.

[15] Kleinbaum DG, Kupper LL, Muller KE. Applied Regression Analysis and Other Multivariable Methods. 2nd ed. Boston, Mass: PWS-Kent Publishing Co; 1988:23.

[16] Armitage P, Berry G. Statistical Methods in Medical Research. 3rd ed. Boston, Mass: Blackwell Scientific Publications; 1994:115-117, 221-223.

[17] Snedecor GW, Cochran WG. Statistical Methods. 6th ed. Ames, Iowa: Iowa State University Press; 1967:195-197.

[18] Stine RA. An introduction to bootstrap methods: examples and ideas. Sociological Methods Research. 1989;18:243-291.

[19] Efron B, Gong G. A leisurely look at the bootstrap, the jackknife, and cross-validation. American Statistician. 1983;37:36-48.

PW Stratford, PT, is Assistant Professor, School of Rehabilitation Science, and Associate Member, Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada. Address all correspondence to Mr Stratford at Faculty of Health Sciences, School of Rehabilitation Science, McMaster University, OT/PT Bldg T-16, 1280 Main St W, Hamilton, Ontario, Canada LOS 4K1 ([e-mail]).

CH Goldsmith, PhD, is Professor of Clinical Epidemiology and Biostatistics, McMaster University, and Honorary Professor of Physical Therapy, Department of Physical Therapy, University of Western Ontario, London, Ontario, Canada.

This article was submitted June 2, 1996, and was accepted January 7, 1997.