The effect of awareness of measurement error on physical therapists' confidence on their decisions.

Clinical decision making is currently of great importance in physical therapy. The American Physical Therapy Association (APTA) has sponsored two symposia devoted to various aspects of decision making, including decision-making processes and the many factors that contribute to clinical decisions. At one of those conferences, Rothstein(1) suggested that one important factor is an understanding of error of measurement, To that end, many articles on the validity and reliability of data from various tests have appeared in the recent professional literature.

Validity refers to the ability of a testing procedure to measure what it is supposed to measure. Reliability refers to the precision of measurements or the consistency of scores on multiple measurements of the same construct or across time or evaluators. Reliability is a reflection of the degree to which data are free of random error and represents uncertainty associated with the data. An estimate of the error associated with an individual's score is the standard error of measurement (SEM). One SEM represents the 68% confidence interval surrounding the individual's score. For example, if a student received a score of 64 points on a classroom test, and the lowest passing score was set at 65 points, the student would probably be failed. If one SEM for that examination were 3 points, however, there would be a 68% chance that the resuiting 61- to 67-point interval would include the student's true score. The student's true score might be above 65, and the failure decision could be erroneous.

The SEM for a data set can be calculated from the reliability (r) of the data and the standard deviation (SD) of the distribution of scores in the data set (SD [square root of 1-r]). When the reliability is 1.00, the SEM is 0.00. As the reliability decreases, the SEM increases to a maximum value equal to the standard deviation of the distribution of scores.

Because reliability of clinical test data represents a degree of uncertainty associated with those data, it is important to know whether reliability and error information affect physical therapists' confidence in their decision making. The broad goal of this study was to determine whether physical therapists use measurement information in their decision making. The specific purposes of the study were (1) to determine whether physical therapists understand the concepts of reliability and measurement error, (2) to determine whether physical therapists have reasonable estimates of the error associated with their measurements, and (3) to determine whether confidence or certainty about a decision is affected by information about reliability and error of measurement.

Research on confidence in decisions indicates that people tend to be overconfident about their decisions.(2-4) Reasons for overconfidence include fear of invalidity (being wrong),(5) need for structure,(5) and failure to use disconfirming evidence.(6) People who fear being wrong have low confidence in early hypotheses, but search for more information and have high confidence in their final hypotheses.(5) Physical therapists make decisions that potentially have a high cost if wrong. If the cost of a wrong decision triggers this fear, physical therapists could display high confidence in their final decisions. People with a high need for structure inhibit information search, basing a judgment on little information with high confidence.(5) Physical therapists have concrete cognitive Styles,(7,8) suggesting a need for structure. Therefore, they might be expected to have high confidence in their decisions if little information is available. On the other hand, confidence is lower when people are instructed to provide reasons against a decision.(6) If physical therapists perceive error information as disconfirming evidence, then their certainty would be expected to decrease when measurement error information is provided.

Physical therapists could obtain information about reliability and error of measurement in their formal education or from reading the professional literature. Physical therapists are not likely, however, to use the professional literature for this purpose. Physical therapists have not used journals preferentially for professional development(9) or actively sought information in journals,(10) and they have not found research articles to be useful in their practices.(11)

Rothstein(1) suggested that expert practitioners develop a sense of error from experience. If experienced physical therapists understand the meaning of error, then they might alter their confidence in their decisions. Expert practitioners in psychology have been shown to be less confident than less experienced practitioners.(12-14) This relationship, however, has not been universally supported.(3,4)

Physical therapists should know how to integrate measurement information into their decision making. Other professionals tend to discount low-credibility information(15-19) and decrease their confidence.(20) Reliability information, however, is not consistently considered in making judgments.(21) Validity of information appears to affect judgments more than does reliability.(20,22)

Because so much information on reliability and measurement error has been published recently, the goal of this study was to determine whether physical therapists understand and use that information in their decision making. If physical therapists understand reliability and error of measurement, a high percentage of them would be able to define reliability, discuss the relationship between reliability and measurement error, and have a reasonable estimate of the error of measurement of commonly used assessment procedures. Therapists often make decisions to initiate an intervention (eg, to begin weight-bearing gait) by comparing a measurement they take with a predetermined criterion (eg, the amount of quadriceps femoris musde torque necessary for safe weight-bearing gait). If physical therapists understand reliability and error of measurement, they would experience little certainty about initiating the intervention when the observed measurement and the criterion measurement are close together and more certainty when these measurements are farther apart. If therapists did not already have an estimate of the error of measurement of an assessment procedure, the provision of that information would be expected to alert therapists that the difference between the observed and criterion measurements could be measurement error. If the observed and criterion measurements were close together, error could account for the entire difference and confidence might be low. If those measurements were farther apart, the difference might be real and confidence might be high.

I believe that physical therapists make clinical decisions based on many factors, only one of which is the results of measurements. Even if other factors are considered, however, awareness of measurement error may alter confidence in those decisions. If measurement error is ignored and the other factors are used as the only means of decision making, the provision of measurement information should not alter confidence in the decision.

The hypotheses for this study were (1) the mean of physical therapists' estimates of error will differ from the range of error estimates reported in the literature, (2) confidence in judgment will be higher among physical therapists when their decisions are based on observed measurements that are far from a given criterion measurement than when their decisions are based on observed and criterion measurements that are close, and (3) confidence in judgment of physical therapists prompted with reliability or error information will differ from confidence of therapists who are not prompted.

Method

Subjects

Subjects were physical therapists actively employed in physical therapy in some capacity. The APTA generated by computer a nationwide, pseudorandom sample of 1,000 names. The computer began at a random point and generated a list of consecutive names that had been entered in no particular order. A power analysis indicated that a sample of 272 would detect an estimated effect size of 0.5 (group mean difference in standard deviation units) at a power of 0.80 and an alpha level of .05. To allow for unusable or missing data or ineligible subjects and to increase the randomness of the sample, 500 names were randomly selected from the larger sampling frame.

Instrumentation

A survey instrument was used to gather all information. The survey instrument contained two booklets. In one booklet, therapists were asked to provide their definitions of the term "reliability of measurement" and to describe "the relationship between reliability and error of measurement." They were then asked to estimate the error of measurement of two assessment procedures under specific conditions. Because error will vary depending on the conditions under which data are gathered, therapists were requested to estimate the interrater error of two therapists taking goniometric measurements of the same healthy subject using a standardized procedure and the intrarater error of one therapist taking repeated leg-length measurements of a patient using the same technique both times. Therapists were also asked to provide demographic information and information about their education, work responsibilities, and reading in the professional literature.

The second booklet contained one of four versions of two physical therapy problems. Both problems involved situations in which an observed measurement must be compared with a criterion measurement to make a decision. Problems were designed so that this comparison was pivotal to making the decision, although other information contributing to the decision was not excluded. Therapists were asked to make the decision and to rate their confidence in their decision. One problem was an educational problem involving goniometry; the second problem was a clinical problem involving leg-length measurements (Appendix).

Goniometry was selected as one of the assessment procedures because physical therapists are likely to have a reasonable estimate of the error of goniometric measurements based on their experience, reading in professional journals and texts, or classroom instruction. In contrast, estimates of error of leg-length measurements are not as well publicized in the physical therapy literature and are probably not taught in professional education. Physical therapists are not likely to have reasonable estimates of the error associated with this measurement unless they have developed a sense of the error through experience.

In some versions of the problems, estimates of reliability and error of measurement were provided. Thus, one independent variable in the study was the presence or absence of reliability and measurement error information (prompted, unprompted). To approximate the situation in the goniometry problem, the values reported were intertester reliability and error estimates for healthy subjects using a standardized technique. The information was drawn from a study by Boone and associates(23) that is often cited as the source of information about error of goniometry. These investigators reported the standard deviation of the total distribution of measurements of knee range of motion among testers as 5.2 degrees and described this value as the error of measurement. Consequently, many practitioners use 5 degrees as their sole estimate of error for goniometry. The investigators did not report the SEM in their analysis, although it could be calculated (4.17[degrees]) from the reported data.

In the leg-length scenario, a single therapist obtained all measurements, so the values given to subjects were intratester reliability and error estimates for patients with leg-length discrepancies measured with a standardized technique. The information for the problem was drawn from a study by Beattie and associates,(24) who provided raw data from which a Pearson correlation coefficient of .68 could be calculated to represent reliability. The standard deviation of the scores could be calculated from their raw data (12.5 mm) and used to calculate the SEM of 7.1 mm (12.5[square root of [1.00-0.68]]).

The second independent variable was the distance between the observed measurement and the criterion measurement (far, near). In the far condition, the distance was greater than one SEM. In the near condition, the distance was less than one SEM (Figure). For example, in the leg-length problem, the criterion measurement was 15 mm and one SEM was 7 mm. In the far condition, the physical therapist measured 23 mm, so the difference of 8 mm between the observed and criterion measurements was greater than one SEM. In the near condition, the physical therapist measured 17 mm, so the 2-mm difference was within one SEM.

One SEM was chosen because it is frequently the estimate of error reported in the literature. In addition, the 68% confidence interval represented by one SEM would allow an adequate distribution of confidence ratings, whereas a 95% confidence interval would likely produce little variability in responses.

Four versions of each of the two problems were created, representing four combinations of the two independent variables, prompting and distance. Problems were assembled in a booklet containing one goniometry problem and one leg-length problem from the same experimental condition (ie, near, prompted). In half of the booklets, the goniometry problem was presented first; in the other half, the leg-length problem appeared first. Both booklets were submitted to three rounds of pilot testing with three groups of academic and clinical physical therapists who were not involved in the study.

Dependent Variables

Physical therapists' understanding of the concepts of reliability and measurement error was determined from their responses to the questions in the questionnaire. Therapists' responses were coded to reflect three levels of understanding that measurements are imperfect: (1) those that indicated in some way that reliability relates to the reproducibility or consistency of measurements and that the lower the reliability of the measurements, the greater the error; (2) those that had some correct concepts but did not state the specific relationship; and (3) those that were completely incorrect. Therapists' estimates of error of measurement of goniometry and leg-length measurements provided the dependent variables for analyses of the reasonableness or accuracy of their estimates.

Therapists' confidence in their decisions was determined by asking them to make a slash across the line on a 140-mm bipolar visual analogue scale. The center of the scale indicated complete uncertainty or lack of confidence; the right end indicated complete confidence that the student in the goniometry problem should be given extra practice sessions or that the patient in the leg-length problem should be fitted with a heel lift. The left end indicated complete confidence that the student did not need extra practice or that the patient should not be fitted with a heel lift. Therapists were encouraged to use any place on the scale to indicate their degrees of certainty between the extremes.

The distance from the left end of the scale to the therapist's mark was measured (in millimeters) and converted to a value on a scale of 0 to 100 ([distance/140] x 100), with 50 representing no confidence. Confidence expressions of 50 to 0 indicated varying degrees of confidence that the student or the patient in the survey problems did not need the intervention. Confidence expressions of 50 to 100 represented varying degrees of confidence that the student or the patient in the problems needed the intervention.

Using the raw data, differences in confidence could be due to the direction of the decision or to the strength of the therapist's confidence. Because 50 on the scale was "no confidence," it was converted to O. Data were recoded so that responses were assigned values corresponding to the distance from the no-confidence point without regard to direction. The resuiting variable was used as an indication of the strength of the therapist's confidence.

Procedure

The four versions of the problems were randomly assigned and evenly distributed among the therapists using a 2 x 2 factorial design. The survey instruments were sent to the therapists through first-class mail. Unprompted therapists (ie, those not provided with reliability and measurement error information) were asked to complete and seal the problem booklet and then to complete the booklet containing the demographic information and the error estimates. Prompted therapists (ie, those provided with reliability and measurement error information) were asked to complete the booklets in reverse order. This procedure was followed to prevent the act of error estimating from serving as a prompt to unprompted therapists. It also served to protect prompted therapists' estimates from contamination by the error value given in the problems.

Data Analysis

To test the randomization, the four groups were compared on demographic variables using an analysis of variance (ANOVA), the Kruskal-Wallis Test (for nonnormal variables), or a chi-square analysis. Order effects were studied by comparing confidence expressions of therapists who performed the goniometry problem before the leg-length problem with those of therapists who performed the two problems in reverse order using a multivariate analysis of variance. To estimate whether respondents might differ from those who did not respond, confidence expressions of those who responded to the first mailing were compared with those of therapists who responded to a second mailing using an ANOVA, the Mann-Whitney Rank-Sums Test, and a chi-square analysis. The alpha level was .05.

To examine therapists' understanding of measurement concepts, frequencies of therapists in each response category were tabulated. To test whether physical therapists have reasonable estimates of the error of goniometry and leg-length measurements, the means of the therapists' error estimates were compared with the upper and lower boundaries of the ranges of error reported in the literature using single-sample t tests and an alpha level of .01. The boundaries of error for knee goniometry measurements of healthy subjects using a standardized technique were determined to be 1.5 and 5.2 degrees.(23,25,26) The boundaries of error for leg-length measurements of patients with leg-length discrepancies using a standardized technique were determined to be 5.0 and 10.0 mm.(27-29)

To determine whether confidence in decisions was affected by whether physical therapists were prompted with error information or by the distance between observed and criterion measurements, 2 x 2 univariate ANOVAs were used. The alpha level was .05 for the analyses. Data were analyzed using the Statistical Package for the Social Sciences (SPSS)(*) and a Macintosh personal computer.(t)

(*) SPSS Inc, 444 N Michigan Ave, Chicago, IL 60611.

(t) Apple Computer Inc, 20525 Mariani Ave, Cupertino, CA 95104.

Results

The response rate was 62%; consenting, eligible therapists (n=275) represented 55% of the sample contacted. The responding group was predominantly female (78%), was educated in physical therapy at a baccalaureate level (75%), and spent the largest amount of their time in direct patient care (87%). They had a mean of 9.12 (SD=5.97, range=22-64) years of experience and a mean age of 33.75 years (SD--7.38, range= 0.25-35).

Therapists in all four groups were similar on characteristics of sex, age, education, work responsibilities, and reading in the professional literature. Confidence expressions of therapists were similar based on the order of problems Crab. 1). Early and late respondents were similar on demographic variables as well as their confidence ratings. The therapists, therefore, remained pooled within their group assignments for subsequent analyses.

Understanding 01 Concepts of Reliability and Error of Measurement

A total of 63.6% of the therapists (n=175) could define reliability correctly, 23.3% (n=64) had some correct information, 8.4% (n=23) could not define it correctly, and 4.7% (n=13) did not answer the question. In contrast, only 31.3% of the therapists (n=86) could accurately state the relationship between reliability and error of measurement, 41.5% (n=114) had some correct information, 16.7% (n=46) were incorrect, and 10.5% (n=29) did not answer the question. In both problems, the therapists who did not respond to the question may not have had an answer to provide, and their numbers might be added to those who were incorrect.

Estimates of Measurement Error

The mean of the therapists' estimates of error of goniometry was 5.52 degrees, and the mean error estimate for leg length was 5.87 mm. Because the mean estimate for leg-length measurement error fell within the range of errors reported in the literature, no t test was performed. The mean estimate for goniometry error was 0.30 degree above the highest error reported in the literature. This difference was not statistically significant (t=1.78, P [is greater than] .01). Based on the mean error estimates, therapists seemed to have learned the error of measurement of both assessment procedures. Twenty-three percent of the therapists, however, made estimates outside of the published range of estimates of goniometry error.(25,25,26) All but one of these estimates were above the highest published estimate.(23) For leg-length measurement error, about half of the therapists' estimates were outside the ranges of published estimates(27-29); 7.7% of the estimates were above the highest published estimate,(28) and 38.9% of the estimates were below the lowest published estimate.(28)

Therapists could develop an estimate of the amount of error through education, reading, or experience. Three available indicators of learning about measurement error were the number of formal measurement courses the therapists had taken, the scope of their reading about measurement in the professional literature, and their amount of practice experience. To study the relationship between these variables and the error estimates, an "accuracy" variable was created by coding the estimates to reflect whether they were within or outside of the boundaries of the published estimates.

The respondents had very little formal study in measurement. On the average, they had completed about one half of a course in measurement. Based on comments made on the questionnaire, this figure may be an overestimate. Some respondents appeared to have interpreted "measurement courses" as courses in which various tests and measurement skills are taught. The accuracy of therapists who had taken courses in measurement (n=119) was compared with that of therapists who had taken no measurement courses (n=143) using chi-square analysis. There was no difference in accuracy of the error estimate for the goniometry problem ([X.sup.2] = 1.27, P [is greater than] .05) or for the leg-length problem ([X.sup.2] = 0.81, P [is greater than] .05), suggesting that formal study does not affect the accuracy of error estimates.

As a group, the therapists reported that 6.29% of their reading was about measurement. Chi-square analysis was used to compare the accuracy of the therapists who fell at or below the median of 5.00% in their reading of measurement issues (n=164) with the accuracy of those who were above the median (n=97). There was no difference in accuracy for the goniometry problem ([X.sup.2]=2.62, P [is greater than] .05) or the leglength problem ([X.sup.2] =0.45, P [is greater than].05), suggesting that reading about measurement also does not affect the accuracy of estimates of error of measurement.

To explore whether experience might affect the accuracy of error estimates, relatively recent graduates with 2 or fewer years of experience (n=26) were compared with those who had had more than 2 years of experience (n=240). A significant difference existed between the groups for the goniometry problem ([X.sup.2]=8.39, P(.05). No therapist in the group with little experience was outside the published boundaries in estimating goniometry error, but 22.6% of the group with more experience was outside of those boundaries. There was no difference in accuracy in the leg-length problem ([X.sup.2]= 1.21, P>.05).

Strength 01 Confidence

Goniometry problem. Therapists in the near conditions felt that the student in this survey problem should not have extra practice sessions, whereas therapists in the far conditions felt that the student should have extra practice. Therapists in the near conditions felt significantly more certain about their decisions than did those in the far conditions. The therapists who were prompted with the information that a measurement had limited reliability had significantly higher confidence in the decisions they made than did unprompted therapists (Tabs. 2, 3). A Scheffe post hoc analysis showed that the therapists in the prompted, near condition had strength values significantly higher than those of therapists in the other three conditions (F=9.59; df=3,271; P [is not greater than] .005. The interaction between prompting and distance for strength of confidence in the goniometry problem was not significant.

Leg-length problem. Therapists in all conditions felt that the patient in this survey problem should have the heel lift. There was no difference in strength of confidence based on the distance between observed and criterion measurements or on prompting Crabs. 2, 3). The interaction between distance and prompting was not significant.

In either problem, confidence might have been affected by the accuracy of the therapist's estimates of measurement error. There were no differences in strength of confidence for either problem based on whether error estimates resembled those reported in the literature for goniometry (F= 1.91; df=2,264; P [is greater than] .05) or for leg length (F= 1.63; df=2,253; P [is greater than] .05).

Previous studies(12-14) have suggested that more expert practitioners are less confident than those who are less expert. To analyze whether experience affected confidence, the strength of confidence of therapists with 2 or fewer years of practice was compared with the strength of confidence of therapists with more experience. There was no difference in confidence based on experience for either the goniometry problem (F=0.74; df= 1,264; P [ is greater than] .05) or the leg-length problem (F=0.00, df=1,264; P [is greater than] .05).

Other factors that might contribute to expertise, such as formal study, level of education, or type of reading also did not affect strength of confidence. Strength of confidence of therapists with no formal measurement courses was not different from that of therapists who had taken such courses (F=0.45 and 0.76 for goniometry and leg-length problems, respectively; df= 1,261; P [is greater than] .05). Strength of confidence did not differ depending on whether therapists had a baccalaureate degree (n=202), an entry-level master's degree (n=34), a postprofessional master's degree (n=31), or a doctoral degree (n=3) (F= 1.44 and 0.88 for goniometry and leg-length problems, respectively; df=3,266; P [is greater than] .05). Therapists who devoted greater than 5.00% of their reading to measurement topics did not differ in confidence from those whose reading in measurement was less (F=0.17 and 0.10 for goniometry and leg-length problems, respectively; df= 1,260; P [is greater than] .05).

Differences in Performance on the Two Problems

Therapists responded to the two problems differently, as evidenced by the differences in the effects of prompting and distance across the two problems. There was no significant correlation between strength of confidence for the two problems in any of the conditions Crab. 4). The low correlations suggest that physical therapists viewed the two problems as unique and distinct entities. Significant differences between the two problems were apparent when a repeated-measures ANOVA was applied to the strength of confidence scores (F= 15.65; df= 1,270; P [is not greater than] .05). There was also a significant interaction between problem and condition (F=8.78; df=3,270; P [is not greater than] .05). Therapists in the prompted, near condition had the highest confidence in their decisions regarding the goniometry problem and the lowest confidence in their decisions regarding the leglength problem.

Discussion

The results of this study suggest that physical therapists do not alter their confidence in their decisions based on information about reliability and error of their measurements. If therapists understood that the presence of measurement error can make an observed measurement closer to or farther from the true measurement, then in the near conditions one might ex

  Table 4.  Correlations Between
Strength of Confidence in the Goniometry
Problem and Strength of Confidence in
the Leg-Length Problem by Condition
Condition                  r     P
Prompted, near (n=61)     -.05   NS
Prompted, far (n=70)       .13   NS
Unprompted, near (n=67)    .21   NS
Unprompted, far (n=76)     .06   NS
Overall (n=274)            .06   NS

pect great uncertainty. Assuming the therapists in this study did not already have a reasonable estimate of the error, prompting with information about measurement error would be expected to highlight uncertainty and decrease confidence.

For the goniometry problem, therapists in both near conditions appeared to feel strongly that the student's measurement was "close enough" and that no extra practice was necessary. Therapists prompted with information about reliability and error were even more confident than unprompted therapists, as if they had received confirmation that the student's measurement fell within the error band and therefore was indeed close enough. In the leg-length problem, therapists in both near conditions chose to intervene and were not affected by prompting with measurement information. In neither problem did the therapists seem to understand that measurement error could have worked to make the observed measurement falsely close to the criterion measurement.

Confidence would be expected to be higher in the far conditions than in the near conditions. Prompting might be expected to highlight that the observed measurement was really different from the criterion measurement and serve to increase confidence. Such an increase in confidence, however, did not occur in either problem.

The lack of difference in confidence levels within near and far conditions might suggest that the therapists already had reasonable estimates of error. If the therapists had used those estimates in rating their confidence, they would have applied the knowledge whether they were prompted or not, and their confidence levels would not be expected to differ as a result of prompting. The analysis of the therapists' estimates, however, does not support this explanation. Estimates were reasonably accurate for goniometry, but if those estimates had been used in decision making, then confidence in the near conditions would have been approximately equal. In the leg-length problem, nearly half of the therapists did not have a reasonable estimate of the error, so previous knowledge does not explain the result. The therapists reacted to the provision of measurement information in the problem for which they had reasonable estimates of error, and they did not react to the provision of that information in the problem for which they could not provide reasonably accurate estimates. They appear to have used the measurement information only in one problem and to have been somewhat overconfident, although confidence was moderate in both problems.

Moderate confidence was expected and is appropriate because one SEM, representing a 68% confidence interval, was used in constructing the problems. A moderate level of confidence may also reflect the decreased confidence that some investigators have found with increased expertise.(12-14) In this study, there were no differences in confidence based on years of practice, supporting the opposing view that experience does not necessarily decrease confidence.(3,4) Years of practice, however, may not represent expertise because it does not indicate whether therapists have used these measures frequently.

There are several potential reasons for results suggesting that measurement information is not used. Therapists may not use the concepts in decision making because they do not understand them. The high percentage of therapists in this study who were able to define the concept of reliability and the smaller percentage who were able to describe the relationship between reliability and error of measurement suggest that physical therapists understand measurement principles at the lowest two levels of Bloom's taxonomy of the cognitive domain (knowledge and comprehension) (30) Because the information did not affect confidence, they appear not to use the concepts at the higher levels of application, analysis, synthesis, or evaluation.

In addition, this study assessed the therapists' understanding of only one measurement issue--the ability to consider the amount of measurement error in making decisions. Rothstein(1) suggested that experienced practitioners may not always know the reliability coefficients but may nevertheless develop a sense of error through experience. In this study, experienced therapists' estimates of error of goniometry differed from those reported in the literature(23,25-29) more than did those of recently graduated therapists. Recent graduates' estimates would resemble those found in the literature, not as a result of experience, but rather as a result of provision of the information in entry-level curricula. Perhaps the sense of error that experienced practitioners develop is based on an understanding of sources of error and the need to standardize measurement procedures. They may not develop an appreciation of the amount of error associated with their measurements or an understanding of how to respond to that error. Standardized procedures were followed in taking the measurements used in both survey problems in this study. If the therapists thought that standardization controlled the measurement error, their decisions and their confidence in those decisions might not be affected by the amount of measurement error but would be determined by other factors.

Decisions are based on multiple factors. This study was designed to investigate the effect of only one factor. In the leg-length problem, reliability and measurement error information was not used in making the decision, indicating that other factors played a major role. Some therapists indicated that leg-length measurements themselves did not play a role in their clinical decision making for the kind of problem presented. Some therapists questioned the validity of leg-length measurements and suggested they would prefer to use visual assessment of pelvic level as an indicator of leg-length discrepancy, a technique that appears to be less valid and reliable than the tape measure method.(31,32) Several therapists suggested evaluating the spine as a cause of apparent leg shortening, despite the information that the leglength discrepancy followed a femoral fracture. Based on the failure of the therapists to react to the provision of measurement information in the leglength problem and on comments that the leg-length measure itself was not germane, study of the factors that do contribute to these decisions in physical therapy is warranted.

I contend that if physical therapy clinicians are to use measurement information in their decisions more consistently, better education is needed. Curricula should include information on measurement principles, including sources of error and estimates of error, and they should provide exposure to the measurement literature. Even if these issues are formally taught, physical therapists also need strategies for using that information in their decision making. They may understand these principles sufficiently to talk about them, but they may not know how to respond to the information in a practice setting. They may understand that measurement error could contribute to inappropriate decisions but may not perceive that low reliability would affect their certainty about those decisions. In this study, six therapists did mention that they would want to take more measurements and average them to increase their certainty.

I argue that strategies must be taught, reinforced, and practiced.(33) Several strategies are appropriate, especially when an observed measurement is dose to the criterion measurement or when the decision is costly or risky. I believe these strategies should include, when appropriate, averaging several measurements to approximate the true value; using another test to corroborate the results of the test in question; using a more reliable measurement, even if more costly; controlling extraneous variables by standardizing procedures; and examining differing test results to determine how decision making is affected. Measurement information and strategies must be presented at levels of the cognitive domain higher than knowledge and comprehension. Practice in applying measurement information and evaluating decisions based on measurements is indicated.

Differences in Performance on the Two Problems

The different performances on the two problems and the lack of correlation between confidence variables suggest task specificity in physical therapy decisions. Other investigators have found that the ability to make appropriate decisions is case specific in medicine(34-37) and in physical therapy.(38.39) The two problems were constructed to be equivalent in the direction of the decision, the distance between observed and criterion measurements, the moderate nature of the reliability, and the balance of consequences of a decision in either direction. Both problems were constructed so that the decision would tend toward the intervention. The direction of the decision, however, was not the same for the two problems, nor was the magnitude of confidence.

The problems differed on several dimensions. In the goniometry problem, the therapists had little information besides measurement error, so they would be more likely to use measurement error in their decision making. In the leg-length problem, therapists had other information about the patient as well as their own previous experience with similar patients on which to base a decision. The difference in the amount of information may also account for the significantly higher confidence in the goniometry problem ([closure of the set]=28.60, SD= 15.74) compared with the leglength problem ([closure of the set]=24.17, SD= 15.48). To the extent that physical therapists have a high need for structure, they would display high confidence in the problem that was structured and provided limited information on which to base a decision.

The problems referred to two different, but commonly used, physical therapy measures; one problem was a classroom situation, and one was a clinical situation. The classroom situation might have been novel for most of the therapists. Only 5 therapists indicated that academic education was their primary responsibility; 246 therapists were primarily involved in direct patient care. The clinical situation was more like most therapists' usual activities. These therapists might have used measurement information in the classroom situation to help make an unfamiliar decision and relied on habitual patterns of decision making in the clinical situation. Some evidence suggests that clinicians tend to make decisions based on experience with similar patients; they resort to scientific reasoning when the problem that faces them is novel.(40)

The problems may have been different in other ways. The problems were designed to have balanced consequences of the decision in either direction. Subjects may not have seen the consequences as balanced, and their decisions may reflect a belief that the consequences in one direction were more serious than the consequences in the other direction.

The decisions themselves were different, and either decision may have been viewed with more or less concern. Consequences of the goniometry decision may have been perceived as less important than those of the leg-length decision; that is, the decision of whether to provide a heel lift for a patient may have been perceived as more serious than the decision of whether a student should have extra practice. Further study on the nature of consequences is indicated. Either of these problems could be examined from the standpoint of severity of the outcomes, ranging from trivial to severe. Perhaps a more severe outcome triggers a greater fear of being wrong and lowers early confidence.(5)That fear may have been triggered even in these problems. Strength of confidence in the leg-length problem, which could be viewed as having the more serious outcomes, was significantly lower than strength of confidence in the goniometry problem.

The results of this study suggest that the problems were viewed as different types of problems rather than two instances of the same type of problem-that of comparing a measurement with a criterion figure to determine a course of action. In the leglength problem, the decision was based on what action to take if the observed measurement fell above or below a stated criterion. In the goniometry problem, the action depended, not on whether the observed measurement was above or below the criterion measurement, but rather on the magnitude of the difference between them. In the goniometry problem, the proximity of the observed measurement to the criterion measurement in the near conditions may have been substantially less than some threshold for action, and thus confidence was high. In the far conditions, the measurement may have been just over the therapists' personal threshold for action. Thus, the direction of the decision changed, and the confidence in the decision was lower. Further study on the amplitude and mechanism of action thresholds is warranted.

The task specificity in this study suggests that therapists use different decision processes with different decision types. Previous studies of decision making have focused on a single decision or a single decision type repeated several times. Results from such studies regarding decision-making processes cannot be generalized to different decision types. This lack of generality should stimulate further exploration of the structure of decision problems and the impact of structure on decision-making processes.

Conclusions

The results of this study suggest that physical therapists have a fundamental understanding of the concepts of reliability and measurement error. Their understanding of error of measurement was not affected by formal study, reading about measurement, or experience. Physical therapists seem to have reasonable estimates of error of measurement only for some assessment procedures, perhaps from the inclusion of selected measurement information in entry-level education.

Under the conditions in this study, physical therapists do not seem to have the level of understanding of the role of error that would be reflected in appropriate adjustments of their confidence. They appear to be making decisions in response to factors other than measurements. The use of reliability and error information was not affected by experience, type of reading, amount of formal study, or degree earned.

Therapists responded to the stimulus problems as distinctly different types of decision problems. The task specificity in this study suggests the process of decision making cannot be generalized to all decision types. Further research is warranted in several areas of decision making, including the nature of decision problems, their structure and consequences, and individual thresholds for action.

If physical therapists are to develop an appropriate understanding of the meaning of measurement error, this information and strategies of how to use it in decision making must be included in physical therapy curricula. Evaluation of different methods of teaching this material is necessary, as is the continued monitoring of the impact on future practice of the inclusion of measurement principles in physical therapy curricula.

Acknowledgments

I would like to thank the following individuals: Michael M Ravitch, PhD, Judith Falconer, PhD, and Michael Garet, PhD, for their assistance in preparing and reporting this research; Dale Schuit, PhD, PT, Jean Rogers, PT, and Kathy Hall, PT, for their assistance with creating the vignettes; Babette Sanders, PT, for her helpful comments on an earlier version of the manuscript; and the physical therapists who shared their decisions.

KW Hayes, PhD, PT, is Assistant Professor and Curriculum Coordinator, Programs in Physical Therapy, Northwestern University Medical School, 345 E Superior St, Room 1323, Chicago, IL 60611 (USA).

This study was completed in partial fulfillment of the requirements for Dr Hayes's Doctor of Philosophy degree at Northwestern University.

The study was supported by a grant from the Foundation for Physical Therapy Inc.

This study was exempted from human subjects review by the Institutional Review Board of Northwestern University.

This article was submitted July 24, 1991, and was accepted February 24, 1992.

References

1 Rothstein JM. On defining subjective and objective measurements. Phys Ther. 1989;69: 577-579.

2 Fischhoff B, Slovic P, Lichtenstein S. Knowing with certainty: the appropriateness of extreme confidence. J Exp Psychol [Hum Percept and Performance]. 1977;3:552-564.

3 Lichtenstein S, Fischhoff B. Do those who know more also know more about how much they know? Organizational Behavior and Human Performance. 1977;20:159-183.

4 Oskamp S. Overconfidence in case-study judgments. J Consult Psychol. 1965;29:261-265.

5 Maysdess O, Kruglanski AW. What makes you so sure? Effects of epistemic motivations on judgmental confidence. Organizational Behavior and Human Decision Processes. 1987;39:162-183.

6 Koriat A, Lichtenstein S, Fischhoff B. Reasons for confidence. J Exp Psychol [Hum Learn Mem]. 1980;6:107-118.

7 Payton OD, Heuter AE, McDonald ME. Learning style preferences: physical therapy students in the United States. Phys Ther. 1979; 59:147-152.

8 Rezler AG, French RM. Personality types and learning preferences of students in six allied health professions. J Allied Health. 1975;4:2026.

9 Hightower AB. Continuing education in physical therapy. Phys Ther. 1973;53:16-24.

10 Bohannon RW. Objective measures. Phys Ther. 1989;69:590-593.

11 Baliin AJ, Breslin WH, Wierenga KAS, Shepard KF. Research in physical therapy: philosophy, barriers to involvement, and use among California physical therapists. Phys Ther. 1980; 60:88-895.

12 Goldberg LR. The effectiveness of clinicians' judgments: the diagnosis of organic brain damage from the Bender-Gestalt test. J Consult Phychol 1959;23:25-33.

13 Holsopple JQ, Phelan JG. The skills of clinicians in analysis of projective tests. J Clin Psychol. 1954;10:307-320.

14 Oskamp S. The relationship of clinical experience and training methods to several criteria of clinical prediction. Psychol Monogr: General and Applied. 1962;76(Whole No. 547): 1-25.

15 Beach LR, Mitchell TR, Deaton MD, et al. Information relevance, content and source credibility in the revision of opinions. Organizational Behavior and Human Performance. 1978;21:1-16.

16 Johnson EM, Cavanagh RC, Spooner RL, Samet MG. Utilization of reliability measurements in Bayesian inference: models and human performance. IEEE Transactions on Reliability. 1973;R-22:176-182.

17 Schum DA, DuCharme WM. Comments on the relationship between the impact and the reliability of evidence. Organizational Behavior and Human Performance. 1971;6:111-131.

18 Schum DA, Pfeiffer PE. Observer reliability and human inference. IEEE Transactions on Reliability. 1973;R-22:170-176.

19 Snapper KJ, Fryback DG. Inferences based on unreliable reports. J Exp Psychol 1971;87: 401-404.

20 Slovic P. Cue consistency and cue utilization in judgment. Am J Psycbol 1966;79: 427-434.

21. Borko H, Shavelson RJ. Teachers' sensitivity to the reliability of information in making causal attributions in an achievement situation. J Educ Psychol. 1978;70:271-279.

22 York KM, Doherty ME, Kamouri J. The influence of cue unreliability on judgment in a multiple cue probability learning task. Organizational Behavior and Human Decision Processes. 1987;39:303-317.

23 Boone DC, Azen SP, Lin C, et al. Reliability of goniometric measurements. Phys Ther. 1978; 58: 1355-1360.

24 Beattie P, Isaacson K, Riddle DL, Rothstein JM. Validity of derived measurements of leglength differences obtained by use of a tape measure. Phys Ther. 1990;70:150-157.

25 Rheault W, Miller M, Nothangel P, et al. Intertester reliability and concurrent validity of fluid-based and universal goniometers for aCtive knee flexion. Phys Ther. 1988;68: 1676-1678.

26 Gogia PP, Braatz JH, Rose SJ, Norton BJ. Reliability and validity of goniometric measurements of the knee. Phys Ther. 1987;67: 192-195.

27 Eichler J. Methodological errors in documenting leg length and leg length discrepancies. In: Hungerford DS, ed. Leg length Discrepancy: The Injured Knee. Berlin, Federal Republic of Germany: Springer-Verlag; 1977;1: 29-39.

28 Morscher E, Figner G. Measurement of leg length. In= Hungerford DS, ed. Leg Length Discrepancy.. The Injured Knee. Berlin, Federal Republic of Germany: Springer-Verlag; 1977;1: 21-27.

29 Friberg L, Nurminen M, Korhonen K, et al. Accuracy and precision of clinical estimation of leg length inequality and lumbar scoliosis: comparison of clinical and radiological measurements. Int Disabil Dud. 1988;10:49-53.

30 Bloom BS, ed. Taxonomy of Educational Objectives, Handbook L. Cognitive Domain. New York, NY: David McKay Co; 1956.

31 Clarke GR. Unequal leg length: an accurate method of detection and some clinical results. Rheumatology and Physical Medicine. 1972;11: 385-390.

32 Mann M, Glasheen-Wray M, Nyberg R. Therapist agreement for palpation and observation of iliac crest heights. Phys Ther. 1984;64: 334-338.

33 Brown AL, Campione JC, Day JD. Learning to learn: on training students to learn from texts. Educational Researcher. 1981;10(2):1421.

34 Elstein AS, Shulman LS, Sprafka SA. Medical Problem Solving: An Analysis of Clinical Reasoning. Cambridge, Mass: Harvard University Press; 1978.

35 Melnick DE. Computer-based clinical simulation: state of the art. Evaluation and the Health Professions. 1990;13:104-120.

36 Norcini JJ, Swanson DB, Grosso LJ, Webster GD. Reliability, validity and efficiency of multiple choice question and patient management problem item formats in assessment of clinical competence. MedEduc. 1985;19:238-247.

37 Skakun EN. The dimensionality of linear patient management problems. In: Proceedings of the Seventeenth Annual Conference on Research in Medical Education. Washington, DC: American Association of Medical Colleges; 1978; 17:63-57.

38 Hayes KW, Rogers J, Sullivan JE, Huber G. Computer-based patient management problems in an entry level physical therapy program: acceptance and cost. Journal of Physical Therapy Education. 1991;5:65-71.

39 Stratford P, Pierce-Fenn H. Modified essay question. Phys Ther. 1985;65: 1075-1079.

40 Schmidt HG, Norman GR, Boshuizen HPA A cognitive perspective on medical expertise: theory and implications. Academic Medicine. 1990;65:611-621.

Appendix. Examples of Physical Therapy Problems in the Survey Instrument (a)

Problem One (Near, Prompted Version)

You are an instructor in an entry-level physical therapy program, teaching a competency based unit on goniometry. In preparation for a practical examination, you prepare a laboratory activity to provide students with feedback about their skill with goniometry. If students do not perform well in a practice measurement, they must attend two supervised practice sessions during the next week. If they do perform well, they do not need to attend the extra sessions.

For the laboratory, you fit a healthy male subject with a full length lower extremity splint. The knee joint on this splint can be adjusted to specific ranges of motion and has been carefully calibrated. You position your subject supine on a plinth and set the range of allowable knee motion at 10-130[degrees]. You then ask one of your students to measure the subject's active range of motion. The student uses a large, clear plastic universal goniometer and the standard technique and measures the knee at 7-130[degrees]. Past studies have shown that for normal males, active knee goniometric measurements by two different testers have a reliability of about 0.50 and a standard error of measurement of about 4.17[degrees].

What degree of certainty do you have about the student's need for extra practice in measuring joint range?

Problem Two (Near, Prompted Version)

Your patient is a 25 year old salesman who had a femoral fracture two years ago. He is having some pain in his back and hip and has a slight functional scoliosis which is fairly well compensated. He appears to have a leg length discrepancy. You know that research shows that patients like this one usually benefit from an external heel lift with a compensatory forefoot lift if the discrepancy is 15 mm or greater. The lift should help decrease his pain and prevent disk degeneration in the lumbar spine from the compressive forces generated by the scoliosis. On the other hand, the addition of even a small lift can impair foot mechanics and alter his gait. It may also increase vertical forces through the lower extremity and increase the risk of osteoarthritis of the hip. In addition, this patient is concerned that his expensive shoes will look unattractive. Together you and the patient decide to begin gradual correction of the discrepancy only if his measurement really exceeds the 15 mm criterion.

You use the standard procedure to measure this patient. You place him supine on a firm surface with his pelvis level and place his lower extremities in a neutral position. You measure the leg length using the direct method, measuring from the origin of the sartorius on the anterior superior iliac spine to the medial malleolus. Your plastic covered cloth tape measure is pulled taut so there is no slack. You measure his leg length discrepancy as 17 mm. In past studies, the reliability of tape measurements of leg length discrepancy has been shown to be about 0.68 and the error of measurement has been shown to be about 7 mm.

Should you fit this patient with the lift? What degree of certainty do you have about your decision?

(a) These examples are presented exactly as they appear in the survey instrument booklet.

Commentaries

Following are two commentaries on "The Effect of Awareness of Measurement Error on Physical Therapists' Confidence in Their Decisions."

Physical therapists with little or no formal training in cognitive social psychology, judgment analysis, decision theory, or measurement theory might find this article difficult to analyze critically for at least two reasons. First, therapists without such a background may have little basis for deciding whether the line of reasoning that led to the posing of the specific hypotheses stated in the report is reasonable. Second, because some of the terms used in this article either are not commonly used by physical therapists or are used with less specificity by physical therapists than in the context of psychological theory, misinterpretation is possible. Nonetheless, the article raises several issues that are important for clinicians to consider. The intent of the following is to comment on issues that may attract the attention of experts trained in other disciplines.

One of the most unique aspects of this article is that it presents a very interesting union of concepts from two somewhat distinct areas in the literature: (1) measurement theory and (2) judgment and decision making. As noted by the author, there have been many articles on the reliability of a variety of tests, but little has been done to examine the impact of the growing reliability literature on physical therapy practice. Thus, an important contribution of this report is that it causes us to shift our focus from reliability "for the sake of reliability" to reliability as "a factor that should affect clinical judgments."As demonstrated in the clinical judgment literature, clinicians may have difficulty producing accurate clinical judgments when they are required to combine numerous items of information. If the individual items are erroneous, the probability of producing accurate judgments will be even lower than if the items were correct. Thus, the issues addressed in this study are very important for clinicians.

At first glance, the research design for the study reported seems reasonable for addressing the stated hypotheses. On closer inspection, however, several factors seem to raise doubt regarding the main conclusions. One of the most important statements in the literature review is the following: "If physical therapists perceive error information as disconfirming evidence, then their certainty would be expected to decrease when measurement error information is provided." This statement is pivotal because it explicates the logic underlying the author's second and third hypotheses. Implicit in this logic is the assumption that if the therapists understood the concepts of reliability and measurement error, then they would interpret the information provided in the near conditions as disconfirming evidence. In the goniometry problem, the therapists were accurate in their judgments, but their confidence ratings were not consistent with the stated hypotheses. Given this combination of results, the relevant question may be whether the information provided in the near conditions actually was disconfirming, not whether the therapists applied the information correctly. The essential question for this scenario was whether the student's measurement was accurate. The evidence that the student's measurement was well within the stated criteria does not appear to be disconfirming. In the leglength problem, the therapists responded inaccurately for the near condition, even though the evidence appeared to be discomfirming relative to the question of whether the difference between the criterion and the observed measurement is sufficient to be considered a "real" difference that should warrant intervention. As noted by the author, however, the therapists' responses may have been contaminated by previous experience and based on information that was not stated in the scenario. To the extent that this interpretation is correct, the second scenario exemplifies two related general issues: (1) the potential for problems associated with using ecologically valid scenarios and (2) the effects of experience on clinical judgment.

In general, the use of ecologically valid materials (ie, those that bear a close resemblance to "real-world" situations) enhances generalizability of the results to actual problems. It can be difficult, however, to construct materials that are both ecologically valid and "uncontaminated" by experience. There is increasing evidence in both the cognitive psychology and the clinical judgment literature that differences between experts and novices may be related to differences in content knowledge rather than to differences in processing strategies. In some of the most recent studies, considerable attention has been devoted to examining the enduring effects on memory of specific experiences. One study(1) in particular demonstrated that subjects have a tendency to categorize new examples based on similarity to previously seen examples, even though in so doing they may be contradicting a rule they have been given. The relevance of Allen and Brooks' results to the present study is that, especially in the leg-length scenario, therapists may have been responding on the basis of similarity to previously encountered patients and not to the material presented in the scenario.

In addition to the two general issues just noted, consider carefully the author's excellent comments in the "Discussion" section regarding the differences between the scenarios. As implied by her comments, there are many plausible alternative explanations for the findings, given the differences in response to the two scenarios. Additional studies, therefore, are needed to strengthen the evidence.

Finally, it may be wise to exercise caution in interpreting the results reported in relation to the indicators of learning about measurement error. None of the three variables cited-- number of formal measurement courses taken, therapists' percentage of reading about measurement, and amount of practice experience--were manipulated by the investigator. Although it is not inherently wrong to collect data on variables not controlled by the investigator, problems can arise depending on how the data are used. For example, the basis for forming comparison groups on the therapists' percentage of reading about measurement was a median split. The original distribution was somewhat skewed, and the size of the resulting groups was quite disparate. These factors could have a noticeable effect on the statistical significance. In addition, the fact that both the mean and the median were so low suggests that few, if any, of the therapists did enough reading on the subject to make any difference. Thus, the conclusion that "reading about measurement also does not affect the accuracy of estimates of error of measurement" should be held tentatively, at best. Similar points could be made about the other two measures. Admittedly, the author did not state these conclusions strongly, but a note of caution seems warranted. The author advocates better education as a solution to the problems revealed by this study. If formal study and reading do not affect accuracy, why would doing more be better unless not enough was done to make a difference?

In summary, the primary purpose of my comments has been to alert you to issues that may be cause for caution in interpretation and application of the results of this study. The author is to be commended for bringing these important issues to the physical therapy literature. I look forward to seeing the results of subsequent studies.

Barbara J Norton, PT Instructor

Program in Physical Therapy

Washington University School of Medicine

660 S Euclid Ave

St Louis, MO 63110

References

1 Allen SW, Brooks LR. Specializing the operation of any explicit rule. J Exp Psychol [Gen]. 1991;120:3-19.

Congratulations, Dr Hayes, on attacking a very important and difficult subject. I can think of no similar article in the physical therapy literature concerning how a physical therapist might deal with the possibility of measurement error in the clinical environment.

In the literature review of the article, the author cites two references stating that physical therapists have concrete cognitive styles and that this may indicate a need for structure.(1,2) Whether this assumption is still true is questionable. The article by Payton et al(1) was published in 1979; the article by Rezler and French(2) was published in 1975. It seems fairly obvious that students and physical therapists have both changed in the intervening time period, and this assumed need for structure may no longer be true. The same argument could be made concerning the references about how physical therapists use journals.(3,4) The author, using information from an article by Hightower,(3) states that physical therapists have not used journals preferentially for professional development, and she refers to an article by Bohannon(4) to support this statement. Bohannon was referenced for remarks concerning data collected and reported in almost anecdotal fashion. The physical therapy profession continues to change, and the author may be making assumptions about clinicians' behavior that may no longer be true.

In the section of the article in which the author describes the instrument that was used to gather data, she states that in one booklet subjects were asked to provide definitions of the term "reliability of measurement" and then to describe the relationship between reliability and error of measurement. I find the phrase "error of measurement" to be an awkward one. I proceeded to look through several textbooks related to measurement, and I could not find any that used this phrase in the same way. I am not sure whether error of measurement is the same in the author's mind as measurement error. Measurement error is discussed in texts about measurement.

The author chooses two measurement situations-one involving goniometry and the other involving leg-length measurements. She states that estimates of error of leg-length measurements are not as well publicized in the physical therapy literature and are probably not taught in professional education. There have been recent reports about this topic, however, in our professional literature.(5,6) As an individual who teaches a clinical course concerning examination, I discuss this topic with students, and, as a clinician, I have discussed this topic with other clinicians, particularly when attempting to determine at what point a discrepancy regarding leg length would exist.

In examining the author's results, I think the first and most difficult task in the study was that of grading of correct responses about who could define reliability correctly, who had some correct information, and who could not define it accurately. Approximately 64% of the subjects could define reliability correctly in terms of satisfying the author. The fact that only half this number could accurately state the relationship between reliability and error of measurement makes me wonder whether the concept of the error of measurement was misunderstood, especially considering that 3,0% of the respondents did not answer the question. The author discusses the fact that the subjects reported very little formal study of measurement, stating that on the average, subjects had about one half of a course in measurement.

In my experience, very few courses in physical therapy use the term "measurement" as part of the title of a course. My guess is that most students learn measurement concepts indirectly, either from an instructor who is concerned about the problems of measurement in a clinical environment or from courses that are concerned with research in which the concepts of reliability and validity are presented in terms of the importance of measurement in research. It may be that very few individuals understand the relationship of measurement in research and measurement in clinical practice. The fault for this is not within the individuals receiving a physical therapy education, but rather in the faculty in not making a direct connection for students concerning the importance of measurement, not just in research but also in clinical practice.

The author makes the comment that, based on mean error estimates, the subjects of the study seem to have learned the error of both assessment procedures. She goes on to state that many times the estimates were outside of the published range of estimates of error in both goniometry and leg-length measurements. This would seem to support Rothstein's statement that expert practitioners develop a sense of error from experience.(7) The mean number of years of experience of subjects in this study was just over 9 years. This may not qualify them for the status of being expert practitioners, but it seems to indicate that many have had significant clinical experience and should have some sense of the error associated with certain measures that are commonly used.

The fact that the group with the least experience was less likely to be outside the published boundaries for estimating goniometry error than the group with more experience may represent the effect experience has on interpreting a measurement. The experienced practitioner may assume that the variability in the clinical environment of goniometric measurements is greater than that of the published values. The fact that there was no difference between the two groups in leg-length discrepancies may indicate that there is less variability from clinician to clinician in leg-length measurement than there is in goniometric measurement.

Previous studies cited by the author state that the expert practitioner has less confidence in judgments about measurement than do practitioners who are less experienced. The author examined this problem in the study and found no difference between those with 2 or fewer years of practice compared with the rest of the group for either the goniometry or the leg-length problem. This is an interesting situation than can be interpreted in more than one way. One interpretation is that the two clinical tests that were chosen for study represent some of the least complicated decisions that clinicians have to make on whether to accept or not accept the amount of error associated with these measurements and to the clinical decisions based on them. Being experienced may have very little influence on how a clinician will respond to these particular tests. This interpretation is further strengthened by the author's conclusions that advanced degrees, formal course work in measurement, and other factors examined also did not affect the similarities of the two groups on the strength-ofconfidence issue.

In the "Discussion" section, the author suggests that physical therapists do not alter their confidence in their decisions based on information about reliability and error of their measurements. The author assumed that having information about measurement error would increase uncertainty and decrease confidence in decisions based on measurement information. As suggested earlier, this conclusion may not be completely true. It may simply reflect the fact that physical therapists in clinical practice have a reality-based notion about how significant error must be, particularly if there is no significant patient consequence associated with errors of measurement that would be damaging to the patient in accepting something less than perfect measurement.

The author cites literature that deals with the conflict between studies that have shown decreased confidence with increased expertise and the conflicting view that there is no change in confidence based on experience. Nearly all of the citations, however, are from the psychology literature, which raises the issue of whether physical therapists approach clinical practice in some significantly different way than a psychologist may. I think the author makes the correct conclusion when she states that years of practice may not represent expertise, because it does not indicate whether subjects have used a particular measurement frequently in their clinical practice.

The author's conclusion that physical therapists understand measurement principles only at the lowest levels of Bloom's taxonomy of the cognitive domain could be challenged on the basis that clinicians are using the higher levels of the domain in application, analysis, and synthesis and that this is demonstrated by a more realistic appraisal of the importance of these particular measurement errors in approaching patients in theft practice.

The author appears to assume that because research reports have been published that have described the errors of measurement in these two clinical conditions, this information is directly applicable for use by the clinician. Clinicians, however, may be saying that this is not so.

The author points out that recent graduates' estimates of goniometry error resemble those found in the literature and that perhaps this was because of the introduction of this information into the entry-level curriculum. She may be correct in this assumption. Considering that the only textbook concerning measurement in physical therapy was not published until 1985 and that the focus on measurement issues has been relatively recent, it can be assumed that graduates of physical therapy education programs, until recently, may not have been exposed to much discussion about measurement issues and their importance.(8) It is difficult to estimate the importance of measurement issues to practicing physical therapists even though our profession has been one that has taken the lead in publishing standards for measurement that not only cover standards for test users (clinicians), but also provide information for those who are concerned with teaching tests to others, whether in a classroom or continuing education environment.(9) Except for the field of psychology, I am not aware of any other health profession that has published a similar document.

An interesting comment in the "Discussion" section was that certain physical therapists preferred to use visual inspection at the pelvic level as an indicator of leg-length discrepancy and that some clinicians focused on causes other than the apparent cause of the leg-length discrepancy.

Perhaps this points out the importance of the "teacher" in what both clinicans and students learn and apply. The teacher, in this case, is an influential individual who has discussed the significance of certain measurements during the process of examination. This is a characteristic of not only physical therapy, but also other health professions in which someone of superior experience and prestige makes a comment about a particular clinical examination or measurement technique that is then adopted by the learner without further clinical appraisal.

The author states that the solution to better and more consistent decisions by physical therapy clinicians is better education. Obviously, this is a very important factor. She goes on to state that even if measurement principles and exposure to the measurement literature are taught, physical therapists still need to learn strategies for using this information in their decision making. This may be a more difficult problem than the problem of teaching about measurement. The author has rendered an important service in making the connection between measurement information and clinical decision making, which reemphasizes what we already knew but are likely to forget. Decisions in clinical practice are based on observations about the patient that usually include several measurements, many of which have not been subjected to a great deal of thought about reliability. It has only been recently that commonly used clinical tests have begun to be subjected to analysis concerning both the validity and the reliability of measurement.

The author states that the way the two problems were approached by the clinicians in this study suggests task specificity in physical therapy decisions. One of the problems used in this study related to what the author believes is a classroom rather than a clinical situation. Because clinical education is a common expectation for a practicing therapist, however, it seems reasonable to expect that a physical therapist should be able to examine the problem as one having clinical consequences.

The author feels that this study suggests that subjects use different decision processes with different decision problems and that the task specificity found suggested this. If this is so, this increases the importance of students learning clinical decision making as a method of thinking rather than something that is case specific. Examining a clinical problem requires a thought process that is sequential and logical. Ultimately, this is more important than having seen a great number of similar cases.

This research should be considered of importance to both physical therapists in clinical practice and those who are engaged in preparing students to become future practitioners. The author has provided us with a method of study of measurement problems and has pointed out the importance of questions of measurement and the relationships of measurement ideas to clinical decision making. This study should continue the dialogue about measurement issues and should open new doors for examination of these questions. For these reasons, the author should be commended for conducting a study in such a difficult area and for raising our awareness of the importance of these issues.

John L Echternach, EdD, PT Eminent Professor

School of Community Health Professions and Physical Therapy

Old Dominion University

Norfolk, VA 23529

References

1 Payton OD, Heuter AE, McDonald ME. Learning style preferences: physical therapy students in the United States. Phys Ther. 1979; 59:147-152.

2 Rezler AG, French RM. Personality types and learning preferences of students in six allied health professions. J Allied Health. 1975;4:2026.

3 Hightower AB. Continuing education in physical therapy. Phys Ther. 1973;53:16-24.

4 Bohannon RW. Objective measures. Phys Ther. 1989;69:590-593.

5 Beattie P, Isaacson K, Riddle DL, Rothstein JM. Validity of derived measurements of leglength differences obtained by use of a tape measure. Phys Ther. 1990;70:150-157.

6 Hoyle DA, Litour M, Bohannon RW. Intraexaminer, interexaminer and interdevice comparability of leg-length measurements obtained with measuring tape and Metrecom. Journal of Orthopaedic and Sports Physical Therapy. 1991;14:263-267.

7 Rothstein JM. On defining subjective and objective measurements. Phys Ther. 1989;69: 577-579.

8 Rothstein JM. Measurement in Physical Therapy. New York, NY: Churchill Livingstone Inc; 1985.

9 Task Force on Standards and Measurements in Physical Therapy. Standards for tests and measurements in physical therapy. Phys Ther. 1991;71:589422.

Author Response

I appreciate the thoughtful responses to my article submitted by Ms Norton and Dr Echternach. Both commentators echo my own concerns about the expanding literature on the reliability of data from our various measures. Too often, reports about measurement characteristics are published without discussion of implications for the clinician. They do not answer the question of how such information may appropriately be used in making clinical decisions.

Both commentators observed that the terminology used in this study may be unfamiliar to physical therapists. I believe this lack of familiarity is because our profession has adopted measurement theory from the social sciences, and physical therapists have not been routinely exposed to these measurement concepts. As Dr Echternach tightly observed, the first text about measurement in physical therapy was not available until 1985.(1)It is possible that subjects had difficulty describing the relationship between reliability and error because they did not understand the terminology; consequently, I allowed a wide range of responses in scoring. I believe, however, that these concepts are fundamental and that therapists should be able to define them if they have been exposed to them.

I applaud Dr Echternach and other individuals who include information about error associated with goniometry, leg-length measurements, and our other measurements. Regarding his contention that leg-length measurement characteristics have been well publicized, I agree. I argued, however, that error associated with goniometry was better publicized in the physical therapy literature than that associated with leg-length measurements. At the time this study was designed, I examined more than 20 articles on measurement characteristics of goniometry, over half of which appeared in Pjysical Therapy. Several of the other reports were in journals targeted at a physical therapy audience. In contrast, I found about 10 articles examining reliability and validity of leg-length measurements, only 4 of which were in journals that would reach a physical therapy readership. One of the 2 artides to which Dr Echternach refers was not available at the time this study was conducted.

Dr Echternach commented about the dated references supporting a concrete cognitive style of physical therapists and the lack of use of published research by physical therapists. I share his concern, but based on my 20 years of teaching in both entry-level and post professional physical therapy programs, I believe that physical therapists have not changed substantially. More recently, students in an entrylevel postbaccalaureate physical therapy program demonstrated preferences for organization and detail on the Canfield Inventory,(2,3) and experienced practitioners demonstrated a preference for a systematic style of information processing.(4) More informary, we recently administered a learning preference instrument(5) to our students and found that they prefer highly structured activities. I am aware of only one recent study, conducted in Israel, in which physical therapy students and practitioners preferred abstract conceptualization.(6) Unfortunately, I have seen no new data regarding physical therapists' use of the research literature except Bohannon's(7) remarks allegedly drawn from his 1988 doctoral dissertation. I certainly hope that this assumption is no longer true.

Ms Norton underscores an important issue when she states that therapists in this study did not perceive the provision of error information in the near conditions as disconfirming evidence. As I observed in the "Discussion" section, therapists perceived the information as confirming that the criterion measurement was within the error band of the observed measurement, and therefore no intervention was needed. Ms Norton also asserts that the therapists in the near conditions were accurate in their judgments regarding the goniometry problem and inaccurate regarding the leg-length problem, reasoning that if the two measurements are within the error band, no action is necessary. Such reasoning is intuitively appealing, but it assumes that measurement error operates only in one direction. It assumes that random influences caused the student's measurement to deviate from the criterion measurement. As a result of this assumption, extra practice for the student would not be necessary. Measurement error, however, operates in both directions. Those same random influences could have acted to help the student, making his measurement appear closer to the criterion measurement than it actually was. In the scenario, the student's observed measurement of knee extension was 7 degrees, but his true measurement could have been between 3 and 11 degrees. He could have been exactly right or as much as 7 degrees lower than the criterion measurement, making extra practice sessions seem more appropriate. Similarly, in the leg-length problem, the therapist's true measurement could be as much as 5 mm below the criterion, making a heel lift appear unnecessary, or as much as 9 mm above the criterion, making intervention seem more appropriate. In both problems, the possibility that error could move the decision in either direction should cause the decision maker to feel very uncertain about the need for an intervention, unless, as Dr Echternach suggests, therapists felt that the consequences were not substantial enough to cause concern. Unfortunately, we do not have sufficient understanding of the meaning of measurement error to judge the "right" way to make a decision when error is present. In this study, I was less concerned about the accuracy of the decision therapists made than how strongly they felt about it.

Both Dr Echternach and Ms Norton extended my brief discussion of the role of experience in making decisions. I agree that the therapists may have been influenced by their experience, especially in the more clinical problem. Although not a primary focus of this study, experience may have influenced the therapists' responses in several ways, as the commentators suggest. Dr Echternach proposes an interesting additional interpretation for the more experienced therapists estimating errors outside of the published range of error estimates. It is reasonable to surmise that experienced therapists believe that the variability of measurement data is greater than that published in reports of studies conducted under controlled conditions. It remains reasonable to hypothesize that less experienced therapists reproduced information given them in their educational programs, but his interpretation extends the meaning of the hypothesis to suggest that inexperienced therapists had not yet developed an appreciation for the variability of measurements under differing conditions. More investigations of the role of experience in decision making in physical therapy would be welcome.

I concur with Ms Norton's point that the post hoc analyses exploring the impact of the indicators of learning about measurement error should be considered as only suggestive. It is difficult to draw meaningful conclusions when comparing a group having a small amount of an attribute with a group having an even smaller amount. I explored these subgroups in a variety of ways, including comparing extremes (eg, no reading versus [ is greater than] 20% of reading). Regardless of how the groups were created, reading about measurement and formal study did not affect the accuracy of therapists' estimates of measurement error or their confidence.

I share Dr Echternach's concern that few people understand the importance of measurement in both clinical practice and research, and I agree with him that helping to make this connection is the responsibility of faculty members in entry-level programs. Although my experience regarding course titles is at variance with that of Dr Echternach, I concur that students probably learn measurement concepts, not in courses entitled "Tests and Measurements" or "Assessment," but indirectly or in courses with titles such as "Research." To address this issue, rather than recommending more education, as Ms Norton suggests, I recommend better education. I believe that we must stress measurement principles in both research and assessment courses if students are to become aware that informed decisions in both research and clinical practice depend on high-quality data. We must also discuss strategies of how to use measurement information. What does one do with information that data have low reliability? Do we disregard the measurement? Do we take it again? Do we change tests? Answers will differ depending on the particular decision to be made, but these are the kind of questions that must be answered for information on reliability to become meaningful.

Karen W Hayes, PhD, PT

References

l Rothstein JM. Measurement in Physical Therapy. New York, NY: Churchill Livingstone Inc; 1985.

2 Griffin DA, Thomas K, Pinkston D, et al. Instructional and learning style preferences of physical therapy faculty and students. Phys Ther. 1988;68:832. Abstract.

3 Leglet A, Pinkston D, Jensen G, Mikan K. A study of learning style preferences of postbaccalaureate degree students. Phys.Ther. 1989;69: 368-369. Abstract.

4 May BJ, Dennis JK. Expert decision making in physical therapy: a survey of practitioners. Phys Ther. 1991;71:190-202.

5 Rezler AG, Rezmovic V. The learning preference inventory. J Allied Health. 1981;10:28-34.

6 Katz N, Heimann N. Learning styles of students and practitioners in five health professions. The Occupational Therapy Journal of Research. 1991;11:238--244.

7 Bohannon RW. Objective measures. Phys Ther. 1989;69:590-593.