Living with error.

Even as we mourn the tragic end of the shuttle Columbia, we can marvel at NASXs incredible history of 10 successful manned Apollo missions and more than 100 successful shuttle missions. In addition to highly trained personnel, spaceflight requires highly reliable technology. When NASA talks about reliability, the agency ultimately is talking about how long parts function before they are likely to fail. Consider the Apollo space program: Some experts have suggested that there were a total of about 2,000,000 functional parts in the Saturn V rocket, lunar module, and command module. How much error could have been tolerated in this complicated array? Even if the reliability for each part had been 99.9% for its contribution to the mission, the potential existed for about 2,000 parts to fail--in which case, the command module almost certainly would not have made it to the moon and back!

When we physical therapists talk about reliability, of course, we're talking about the error associated with a measurement. Reliability of 99.9% for a measurement used in physical therapy would almost always be astoundingly good! We could only dream....

Reliability can be a critical issue in the planning of a study. It also is a critical issue in clinical practice. The Journal has found that, regardless of whether authors are describing research or a patient case, the reason why a measurement was used cannot always be discerned in the submitted manuscript. Some authors do discuss the selection of measurements; other authors have to be asked to do so during revision. Either way, however, authors rarely clarify their clinical decision making--clarification that would enhance an otherwise superb article.

The truth is, no measurement is perfect. Whether physical therapists are making a diagnosis or determining a change in impairment or disability, all measurements have some error associated with them. All of our decisions can be error ridden! And errors are not eliminated or minimized by ignoring their presence.

Authors often try to justify the use of tests with the statement, "The reliability and validity of the measurements have been established." Even with a supporting reference, that statement is untenable. Reliability isn't like pregnancy. You can say that a woman is either pregnant or not pregnant--but you can't say that a measurement is either reliable or unreliable. Not only does error always exist, it is context dependent, and it relates to how the measurement will be used.

Is the error so large that using the measurement would be unlikely to provide useful information? Both in research and in routine practice, we have to consider whether the error could interfere with understanding the results of research or practice. Unlike statements of pregnancy, estimates of reliability lie along a continuum, and we need to know where along the continuum they lie and what that means for how the measurement can be used.

We also benefit from knowing something about how other authors have studied the reliability of the measurement being used. Did other authors study subjects who are similar to those currently being described? Did the physical therapists who took the measurements in those other studies have training and experience similar to the physical therapists taking the measurements in the current study? Were the procedures similar? Was the research sufficiently robust in terms of numbers of subjects and methods that the estimated error can be accepted as an excellent approximation of the true error? These are not esoteric issues. They relate to the practical world in which we live. And they are the basis on which clinicians should choose the measurements they use with patients.

Unless authors share their thinking about the measurements they used, the concepts in an article cannot be developed, and readers are left to imagine what they should actually have been told. Instead of saying that reliability "was established," careful authors say, "We believe that the measurement was sufficiently reliable to be used because ...," followed by a logical argument, references, and details. In the Journal's experience, it takes only a few sentences to do this fight. When the issue requires more than a few sentences, that usually means there is complexity, and the paper therefore will be made better by addressing the issue and the complexities forthrightly.

In characterizing estimates of error--specifically, statistics that describe reliability--many authors cite experts such as Landis and Koch, (1) who contended that values of kappa above 80% indicated excellent agreement, values above 60% indicated substantial levels of agreement, values of 40% to 60% indicated moderate agreement, and values below 40% indicated poor to fair agreement. Other authors discuss other statistics. Unfortunately, what they have in common is an arbitrary method of judgment that does not relate to how a measurement will be used.

For authors, the convenience of being able to "classify" reliability estimates and then give them value-laden names is clear. By naming reliability estimates, authors can discuss them quickly and "be done with it," claiming that their measurements have been blessed by the well-respected authors of the original papers (if those measurements, for example, reach excellent levels). The problem is that we have no basis for the classification. If we were considering the diagnostic accuracy of two surgeons, for instance, would we find it acceptable that there was a 20% chance of disagreement when it came to the decision to perform life-threatening surgery? On the other hand, that level of disagreement about the necessity to remove a lipoma wouldn't (in my view) be so bad.

Can we tolerate having the same amount of error in all of our measurements? If we use a measurement that has a possible 30% error to determine whether there is normal accessory motion at the glenohumeral joint, can we consider that measurement to be as useful as a measurement with a possible 30% error that is used to determine whether we should refer a patient to a physician in an effort to ward off possible permanent neurological damage due to disc disease?

Authors and clinicians have an obligation to provide an argument as to why any problems with a measurement are not sufficiently large to be consequential, and the amount of error that we can tolerate depends on what we are measuring and how a measurement will be used. The Landis and Koch approach has no context and does not take into account the nature of the measurement and the decisions that might be made based on the use of the measurements. Context and use are critical issues for both authors and clinicians, the difference being that authors must discuss these issues explicitly in submitted papers, whereas clinicians must consider these issues in patient management.

Measurements are not equivalent to aerospace parts, of course, but there is something that the Apollo space program can teach us about reliability. Because NASA could not reduce the error level to "acceptable," they adopted an alternate strategy: planned redundancy, usually triple redundancy. They developed so many backup systems that a catastrophic failure could occur only when there were multiple failures of the same system. When our clinical measurements have more error than we want, the Apollo example should remind us that alternate strategies can be developed but authors need to explain these strategies, and, if they did not use any, authors should explain why.

Journal authors work hard in conducting studies and documenting practice, and harder still at preparing and revising their papers. They do their own work a disservice when they fail to share their thought process in choosing measurements and other aspects of their research methods. The same is true of clinicians who do not elaborate on why they chose measurements and interventions in case reports or who practice without regard to the quality of their measurements. Ignorance about the error level associated with measurements or dogmatic refusal to consider research evidence is poor practice.

Please don't view this Note as a statement that reliability is more important than other measurement properties--it is not! Validity, specificity, sensitivity, and a host of other properties--as well as related topics such as receiver operating characteristics (ROC curves)--are equally, if not more, important. The issue for all of these topics is the use, and the usefulness, of measurements. We need to justify and explain what we do, thereby achieving better articles, better practice, and, in the long run, better physical therapists.

References

(1) Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-174.

Jules M Rothstein, PT, PhD, FAPTA 

Editor in Chief