Sick and Tired of Reliability?

Some readers will peruse this month's table of contents and conclude that the Journal has again indulged itself in its fixation with clinical measurement, a subject many see as irrelevant to clinical practice. I share their wish that we may see future issues devoid of measurement studies. I look forward to printing fewer measurement studies because of what we will know, because of what we will be doing--and because our focus could be on innovation and refinement.

Physical therapists have among their armamentarium for clinical practice literally hundreds of measurements. New measurements seem to be suggested almost daily. We now are awash in measurements of dubious quality that are used for questionable decision making and that are chosen not for their scientific merit but for a host of other reasons. The issue remains, what does a measurement tell us? How certain can we be that a measurement truly reflects what we are trying to measure?

These are real clinical issues. Similarly, it is a clinical issue when people cite or write in texts about what makes a measurement good (eg, what are acceptable levels of reliability). Why is this a clinical issue? How can there be a level of acceptability for all measurements? The acceptable error in a measurement is a clinical judgment based on how a measurement will be used. Would any of us accept the same level of error for a life-and-death decision as we would for seeing whether we can wait a week before having reconstructive surgery for a torn ligament? Too many authors of articles and texts discuss measurements as though they do not relate to the practical world and thus they foster the image of measurement research as an elitist, non-clinically relevant enterprise.

In practice shouldn't we know how certain we should be when we base our decisions on measurements? Not all measurements are equally error free. When people simply say that a measurement is or is not reliable, they err. Reliability is not an all-or-none phenomenon, but rather lies along a continuum. There is error in all measurements, but the issue is whether the error is consequential given how the measurement is being used. Should you repeat examinations when you have to depend on measurements that have poor reliability? Can you proceed with greater certainty when you know that a measurement is almost error free and accurately reflects what you want to measure? Reliability is a property of a measurement, not a test or an instrument.

As we assume more responsibility for the management of patients and for the delivery of interventions, others will ask us about the basis for our decision making and whether we can credibly make diagnoses and prognoses and document the effects of pathology and changes which may occur as a result of intervention. That is why we need measurements of known qualities, and we need to act and use these measurements in a manner that is consistent with current scientific knowledge.

In editing this Journal, we ask authors to address the reliability and validity of measurements. In their papers, authors often mention those two issues and never consider the implications. We find this unacceptable. Researchers can choose whether to proceed with their inquiry when they have trouble obtaining sound measurements. Clinicians have fewer choices, however, and must proceed in some fashion when they have a patient; they therefore often must use measurements of poor or uncertain quality.

We had an author acknowledge in a case report (recognizing that the article is a case report is important because it relates to how a real patient was managed) that the measurement used had never been studied for reliability or validity. In other words, the author was, at our insistence, admitting that the measurement used in the care of the patient could not be justified based on the literature, because it had never been studied. In a subsequent sentence, however, the author stated that this measurement is known to reflect an aspect of neuronal functioning. That is, the author admitted that there is no evidence and then attempted to say that the measurement reflected a property of a nerve. How could the author know this? Here is the dangerous blurring between opinion and knowledge based on data.

Because we want measurements to be useful doesn't mean we can make believe that they are what science has yet to show us. We made sure that the inappropriate claim made in the case report was eliminated. The author has every right to claim that the measurement was used because none is known to be better and that, in the author's opinion, the measurement may be useful. In this way, the author acknowledges making a data-free decision and taking responsibility for that decision--something that professionals are forced to do on a regular basis. When there is no better choice (ie, when there is no evidence), it can be quite useful to discuss how we use our opinions and experiences to shape our clinical decision making.

When anyone uses measurements that cannot be justified based on research (related to reliability, validity, sensitivity, specificity, or any other property of the measurement) because there are no better measurements, he or she is actually engaging in evidence-based practice and has no cause for shame or embarrassment. We often function based on opinion because a patient's care cannot await new research. When we acknowledge that we are using our opinions because there are no data (and are correct in this assertion), we are using the best possible evidence--which in this case is none. On the other hand, if we make believe that there is more than our opinion underlying the use of a measurement or even an intervention, we violate the trust placed in us. Again it is our responsibility to know what is, based on data, the best possible measurement and to use that measurement with appropriate limitations.

Reliability--which also means objectivity, and which can be assessed for almost any type of measurement--is less important than other properties of measurements, but it is often the focus of articles because it is easy to study. In addition, reliability is often studied because a measurement can have little value if reliability is known to be bad. Most important in our selection of measurements, however, is whether the measurement has actually been shown to be useful, and that means we must also consider the validity of these measurements.

Measurements have varying degrees of validity, and this too must be considered in our decision making. In addition, validity, like reliability, is not a property of a tool, test, or instrument, but rather a property of a measurement. There are many additional critical issues as to whether we should use particular measurements in practice. As long as we do not have the answers and find many therapists misusing measurements, this Journal will publish measurement studies. Important decisions should not be based on dubious information, and interventions cannot be evaluated credibly with questionable measurements.

Perhaps if we as a profession commit ourselves to more careful and scientific use of measurements and insist that educators address this topic more fully, we can move beyond what seems like a fixation on measurement studies. The issue has been addressed in APTA's clinical research agenda.[1] Measurement studies are part of what needs to be done, and the general guide to what we should expect in measurements and how they should be used is documented in APTA's Standards for Tests and Measurements in Physical Therapy Practice.[2]

References

[1] Clinical Research Agenda for Physical Therapy. Phys Ther. 2000;80:499-513.

[2] Task Force for Standards for Tests and Measurements in Physical Therapy Practice. Standards for tests and measurements in physical therapy practice. Phys Ther. 1991;71:589-622.

Jules M Rothstein, PT, PhD, FAPTA Editor