Interpreting systematic reviews: sometimes, more is better.

Many of us remember when it was common for journals to publish literature reviews in which content experts summarized topical information for the reader. A review of the literature, however, does not protect the reader against the bias of the content expert. In fact, a literature review provides an opportunity for experts to support their bias using relevant literature.

The systematic review, on the other hand, is designed to locate, appraise, and synthesize primary studies reliably, using a scientific method that is intended to decrease bias and errors. (1) The clinical problem should be focused and clearly stated, and each component of the review should be carefully described: the systematic strategy used to search for relevant articles, the method used to critically appraise the articles before they are included in the review, the method used to collect data from the selected articles, and the methods used to synthesize those collected data. (2) The Cochrane Collaboration (3) is one of the best-known organizations that provide an infrastructure for conducting, maintaining, and updating systematic reviews of the effectiveness of health care interventions. However, because the systematic reviews overseen by the Cochrane Collaboration are not comprehensive, many journals commonly publish systematic reviews on relevant topics, rather than literature reviews.

You may be surprised to find that Physical Therapy has published 2 systematic reviews examining the effectiveness of physical therapy interventions for temporomandibular disorders. In the May 2006 issue, McNeely, Armijo Olivo, and Magee published "A Systematic Review of the Effectiveness of Physical Therapy Interventions for Temporomandibular Disorders"; in this issue, Medlicott and Harris publish "A Systematic Review of the Effectiveness of Exercise, Manual Therapy, Electrotherapy, Relaxation Training, and Biofeedback in the Management of Temporomandibular Disorder." Medlicott and Harris contacted me when the May issue was published and offered to forgo publication of their article. Although their gesture was noble, I decided to publish both articles.

At first blush, this situation would appear to be an editor's worst nightmare. Publishing 2 papers addressing the same topic within a short period of time is highly unusual. I believe, however, that these 2 articles provide us with an opportunity to learn more about the effectiveness of interventions for temporomandibular disorders. With the publication of these 2 articles, we also have the opportunity to compare and contrast the design and execution of 2 systematic reviews on a single topic. Do you, for example, expect that 2 systematic reviews on the same topic by 2 independent groups of researchers will yield similar findings?

Close examination of the 2 articles reveals important differences. McNeely et al limited their review to randomized controlled trials, searched 7 different databases, included English and non-English language articles, and sought unpublished data in a systematic manner. McNeely et al defined the scope of physical therapist practice for temporomandibular disorders to include exercise, acupuncture, electrophysiologic modalities, manual therapy, and mechanical therapy devices. They included articles with multiple interventions, such as splint therapy, acupuncture, electrotherapy, and therapeutic exercise. They identified 1,138 potentially eligible articles, and found that 14 of the articles met the criteria for inclusion in data synthesis. Meanwhile, Medlicott and Harris included studies with a variety of designs, including randomized controlled trials and cohort studies. They used 3 different databases and limited their search to English-language literature. Their review excluded research that used a combination of interventions, acupuncture, or passive mechanical devices. Thirty studies of a possible 108 met the criteria for inclusion into the data synthesis by Medlicott and Harris.

Substantive differences also exist in the quality ranking of studies included in the reviews. For example, McNeely and colleagues used the Jadad scale (4) and ranked all 4 of the studies examining exercise and manual therapy interventions as weak, (5-8) whereas Medlicott and Harris used a different ranking system (9-12) and reported that, of 14 studies examining the effect of exercise and manual therapy, 4 met the threshold for acceptable methodological rigor.

Despite the use of different inclusion criteria, Medlicott and Harris selected 2 of the final 4 articles (5,6) identified by McNeely et al to examine the effect of exercise on reducing pain and increasing oral function. Both McNeely et al and Medlicott and Harris agreed that there is evidence to support the use of oral or postural exercise including manual therapy and that the evidence is weak.

Comparison of the 2 systematic reviews also emphasizes the need to carefully examine the inclusion criteria for study entry. (13) For example, the reader interested in the effect of a combination of interventions should focus on the McNeely et al article, because they examined combination as well as individual interventions, whereas Medlicott and Harris excluded combination intervention trials in their review.

It is wonderful to see that several of the results reported in these 2 systematic reviews are similar, which is a tribute to both groups of authors and a validation of the methods used in the reviews. Both reviews contained all of the elements required for a systematic review and used well-defined methods to rate the quality of the articles and to critically appraise them.

It is rare for the Journal to publish 2 articles within a short period of time that examine essentially the same question. In this case, I believe that these 2 systematic reviews--both separately and in combination--make an important contribution to our literature. When combined, they teach us not only about the management of temporomandibular disorders but also about the subtleties of conducting and interpreting systematic reviews.

References

(1) Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126:376-380.

(2) An introduction to systematic reviews. Changing Practice: Evidence Based Practice Information Sheets for Health Professions. 2001; supplement 1:1-6. Available at: http://www.joannabriggs.edu.au/pbf/CP2.pdf. Accessed June 1, 2006.

(3) The Cochrane Collaboration. The Cochrane Manual. Issue 3, 2006 [Updated May 24, 2006]. Available at: http://cochrane.org/admin/manual.htm. Accessed June 1, 2006.

(4) Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17:1-12.

(5) Carmeli E, Sheklow SL, Bloomenfeld I. Comparative study of repositioning splint therapy and passive range of motion techniques lot anterior displaced temporomandibular discs with unstable excursive reduction. Physiotherapy. 2001;87:26-36.

(6) Wright EF, Domenech MA, Fischer JR Jr. Usefulness of posture training for patients with temporomandibular disorders. J Am Dent Assoc. 2000;131:202-210.

(7) Grace EG, Sarlani E, Reid B. The use of an oral exercise device in the treatment of muscular TMD. Cranio. 2002;20:204-208.

(8) Komiyama O, Kawara M, Arai M, et al. Posture correction as part of behavioural therapy in treatment of myofascial pain with limited opening. J Oral Rehabil. 1999;26:428-435.

(9) Sackett DL. Rules of evidence and clinical recommendations for the use of antithrombotic agents. Chest. 1986;89 (2 suppl):2S-3S.

(10) Megens A, Harris SR. Physical therapy management of lymphedema following treatment for breast cancer: a critical review of its effectiveness. Phys Ther. 1998;78:1302-1311.

(11) Harris SR. How should treatments be critiqued for scientific merit? Phys Ther. 1996;76:175-181.

(12) Law M, Stewart D, Pollock N, et al. Guidelines for critical review form--quantitative studies. 1998. Available at: http://www.fhs.mcmaster.ca/rehab/ebp/. Accessed June 2, 2006.

(13) Linde K, Willich S. How objective are systematic reviews? Difference between reviews on complementary medicine. J R Soc Med. 2003;369:17-22.

Rebecca L Craik, PT, PhD, FAPTA 

Editor in Chief

[e-mail]