A quantitative analysis of research publications in physical therapy journals. (Research Report).

Evidence-based medicine or health care means integrating the "best research evidence with clinical expertise and patient values." (1(p1)) Herbert et al (2) suggested that evidence-based practice is the best model of clinical practice that is currently available to physical therapists and the one most likely to produce the best outcomes in patients because it leads to use of the best evidence, evidence that arises from carefully-conducted research, and combines it with the needs and preferences of the patient in determining the appropriate intervention. The practice of evidence-based health care is usually thought to involve 5 steps: defining the question, collecting the best evidence related to a question, critically appraising the evidence, integrating the evidence with clinical expertise and patient factors to carry out the decision, and finally evaluating the process so that it can be improved the next time. (1,3)

Some authors (2) have suggested that engaging in evidence-based practice is too time-consuming. Evidence is now being organized in new ways to make it easier for physical therapists and other health care professionals to access information that is valid and directly applicable to patient care. (3,4) For example, secondary publications such as systematic review articles or distilled clinical research literature found on the Internet or in abstract journals are easily accessible and offer synthesized information to clinicians. The Cochrane Collaboration produces the Cochrane Library, consisting of systematic reviews of randomized controlled trials (RCTs). It is available on CD-ROM and via the Internet (available at: http://www.update-software.com/cochrane/). Abstracts can be viewed for free there; yearly subscriptions are US$235.00. Internet sites such as "PEDro," produced by the Centre for Evidence-Based Physiotherapy in Australia (available at: http://ptwww.cchs.usyd.edu.au/ pedro/), which presents a database of RCTs and systematic reviews of RCTs, and DARE (Database of Abstracts of Reviews of Effectiveness) (available at: http://nhscrd. york.ac.uk/darehp.htm) are just 2 examples of Web sites that physical therapists can use to look for evidence. Both of these Web sites can be accessed free of charge.

A variety of evidence-based journals also present distilled clinical research. These journals present the research findings in a systematic and condensed way in order to make the findings more "digestible and manageable." (5 (p180) Using distilled or secondary publications can be a time-efficient choice for busy clinicians looking for the best evidence. ACP Journal Club and Evidence-Based Medicine are examples of evidence-based abstract journals for physicians, produced by a team at the Health Information Research Unit (HIRU) at McMaster University in Hamilton, Ontario, Canada. ACP Journal Club is published by the American College of Physicians and is targeted to internists, and Evidence-Based Medicine is jointly published by the American College of Physicians and the BMJ Publishing Group in the United Kingdom and is targeted to primary care general practice.

The BMJ Publishing Group also produces Evidence-Based Nursing and Evidence-Based Mental Health. The editorial staff for these journals have established review criteria that are used to identify "studies and reviews that warrant immediate attention by physicians attempting to keep pace with important advances." (6 (pA15) Each publication comprises a selection of systematic review articles and original studies of prevention or treatment, diagnosis, differential diagnosis, prognosis, causation, quality improvement and continuing education, economics of health care programs or interventions, and clinical prediction guides. These articles are identified following a standardized review process applied to more than 150 journals. When an article is presented as an abstract in any of the journals, it is accompanied by a commentary from an expert in the content area, who provides comments on the quality, clinical context, and appropriate application of the research. The accumulated contents since 1991 are on the Web as a database (available at: http://acpjc.org), which can be accessed free of charge at present, and also as part of Ovid's Evidence-Based Medicine Reviews (available at: http://www.ovid. com), available through many institutional libraries; the database contains a considerable amount of content of relevance to physical therapists.

Recently, the HIRU evidence-based journals team undertook a project to develop MEDLINE search strategies to retrieve articles that meet the standards of ACP Journal Club and the other evidence-based journals. This project, known as the Hedges Project, is funded by the National Library of Medicine in Washington, DC. Interrater reliability for assessments of these standards has been shown to be high (a level of agreement at least 80% greater than that expected by chance, as determined with the Kappa statistic). (7) Simple guidelines for critical appraisal of health care research evidence are widely available in print and on the Internet (eg, http://www/cche.net/ usersguides/main.asp).

Optimal study designs differ for determining cause, course, diagnosis, prognosis, prevention, therapy, and rehabilitation, so the rules for assessing validity differ according to the questions being studied. For example, randomized allocation of participants to intervention and control groups is held to be better than nonrandom allocation for controlling bias in intervention studies. This is not merely a matter of logic, common sense, or faith: nonrandom allocation usually results in larger and more variable estimates of the effects of an intervention than does random allocation. (8) Similarly, in observational study designs for assessing the accuracy of diagnostic tests, independent interpretation of the tests that are being compared is known to result in less optimistic reports on test performance. (9) Qualitative articles that are included in Evidence-Based Nursing and Evidence-Based Mental Health, but not in ACP Journal Club, also were evaluated using the established criteria. The criteria were applied to all issues of more than 170 journals during the year 2000. These journals did not include any physical therapy professional journals, such as the 4 journals we studied. The project that we describe in this article is an extension of this work to assess the quality of the literature in 4 prominent physical therapy professional journals.

We reviewed the contents of 4 physical therapy professional journals with large circulations (American, Australian, British, and Canadian) to identify the proportion of articles that meet the HIRU evidence-based journal standards for high-quality evidence suitable for direct application to clinical practice. The applicability of these scoring criteria for physical therapy's professional literature and different sources of high-quality evidence for physical therapist practice will be discussed.

Methods

The review was conducted by one individual (PAM). All articles published in 6 consecutive issues of the Australian Journal of Physiotherapy, Physical Therapy, Physiotherapy, and Physiotherapy Canada between January 2000 and June 2001 were reviewed. Each article was categorized using the Hedges Project scoring criteria (Appendix). Using the Hedges Project criteria, all studies were classified according to format: an original study reporting firsthand observations, a review or meta-analysis of the literature on a particular topic, a general or miscellaneous article that presents a general or philosophical discussion of a topic without original observation, or a case report. For all articles except those determined to be general and miscellaneous articles, we then determined whether the article was of interest to health care providers (it meets this criterion if it deals with information relevant to patient care). Only reviews or studies that dealt with patient care issues were evaluated in greater detail. For these articles, the purpose (eg, etiology, prognosis, diagnosis, treatment, prevention, continuing education, quality improvement, economics, clinical prediction guide, qualitative study) was identified. Articles that were relevant to patient care but did not fit into one of these categories were labeled "Something Else." All articles other than those labeled "Something Else" were then evaluated by standards for rigor and assigned a pass or fail score. The article must meet all of the defined criteria in order to pass. The Figure presents a flowsheet of the review process.

[FIGURE OMITTED]

Reliability of Reviewer's Assessments

The reviewer (PAM) attended the training sessions on the Hedges Project criteria presented to new HIRU staff members. These sessions included a review of criteria and application of the criteria to journal articles, guided by experienced staff. Following training, the contents of 6 journal issues representing 3 of the 4 journals were scored in collaboration with an experienced evaluator from the HIRU staff (KAM) in order to practice the application of the criteria to physical therapy publications. The reviewer then proceeded to review the journals in this study on her own, flagging articles where she was uncertain about the scoring, as is the practice in the Hedges Project. The appropriate scoring of these articles was then determined in consultation with the experienced staffperson (KAM). Table 1 lists the journal issues that were reviewed.

The test-retest reliability of the reviewer's assessments was determined by a blind reclassification of all of the articles (n=29) that had previously been evaluated for rigor as passing or failing to the meet the Hedges Project criteria. The purpose of the article (treatment [original study], treatment [review], clinical prediction guide, qualitative study) also was identified. This reassessment was conducted 8 weeks following the initial scoring by the same reviewer (PAM). The articles were evaluated in random order, as determined using a random numbers table. Using the formula for sample size determination reported by Walter et al (10) to test the hypothesis that the reliability is greater than .80 (generally regarded as an excellent level of agreement) and to compare it with the alternative hypothesis that it is not less than .50, with an alpha of .05 and a beta of .20, the required sample size would be 22 articles. Intraclass correlation coefficients (3,1) (11) for agreement between the original classification and the reclassification were .92 for evaluation of passing or failing and .93 for the articles' purpose. Robertson (12) reported a coefficient of agreement of .93 in the reclassification of 27 articles according to methods (n=8), and Klassen et al (13) reported an agreement of 91.9% in the reclassification of 62 articles in earlier studies reviewing journal content.

Results

A total of 179 articles were reviewed in 25 issues. One additional issue of Physical Therapy was required because one entire issue was devoted to a professional document (volume 81, number 1, January 2001).

The format of the articles determined for the 6 issues of each publication is shown in Table 2. The majority of the articles in 3 journals (Australian Journal of Physiotherapy, Physical Therapy, and Physiotherapy Canada) were categorized as original studies reporting firsthand observations (62%, 57%, 61%, respectively). The highest proportion of articles in Physiotherapy were categorized as "General and Miscellaneous" (47%) and the second most frequent format used was "Original Study" (38%).

For articles of interest to human health care, the majority of articles in the Australian Journal of Physiotherapy, Physical Therapy, Physiotherapy, and Physiotherapy Canada were determined to be "Something Else" (75%, 67%, 50%, and 65%, respectively). Articles in this category included investigations of the psychometric properties of new and existing measures and studies involving subjects without known impairments or disabilities. Reviews or studies of interventions were the second most common purpose identified for all publications. One prediction study and one qualitative study dealing with patient care issues also were identified. There were no studies dealing with etiology, prognosis, diagnosis, prevention, continuing education, quality improvement, or economics. Table 3 provides a summary of the results.

As shown in Table 4, a total of 19 articles (11% in all) met the Hedges Project criteria. The majority of these articles dealt with treatment (95%), as identified under the "Treatment (Original Study)" (14/19) and "Treatment (Review)" (4/19) headings. One qualitative study met the criteria. Physical Therapy had the greatest proportion of articles that met the criteria (7/7=100%).

Ten articles failed to meet the Hedges Project criteria for rigor. In 6 treatment studies, the design was not an RCT (n=5) or greater than 20% of the subjects were lost to follow-up assessment of a primary outcome (n=1). The 3 review articles dealing with treatment failed to meet the criteria because the authors did not provide a detailed outline of the search strategy that should identify the databases used (n=2) or the review did not include at least one RCT (n=1). In the one article in the "Clinical Prediction Guide" category, the predictive equation was not cross-validated in a second set of patients.

Discussion

The 4 journals in this study were selected because they are the publications of 4 professional physical therapy associations with large circulations. They delineate a group of well-established international journals for physical therapists and are distributed to more than 125,000 association members around the world. These journals have been included in similar previous studies where different review strategies have been used to quantify or rank physical therapy publications (13-15) Physical Therapy and the Australian Journal of Physiotherapy are the only journals indexed in Index Medicus (MEDLINE); all 4 journals are indexed in the Cumulative Index to Nursing and Allied Health Literature (CINAHL) database. Six issues per publication were reviewed because of the time constraints of this project. However, Haynes (16) has noted that this number is usually adequate to assess the quality of the evidence in a journal, giving a reasonable estimate of its performance over time.

Only 2 types of original and review articles were identified: those dealing with treatment (n=27) and clinical prediction (n=1). During the 6-issue period, there were no reviews or studies that would be classified as having an etiology, prognosis, diagnosis, or economics purpose using the Hedges Project criteria. One reason for this could be the small sample of issues that were reviewed. Our findings also may reflect the interests of those researchers who are submitting manuscripts to the journals, rather than being reflective of the values of practicing physical therapists. Perhaps the question of etiology or causation is not of primary interest to physical therapists, but analyzing the underlying problem (eg, diagnosis) and prognosis are 2 of the 5 integral elements of patient/client management as defined in the American Physical Therapy Association's (APTA) Guide to Physical Therapist Practice. (17)

Economic evaluation is becoming more relevant as health care professionals are asked to be more accountable for the services they provide and the outcomes that are achieved. In comparison, data from the Hedges Project research indicate that 28 of the 377 articles in 14 issues of the Archives of Physical Medicine and Rehabilitation during 2000 met the Hedges Project criteria. They dealt with treatment (15/377 [4%]), prognosis (8/377 [2%]), clinical prediction (3/377 [0.8%]), and diagnosis (2/377 [0.53%]). Twenty-five of the 623 articles in the 26 issues of Spine in 2000 met the Hedges Project criteria. They dealt with treatment (n=20), etiology (n=3), diagnosis (n=1), and prognosis (n=1). We would expect future publications in physical therapy journals to address a broader range of topics as well.

Only one qualitative research article was identified in our study. Although there are no qualitative studies abstracted in ACP Journal Club, each issue of Evidence-Based Nursing contains about 6 reports of qualitative studies. Ritchie (18) stated that physical therapists need to include qualitative research methods in order to bring together evidence-based practice and client-centered care. We would anticipate seeing the results of more qualitative studies published in the future.

The Institute for Scientific Information impact factor (IF), reported in the 2000 Journal Citation Report, (19) provides another way to address the quality of a journal. The Journal Citation Report covers over 6,000 multidisciplinary and international journals. The IF is the average number of times that a journal's recent articles (eg, those published within the 2 years proceeding the Journal Citation Report's cover year) were cited in other journals in the specific year. The larger the number, the more frequently other authors refer, in their own articles, to articles within the publication. In the 2000 Journal Citation Report, (19) Physical Therapy and the Journal of Orthopaedic and Sports Physical Therapy (with an IF of 1.424) were the only physical therapy journals noted. Physical Therapy had an IF of 1.222, which was slightly less than the IF for the Archives of Physical Medicine and Rehabilitation (1.409). This finding suggests that the quality of the articles in Physical Therapy is relatively high and that other authors make reference to its content, the performance of other biomedical journals such as the New England Journal of Medicine, which has an IF of over 29, must be taken into account. Although Maher et al (15) cautioned physical therapists not to accept the IF as a measure of the quality of physical therapy research or of a journal's prestige, it is a widely accepted manner of assessing the quality of journals used across many professions.

This review of these 4 physical therapy publications produced a low yield of articles that met the Hedges Project criteria. When each publication was examined individually and all of the articles in the 6 issues of each publication were included, the percentage of articles that met the ACP Journal Club criteria was 10% (4/42) for the Australian Journal of Physiotherapy, 15% (7/47) for Physical Therapy, 12% (4/34) for Physiotherapy, and 7% (4/56) for Physiotherapy Canada. For comparison, 7% (28/377) of articles published during 2000 in the Archives of Physical Medicine and Rehabilitation met the criteria. McKibbon et al (3) identified the "big four" journals (ie, British Medical Journal, Journal of the American Medical Association, The Lancet, and New England Journal of Medicine) as those where the majority of important advances in many health care disciplines are published. Bohannon (14) has argued that these 4 journals are also part of the core journals in physical therapy. The percentages of articles that met the ACP Journal Club standards in these 4 journals in 1999 were 14%, 22%, 25%, and 34%, respectively.

Haynes (16,20) noted that there is a small yield of articles of clinically useful and strong evidence in peer-reviewed clinical journals. He identified 4 forms of communication that journals support: (1) scientist-to-scientist information, where preliminary studies are reported; (2) scientist-to-practitioner communication, conveying information about studies that are ready for clinical application; (3) practitioner-to-practitioner communication such as review articles; and (4) practitioner-to-scientist information, which includes case reports and case series research. (20) The 4 journals that we studied would appear to comprise all 4 forms of communication. Haynes suggested that journals that include all 4 types of articles are a "hazard to clinicians and their patients" (20 (p725)) because most clinicians do not have the necessary skills to critically appraise these articles and understand if and how they apply to clinical practice. He suggested that publications need to clearly outline their purpose in determining the audience they will serve and that clinicians must cultivate excellent skills to critically appraise the literature. Journal editors can assist the readers by describing the purpose of articles. For example, when journals such as Physical Therapy identify their articles in the index as "research reports" and "case reports" and discussion articles are labeled as "viewpoints" as in Physiotherapy Canada, the purpose of the article is clearly identified for the clinician, possibly eliminating inappropriate application of the article's content. Haynes (16,20) suggested that the practicing clinician can benefit most from abstract journals such as ACP Journal Club that concentrate clinically useful information relevant to practice.

One reason for a low proportion of articles meeting the criteria is that the standards for evidence have been set at a high level. For example, for treatment articles, the design, according to the criteria we used, must be an RCT. This is in keeping with level I and level II of Sackett's levels of evidence. (21) If the article is a review, according to the criteria, it must have explicit information about the sources of data and the results must contain at least one study that meets the criteria for an original study (eg, includes at least one RCT if the review is about a therapeutic approach or about quality improvement or continuing education). All articles regarding intervention published in Physical Therapy met the Hedges Project criteria for treatment, whereas only one third of the articles in Physiotherapy Canada met them.

In another study conducted to examine the rigor of research articles, Maher et al (15) used the PEDro quality rating to evaluate the quality of RCTs. The intraclass correlation coefficient (1,1) for reliability of the consensus ratings for the total PEDro score has been shown to be .68 (95% confidence interval=.57-.76) (Christopher Maher, PT, PhD; personal communication; April 2002), Four PEDro scale items (randomization, concealed allocation, blinding, and adequacy of follow-up) have been validated in the medical literature, (22) but these scale items have yet to be validated for RCTs in the physical therapy literature. Maher and colleagues' results yielded a different ranking of these 4 journals. When RCTs since 1990 were considered, the Australian Journal of Physiotherapy had the highest mean score, followed by Physiotherapy Canada, Physical Therapy, and Physiotherapy. The differences in ranking of these 4 journals may be due to a number of different methodological considerations, including the dates of the journals reviewed and the scoring standards. Furthermore, Maher et al considered only RCTs, and we included research that addressed more than interventions.

The Hedges Project criteria exclude a number of research articles that could assist in the advancement of physical therapist practice. For example, articles are excluded because they do not include information about patients or patient care. These articles include studies that looked at subjects without known impairments or disabilities and that would heighten our understanding of basic mechanisms, thereby allowing us to make hypotheses about patients in future studies. Surveys and research regarding professional practice issues such as students and quality monitoring also are excluded. Original studies that deal with measurement issues of new or existing measures are not judged for rigor. Rothstein, the editor of Physical Therapy, noted that, although the physical therapy profession appears to have a "fixation on measurement studies," (23 (p775)) he encourages members of the profession to commit to a careful and scientific use of measurement in our research, as described in APTA's Standards for Tests and Measurements in Physical Therapy Practice. (24) If the application of the Hedges Project criteria to our professional journals were considered, the criteria would need to be modified to take into account the contemporary practice and research topics within the rehabilitation professions. As noted earlier, physical therapy journals serve to communicate a broad range of clinical and professional topics and research findings to their readers.

Where is high-quality evidence that may change physical therapist practice published? Physical therapy professional journals do contain high-quality evidence, although these articles do not appear in great numbers (ie, approximately 1 article per issue). This is also the same in medicine. (16) Robertson (25) noted that that authors in physical therapy journals place a high degree of reliance on literature sources other than physical therapy publications, including those of medicine and sports science. In searching the abstracted database of ACP Journal Club for the year 2000 using the search words "physiotherapy" and "physical therapy," 5 articles were identified. All 5 articles dealt with "treatment/therapeutics." These articles were published in the following journals: BMJ (2 articles), Lancet (2 articles), and Annals of Internal Medicine (1 article). It may be that when physical therapists complete RCTs and other important studies, they choose a medical journal that serves a larger multidisciplinary audience rather than a professional one. McKibbon et al (3) recommended that today's health care professionals need to make use of a combination of health care journals, sampling from their own professional discipline, from the "big four" journals, and from secondary journals to access appropriate journal coverage of important clinical information.

A limitation of our study was the small number of issues of each journal that were reviewed. A larger number of issues may have produced different results and indicated different trends in professional research activities. Further research with a larger sample size appears to be warranted.

Conclusion

Evidenced-based medicine or practice is defined as "the conscientious, explicit, and judicious use of current best-evidence in making decisions about the care of individual patients." (26(p71)) The greatest percentage of articles in the 4 physical therapy journals reviewed (N=179) were original research (56%) and general discussion and miscellaneous articles (33%), as defined by the Hedges Project criteria. When the original research articles were examined, many articles addressed topics that were not directly applicable to the provision of health care (eg, studies on subjects without known impairments or disabilities, research addressing measurement topics). Only 19 (11%) of the 179 articles in the 4 journals met the Hedges Project criteria of high-quality evidence suitable for direct application to patient care. The majority of these research articles dealt with treatment. In our study, we demonstrated that there were a small number of articles in this sample of physical therapy journals that can be utilized by the evidence-based practitioner. Clinicians who are looking for high-quality evidence to change patient care practices need to supplement their professional journal readings by accessing other sources of research findings, including medical journals, abstract journals, and resources on the Internet. A larger sample with a greater number of issues per journal may have yielded different results, and further research appears to be warranted.

Appendix.

Summary of the Hedges Project Criteria (a)

Format

Original study--a full-text article in which investigators report firsthand observations

Review--a full-text article that is bannered "review," "overview," or "meta-analysis' in the title or in a section heading or that somewhere in the text indicates that the intention of the article is to review, summarize, or highlight the literature on a particular topic

General and miscellaneous articles--a general or philosophical discussion of a topic without original observation and without a statement that the purpose is to review or appraise a body of knowledge

Case report--an original study or report that presents individualized data with fewer than 10 subjects

Of Interest to the Health Care of Humans

Yes--concerned with the understanding of health care in humans, will have an effect on the care of the patient/subject

No--not concerned with the understanding of health care in humans, will not have an effect on the care of the patient/subject leg, studies undertaken in subjects without known impairments or disabilities, the profession looking at itself, studies looking at the structure and function of the health care system, studies looking at research methodology issues)

Purpose of Original Study or Review

Etiology--content pertains directly to determining whether there is an association (causal link) between an exposure and a disease or condition

Prognosis--content pertains directly to the prediction of the clinical course or the natural history of a disease or condition, with the disease or condition existing at the beginning of the study

Diagnosis--contents pertain directly to using a tool to arrive at a diagnosis of a disease or condition. Screening to make a diagnosis is included here.

Treatment, prevention, continuing medical education, or quality improvement--content pertains directly to therapy, prevention, rehabilitation, quality improvement, or continuing medical education

Economics--content pertains to the economics of a health care issue

Clinical prediction guide--content pertains directly to the prediction of some aspect of a disease or condition, and the authors must indicate that the purpose of the study is to develop or validate a rule, guide, index, equation, scale, score, or model to predict a diagnosis, prognosis, risk, therapeutic response, or clinical outcome

Qualitative study--the content relates to how people feel or experience certain situations that relate to health care in humans. Collection methods and analyses are appropriate for qualitative data.

Something else--when the purpose of the study does not fit any of the above categories

Rigor

Each article is judged as "pass" or "fail," and to pass, it must include, in part (a):

Etiology--prospective standardized data collection, a clearly identified comparison group for those at risk for, or having, the outcome of interest, and blinding of observers to outcome of exposure

Prognosis--an inception cohort of individuals, all initially free of the outcome of interest, and follow-up of at least 80% of patients until a major study endpoint occurs or the study ends

Diagnosis--clearly identified comparison groups, at least one of which is free of the disorder of interest, interpretation of a diagnostic ("gold") standard without knowledge of the test results, interpretation of test without knowledge of the diagnostic standard result

Treatment and prevention--random allocation of participants to comparison groups, follow-up of at least 80% of the participants entering the investigation

Continuing medical education and quality improvement--random allocation of participants or units to comparison groups and follow-up of at least 80% of the participants

Economics--the economic question addressed must be based on comparison of alternatives. Alternative diagnostic or therapeutic services are compared on the basis of both the outcome produced (effectiveness) and the resources consumed (cost)

Clinical prediction guide--generation of the guide in one or more sets of patients (training set), validation in another set of patients (test sets)

Reviews--explicit statement of the inclusion and exclusion criteria applied for selecting articles, a description of the methods indicating the specific database sources, and at least one of the articles must meet the above noted criteria for treatment, diagnosis, prognosis, clinical prediction, etiology, quality improvement, or economics of health care

Qualitative study--the content relates to how people feel or experience certain situations; collection methods and analyses are appropriate for qualitative data

(a) The complete criteria are available from Dr Haynes ([e-mail]).

Table 1.
Journal Articles Reviewed

Journal                 Issues

Australian Journal of   Volume 46, numbers 1-4 (2000)
  Physiotherapy         Volume 47, numbers 1-2 (2001)
Physical Therapy        Volume 80, number 12
                          (December 2000)
                        Volume 81, numbers 2-6
                          (February-June 2001)
Physiotherapy           Volume 86, number 12
                          (December 2000)
                        Volume 87, numbers 1-5
                          (January-May 2001)
Physiotherapy Canada    Volume 52, numbers 1-4 (2000)
                        Volume 53, numbers 1-2 (2001)

Table 2.
Results of Categorization of Articles (N=179) According to "Format"

                   Original            Case     General and
                   Study      Review   Report   Miscellaneous   Total

Australian
  Journal of
  Physiotherapy     26         1         2       13               42
Physical Therapy    27         0         5       15               47
Physiotherapy       13         2         3       16               34
Physiotherapy
  Canada            34         6         1       15               56
Total              100 (56%)   9 (5%)   11 (6%)  59 (33%)        179

Table 3.
Results of Categorization of Articles Related to Human Health Care
(n=86) According to "Purpose" (a)

                                         Clinical
                        Treatment        Prediction   Qualitative
                        (Study/Review)   Guide        Study

Australian Journal of
  Physiotherapy          5               1            0
Physical Therapy         7               0            0
Physiotherapy            4               0            1
Physiotherapy Canada    11               0            0
Total                   27 (32%)         1 (1%)       1 (1%)

                        Something
                        Else        Total

Australian Journal of
  Physiotherapy         18          24
Physical Therapy        14          21
Physiotherapy           5           10
Physiotherapy Canada    20          31
Total                   57 (66%)    86

(a) There were no articles related to etiology, prognosis, diagnosis,
prevention, continuing medical education, quality improvement, or
economics in this sample.

Table 4.
Results of the Evaluation of Rigor of Articles (n=29)

                                                           Clinical
                        Treatment         Treatment        Prediction
                        (Study)           (Review)         Guide

Australian Journal of   3 pass, 1 fail    1 pass           1 fail
  Physiotherapy
Physical Therapy        7 pass
Physiotherapy           2 pass, 1 fail    1 pass
Physiotherapy Canada    2 pass, 4 fail    2 pass, 3 fail
Total                   14 pass, 6 fail   4 pass, 3 fail   1 fail

                        Qualitative                      Percentage
                        Study         Total              Pass

Australian Journal of                 4 pass, 2 fail      67%
  Physiotherapy
Physical Therapy                      7 pass             100%
Physiotherapy           1 pass        4 pass, 1 fail      80%
Physiotherapy Canada                  4 pass, 7 fail      36%
Total                   1 pass        19 pass, 10 fail    66%

References

(1) Sackett DL, Straus SE, Richardson WS, et al. Evidence-Based Medicine: How to Practice and Teach EBM. 2nd ed. Edinburgh, United Kingdom: Churchill Livingstone; 2000.

(2) Herbert RD, Sherrington C, Maher C, Moseley AM. Evidence-based practice: imperfect but necessary. Physiotherapy Theory and Practice. 2001;17:201-211.

(3) McKibbon A, Eady A, Marks S. PDQ Evidence-Based Principles and Practice. Hamilton, Ontario, Canada: BC Decker; 1999.

(4) Walker-Dilks C. Searching the physiotherapy evidence-based literature. Physiotherapy Theory and Practice. 2001;17:137-142.

(5) Foster N, Barlas P, Chesterton L, Wong J. Critically Appraised Topics (CATS): one method of facilitating evidence-based practice in physiotherapy. Physiotherapy. 2001;87:179-190.

(6) Purpose and procedure. ACP Journal Club. July/August 1999: A15-A16.

(7) Wilczynski NL, McKibbon KA, Haynes RB. Enhancing retrieval of best evidence for health care from bibliographic databases: calibration of the hand search literature. Medinfo. 2001;10(pt 1):390-393.

(8) Schultz KF, Chalmers I, Hayes RJ, Altman D. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273:408-412.

(9) Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of the design-related bias in studies of diagnostic tests. JAMA. 1999;282: 1061-1066.

(10) Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998;17:101-110.

(11) Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420-428.

(12) Robertson VJ. A quantitative analysis of research in Physical Therapy. Phys Ther. 1995;75:313-322.

(13) Klassen LE, Grzybowski WE, Rosser BWC. Trends in physical therapy research and scholarly activity. Physiotherapy Canada. 2001;53(1):40-47.

(14) Bohannon RW. Core journals of physiotherapy. Physiotherapy. 1999; 85:317-321.

(15) Maher C, Moseley AM, Sherrington C, Herbert R. Core journals of evidence-based physiotherapy practice. Physiotherapy Theory and Practice. 2001;17:143-151.

(16) Haynes RB. Where's the meat in clinical journals [editorial]? ACP Journal Club. 1993;119:A22-A23.

(17) Guide to Physical Therapist Practice. 2nd ed. Alexandria, Va: American Physical Therapy Association; 2001.

(18) Ritchie JE. Using qualitative research to enhance the evidence-based practice of health care provider. Aust J Physiother. 1999;45:251-256.

(19) 2000 Journal Citation Report (JCR), Science Edition. Philadelphia, Pa: Institute for Scientific Information Inc; 2001.

(20) Haynes RB. Loose connections between peer-reviewed clinical journals and clinical practice. Ann Intern Med. 1990;113:724-726.

(21) Sackett DL. Rules of evidence and clinical recommendations on the use of thrombotic agents. Chest. 1989;95(2):2S-3S.

(22) Moher D. Does quality of reports of randomized trials affect estimates of intervention efficacy reported in meta-analysis? Lancet. 1998:352:609-613.

(23) Rothstein JM. Editor's note: Sick and tired of reliability? Phys Ther. 2001;81:774-775.

(24) Task Force for Standards for Tests and Measurements in Physical Therapy Practice. Standards for tests and measurements in physical therapy practice. Phys Ther. 1991;71:589-622.

(25) Robertson VJ. Research and the cumulation of knowledge in Physical Therapy. Phys Ther. 1995;75:223-232.

(26) Sackett DL, Rosenberg WMC, Gray MJA, et al. Evidence-based medicine: what it is and what it isn't. BMJ. 1996;312:71-72.

Invited Commentary

Miller et al are to be congratulated on a valuable and timely analysis of research publications in 4 premiere physical therapy journals. I would like to comment on several issues raised by their major findings and the implications I see for the physical therapy research community.

The findings reported in this article inform us that a majority of articles being published by these journals are original research studies (56%). From my perspective, that finding is very good news given the paucity of research evidence in physical therapy. However, although 86 of the articles reviewed were categorized as "containing information relevant to patient care, "just 27 focused on interventions, only 1 addressed clinical prognosis or prediction, and none addressed questions regarding etiology, prevention, diagnosis, quality improvement, or health economics. This finding is less reassuring given the profession's urgent need for clinically relevant research evidence to inform evidence-based practice in physical therapy. Evidence-based practice is not possible in the absence of evidence on topics such as treatment efficacy, diagnosis, prognosis, prevention, and prediction.

If one accepts the implication that the infrequent publication of clinically relevant research in physical therapy is an indicator that insufficient clinically relevant research is being done in physical therapy, one needs to ask, Why is this so? In the United States, the American Physical Therapy Association recently formulated and published Clinical Research Agenda for Physical Therapy, (1) a clear demonstration of the growing importance and emphasis being placed on clinically relevant research to the profession. This document calls on scientists in the field to conduct precisely the type of clinically relevant research evidence cited by these authors as infrequently appearing in these 4 journals.

What can be done to stimulate more research in physical therapy that has direct clinical relevance? In the United States, and possibly elsewhere around the world, I believe clinicians and scientists in physical therapy have functioned too separately, with the university operating in isolation from the clinical setting. Unlike medicine, in physical therapy there has been a divergence of clinical specialization and scientific inquiry, a divergence that I believe has hindered the conduct of clinically relevant research that will meet the contemporary demands of society for clinical interventions rooted in sound theory and scientific evidence, versus tradition and anecdote.

I believe clinically relevant research will proliferate when our university faculties and clinical specialists come together to collaborate closely on the design and conduct of research. Clinicians must seek out researchers, and researchers must find clinicians with whom to collaborate. Better yet, clinical specialists must develop skills to conduct clinical research and be rewarded for it. The time constraints faced in today's insurance-regulated practice environment cannot be allowed to deter the profession from initiating and participating in clinical research. We researchers and clinicians must develop innovative ways to collaborate in the development, evaluation, and dissemination of future clinical innovations in physical therapy.

The second issue I would like to comment on is what the article says about the role of professional journals in physical therapy. The authors used Haynes' 4 forms of communication within professional journals (2) to raise some interesting questions about the role of professional journals. Haynes categorizes professional communication as: (1) preliminary studies (scientist-to-scientist), (2) studies ready for direct clinical application (scientist-to-practitioner), (3) review articles (practitioner-to-practitioner), and (4) case series (practitioner-to-scientist). What does this article tell us about the state of scientific literature in physical therapy? How do we evaluate the current emphasis in professional journals? And most importantly, what mix of articles should a journal aim for in the future?

The evidence presented in the article suggests that contemporary physical therapy journals contain articles that represent all 4 forms of communication as outlined by Haynes. (2) Most attention is directed toward scientist-to-scientist communication. Is this an explicit policy on the part of these journals, and, if so, is it one that should be encouraged or discouraged? Can the needs of all these audiences be well met by one journal? What is the primary audience for each journal? Is it the scientific community? Is it the clinical community? Or both? Has the priority been made clear to readers and those who submit articles to the journals?

Can a professional journal meet the needs of both communities? Haynes challenges us by suggesting that journals that include all 4 types of articles represent a "hazard to clinicians and their patients" (2 (p725)) because most clinicians do not have the necessary skills to critically appraise research articles and understand how they might apply to their practice. Is Haynes correct in this assertion, or does he beg the question of why don't clinicians have the skills needed to critically appraise articles in the literature? Should the profession and academic institutions, in particular, strive to better prepare clinicians with the critical appraisal skills necessary to appraise articles and conduct evidence-based practice, or should journals abandon the goal of communicating directly to practitioners and serving as a vehicle for clinicians to communicate with scientists? Leave communication to clinicians to other vehicles such as evidence-based journals, whose goal is to distill clinical research for its readers?

The findings in this study raise interesting issues for the profession, editors, and editorial boards representing these physical therapy journals. The authors are to be congratulated for bringing these issues to our attention, and I look forward to further dialogue on these issues in the coming years.

Alan M Jette, PT, PhD
Sargent College
Boston University
Boston, MA 02215

References (1) Clinical Research Agenda for Physical Therapy. Phys Ther. 2000;80: 499-513.

(2) Haynes RB. Loose connections between peer-reviewed clinical journals and clinical practice. Ann Intern Med. 1990;113:724-726.

Author Response

We want to thank Dr Jette for his thoughtful commentary on our article. We would be very pleased if our article can encourage increased communication and collaboration among physical therapy clinicians and researchers to facilitate the publication of clinically relevant, new evidence. In so many instances, the clinician who identifies the important clinical question is without the time, knowledge, or resources to design and implement the research project and requires the assistance of individuals with research expertise. Often these resources are not easy to find, and our profession must develop strategies and processes that facilitate collaborative research activities. The North American Orthopaedic Rehabilitation Research Network is one example of how clinicians and clinical researchers can work together to advance the profession by sharing strengths in a collaborative manner. (1) More such partnerships are necessary to help increase the amount of clinical research. We also agree with Dr Jette that a broader scope of research activities, including such topics as prognosis, prediction, and treatment efficacy, is necessary so that clinicians will have the appropriate evidence to guide practice.

We would like to make one point for clarification. Our article is about the state of research presented in selected physical therapy journals, not about the state of physical therapy research. For every clinical profession, the literature is spread among a broad range of journals. Furthermore, the best literature in any field generally appears in the journals with the biggest circulation. So, for example, if the quality of the information is high and the results are clinically relevant (scientist to clinician communication), general medical journals may be chosen as the forum for distribution of the information. This is especially so for interdisciplinary research. Therefore, we remind readers that there are many other journals where high-quality research undertaken by physical therapists is published. A comprehensive understanding of the state of research and publication practices of physical therapists would require a very different process of journal review than the one we present here.

We appreciate this opportunity to discuss strategies to produce and disseminate evidence among our professional colleagues.

Patricia A Miller, PT, MHSc
R Brian Haynes, MD, PhD

Reference

(1) Binkley JM, Stratford PW, Lott SA, Riddle DL. The Lower Extremity Functional Scale (LEFS): scale development, measurement properties, and clinical application. Phys Ther. 1999;79:371-383.

PA Miller, PT, MHSc, is Assistant Clinical Professor, School of Rehabilitation Science, McMaster University, Hamilton, Ontario, Canada, and Chief of Physiotherapy Practice, Hamilton Health Sciences, Hamilton, Ontario, Canada. When this article was written, she was a graduate student in the Health Research Methodology Program at McMaster University. Address all correspondence to Ms Miller at 5 Undercliffe Ave, Hamilton, Ontario, Canada L8P 3G9 ([e-mail]).

KA McKibbon, BSc, MLS, is currently on leave from the Department of Clinical Epidemiology and Biostatistics, McMaster University, for PhD studies at the Centre for Biomedical Informatics, University of Pittsburgh, Pittsburgh, Pa.

RB Haynes, MD, PhD, is Professor of Clinical Epidemiology and Medicine, McMaster University. He is Editor of ACP Journal Club.

Ms Miller provided concept/idea/research design and project management. All authors provided writing and consultation (including review of manuscript before submission). Ms Miller and Ms McKibbon provided data collection, and Ms Miller and Dr Haynes provided data analysis. Ms McKibbon provided training materials, and Dr Haynes provided facilities/equipment and institutional liaisons. The authors acknowledge the statistical assistance of Dr Stephen Walter and the editorial comments provided by Dr Mary Law.

The material in this article was the basis for a podium presentation at the National Congress of the Canadian Physiotherapy Association; July 11-14, 2002; Halifax, Nova Scotia, Canada.

This article was submitted December 12, 2001, and was accepted August 6, 2002.