Evaluating the quality of reporting occupational therapy randomized controlled trials by expanding the CONSORT criteria.Kielhofner, Hammel, Finlayson, Helfrich, and Taylor (2004) have posited that research on outcomes is essential to the profession of occupational therapy. They went on to argue that several different research methodologies are needed to study occupational therapy outcomes. Specifically they described the potential contributions of measurement studies, database research, qualitative research Qualitative research Traditional analysis of firm-specific prospects for future earnings. It may be based on data collected by the analysts, there is no formal quantitative framework used to generate projections. , meta-analyses, and randomized ran·dom·ize tr.v. ran·dom·ized, ran·dom·iz·ing, ran·dom·iz·es To make random in arrangement, especially in order to control the variables in an experiment. controlled trials (also called randomized clinical trials, or RCTs). Of these designs, the RCT RCT Randomized Controlled Trial RCT Regimental Combat Team (infantry regiment with their own artillery, engineers, medical and tanks) RCT Rollercoaster Tycoon RCT Randomized Clinical Trial RCT Rhondda Cynon Taff "most rigorously ensures that outcomes can be attributed to treatment versus other intervention effects" (p. 18). Although it is important to note that RCTs are not appropriate for all occupational therapy research questions, it is also important to recognize the broad and stable consensus among scientists that RCTs provide the most definitive evidence of efficacy and effectiveness in health care (Altman et al., 2002; National Institutes of Health, Division of Research Grants, Research Analysis and Evaluation Branch, 1979). Essentially, an RCT is a refined experiment, with random assignment of participants to different interventions (and/or to control conditions), with systematic testing of outcomes, and with controls over possible confounding confounding when the effects of two, or more, processes on results cannot be separated, the results are said to be confounded, a cause of bias in disease studies. confounding factor variables (Friedman, Furberg, & DeMets, 1998, Chap. 1). The typical problems of studying health care have led to additional refinements in RCTs over the years, including special procedures for randomizing participants (often new patients) entering the study at various times and for blinding (masking mask·ing n. 1. The concealment or the screening of one sensory process or sensation by another. 2. An opaque covering used to camouflage the metal parts of a prosthesis. ) researchers and participants to prevent possible bias (Moher, Schulz, & Altman, 2001). In the 2000 Eleanor Clarke Slagle lecture, Holm holm n. Chiefly British An island in a river. [Middle English, from Old Norse h (2000) argued that the best support for evidenced-based occupational therapy practice consists of systematic reviews of multiple, well-designed RCTs. According to according to prep. 1. As stated or indicated by; on the authority of: according to historians. 2. In keeping with: according to instructions. 3. Holm, the profession must foster increased knowledge among therapists of RCTs, including guidelines used in judging quality. According to the American Occupational Therapy Association (AOTA AOTA American Occupational Therapy Association. ) Standards of Practice (American Occupational Therapy Association [AOTA], 1998), the occupational therapist must be able to apply research findings ethically and appropriately to evaluation and intervention procedures. This competency clearly depends on the ability to evaluate the quality of research articles, particularly RCTs. In a recent paper, Stern (2005, p. 161) posed the following questions to her research students: "How well written were the articles? How do you decide for yourself whether an article is well written?" To answer these questions, students, practitioners, and instructors need detailed guidelines refined over time within the scientific community. Quality of reporting directly affects the reader's ability to evaluate the validity and potential usefulness of a research article: "Critical appraisal Noun 1. critical appraisal - an appraisal based on careful analytical evaluation critical analysis appraisal, assessment - the classification of someone or something with respect to its worth of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in published articles" (Altman et al., 2002, p. 1). To address the issue of adequate reporting of research, international journal editors, trialists, and methodologists developed the CONSORT CONSORT. A man or woman married. The man is the consort of his wife, the woman is the consort of her husband. (Consolidated Standards of Reporting Trials CONSORT Statement CONSORT stands for Consolidated Standards Of Reporting Trials. It encompasses various initiatives developed by the CONSORT Group to alleviate the problems arising from inadequate reporting of randomized controlled trials. ) Statement (Begg et al., 1996). The current, revised version Revised Version n. A British and American revision of the King James Version of the Bible, completed in 1885. Revised Version Noun of the CONSORT provides a checklist of 22 essential items that should be included in an RCT, and a diagram for documenting the flow of participants through a trial (Moher, Shulz, & Altman, 2001). Prominent features of the CONSORT include an emphasis on randomization randomization (ranˈ·d  ·m and detailed reports of participant selection,
allocation, and methods. Numerous medical and health care journals and
organizations have adopted the CONSORT, available on the Internet
(http://consort-statement. org). For example, editors of journals
oriented to complementary medicine have accepted CONSORT and then built
upon it by adding specific guidelines for acupuncture acupuncture (ăk`y pŭng'chər), technique of traditional Chinese medicine, in which a number of very fine metal needles are inserted into the skin at specially designated points. called STRICTA
(Standards for Reporting Interventions in Controlled Trials of
Acupuncture) (MacPherson et al., 2001).
Of course, research reporting in occupational therapy is somewhat different from reporting drug trials in medicine (Nelson & Mathiowetz, 2004). For instance, often there is little previous research on which to build in many areas of occupational therapy practice (CONSORT item 1). It is also difficult to describe all aspects of interventions, such as the therapeutic relationship (CONSORT item 4). Furthermore, it is usually impossible to mask participants and interventionists in occupational therapy research (CONSORT item 11). Nevertheless, it is essential that occupational therapy RCTs move toward compliance with CONSORT criteria in all possible areas. There is no reason why occupational therapy RCTs cannot describe participants, outcome measurements, randomization procedures, and statistical procedures as detailed in CONSORT. A problem in using the CONSORT is that each of the 22 items consists of multiple statements, each of which can be evaluated independently. For example, CONSORT item 1 requires that (a) the term random be used in the title, (b) the term random be used in the abstract, and (c) the abstract be structured by subheadings. This format is adequate for authors and editors, but problematic for quantitative use of CONSORT. One solution would be to require all components within the item in order to certify the item as "passed." However, this would equate an article that complied with none of the components with an article that complied with all but one. Another solution has been to score each CONSORT item as met, partially met, or not met (Herbison, 2005). A third solution has been to divide each CONSORT item into sub-items. Hence Stinson, McGrath, and Yamada (2003) rated articles on 43 items drawn from CONSORT, and Moher, Jones, and Lepage for the CONSORT Group (2001) used 40 items. We reviewed and tested these latter two models, but found that the problem of multiple issues per item remained. Therefore, we developed a rating scale based exclusively on the CONSORT but consisting of 201 sub-items. The primary purpose of this study was to apply this detailed interpretation of the CONSORT criteria, the Nelson-Moberg Expanded CONSORT Instrument (NMECI), to RCTs relevant to occupational therapy. To what extent do reported occupational therapy RCTs comply with detailed CONSORT Statement CONSORT statement a research tool that uses an evidence-based approach to improve the quality of reports of randomized trials. criteria? This study also tested the interrater reliability of the NMECI. Method Preliminary Expansion of the CONSORT Statement The reason for expanding the 22-item CONSORT Statement was that each item consisted of multiple, independent concepts, each of which can be evaluated. Based on the elaboration document, The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration (Altman et al., 2002), the first author, EM, systematically divided each of the 22 CONSORT items into a more detailed checklist of 143 sub-items. The raters (DN and EM) conducted practice analyses on two published RCTs not used in the final analysis. The articles were from the fields of psychology and occupational therapy (Esplen, Garfinkel, Olmsted, Gallop, & Kennedy, 1998; Mann, Ottenbacher, Fraas, Tomita, & Granger, 1999). The raters met to analyze the two practice articles according to the 143-sub-item checklist so that the criteria could be discussed, applied, and analyzed in terms of potential problems of agreement in applying the criteria. Throughout the training, the raters kept a list identifying possible ambiguities in applying the criteria. DN used this list and additional review of the elaboration document to create the 201sub-item NMECI. Both raters then used the final instrument to practice the rating system on three medically oriented articles unrelated to occupational therapy. Rules for Sub-Items of the Nelson-Moberg Expanded CONSORT Instrument Throughout the process of expanding the CONSORT, the authors frequently encountered problems of logic, validity, and interpretability. To overcome these problems, the following rules were developed for consistent expansion of the CONSORT Statement into sub-items: 1. Generally, only one issue of research design or reporting can appear in each Nelson-Moberg sub-item. For example, reporting on the location of recruitment is a separate issue (and a separate sub-item) from reporting on the method of recruitment. The only exception to this rule occurs when an issue is variable (one or more). For example, a study might involve one secondary outcome or might involve several. In order to have the same number of sub-items to evaluate all articles, the decision was made to evaluate each variable issue in an all-or-none way (e.g., if the properties of any of the secondary outcomes were not described, no credit was granted for that sub-item). 2. Except for four sub-items (see 3 below), each item is rated as a 1 (compliance with CONSORT) or 0. 3. Four sub-items in CONSORT item 21 are scored as percentages. CONSORT item 21, dealing with generalizability, refers to CONSORT items 3, 4, 6, and 14, each of which has several Nelson-Moberg sub-items. Therefore, the score for generalizability includes sub-items reflecting the percentages received for CONSORT items 3, 4, 6, and 14. For example, if a study received 11 of 13 possible for CONSORT item 3, then a score of .85 would be given to the corresponding Nelson-Moberg sub-item. 4. Each NMECI sub-item involves an issue that can be scored as present or absent by a reviewer who is not an expert in the particular field under study and who is not a member of the research team. An example of an issue that cannot be scored by a nonexpert is in CONSORT item 2, where authors are urged to report the scientific background for the study. A nonexpert reviewer cannot evaluate the completeness or fairness of the authors' descriptions of scientific background; however, a reviewer can judge whether scientific background was given or not. An example of an issue that cannot be known by someone who was not a part of the research team is the CONSORT exhortation to report deviations from the research protocol. However, the reviewer can make a judgment as to reporting of whether or not deviations from the research protocol occurred (with follow-up explanations if they did occur). 5. If the same CONSORT issue appears in more than one CONSORT item, it is scored separately in each section. For example, CONSORT items 20 and 22 both call for a systematic review. Therefore, a study reporting a systematic review can receive credit on a Nelson-Moberg sub-item within CONSORT item 20 and can also receive credit on a Nelson-Moberg sub-item within CONSORT item 22. 6. Nelson-Moberg sub-items reflect the ideals cited by CONSORT, not the minimal requirements. For example, CONSORT item 13 states that a diagram of participant flow is strongly recommended; therefore, an NMECI sub-item deals with this issue. 7. RCTs that do not comply with CONSORT rules calling for clear identification of primary versus secondary outcomes (and which therefore are not credited on the relevant Nelson-Moberg sub-item) are not further penalized pe·nal·ize tr.v. pe·nal·ized, pe·nal·iz·ing, pe·nal·iz·es 1. To subject to a penalty, especially for infringement of a law or official regulation. See Synonyms at punish. 2. in subsequent sub-items dealing with the outcomes. Hence, an RCT that reports the provenance prov·e·nance n. 1. Place of origin; derivation. 2. Proof of authenticity or of past ownership. Used of art works and antiques. of all its outcomes receives credit for Nelson-Moberg sub-items dealing with reporting the provenance of primary and secondary outcomes. 8. Credit is granted if an NMECI sub-item involves an issue that is irrelevant to the RCT, but only if certain basic information in that CONSORT item is provided. For example, in CONSORT item 8, Nelson-Moberg sub-items dealing with full reporting of blocking and stratification stratification (Lat.,=made in layers), layered structure formed by the deposition of sedimentary rocks. Changes between strata are interpreted as the result of fluctuations in the intensity and persistence of the depositional agent, e.g. can be earned if blocking and stratification were not done in the study, but only if a basic description of the randomization process is given. 9. Generally credit is granted even if the authors provide the information in a different section of the paper. For example, credit for describing the participants (CONSORT item 3, dealing with the Method section) is granted if the authors describe the participants in the results. The exception to this rule is that the scientific background and explanation of rationale (CONSORT item 2) must be given prior to the Method section (e.g., not in the Discussion section). This rule is consistent with the CONSORT elaboration. An example of 13 NMECI sub-items drawn from CONSORT item 3 is provided in Figure 1. Also given are rules of interpretation consistent with the nine rules stated above. Selection of the Articles to be Analyzed RCTs were selected through OTseeker: Occupational Therapy Systematic Evaluation of Evidence, a database that includes meta-analyses and randomized trials pertaining per·tain intr.v. per·tained, per·tain·ing, per·tains 1. To have reference; relate: evidence that pertains to the accident. 2. to occupational therapy (OTseeker, 2003). In order for an RCT to be included in the study, (1) the trials had to be locatable using a search of OTseeker limited to controlled trials with "occupational therapy" in the title of the article; (2) the RCT included random allocation of at least two interventions (or one control condition compared to one or more interventions); (3) each study participant received only one condition of the independent variable (eliminating counterbalanced/crossover designs); (4) at least one of the interventions was part of occupational therapy practice, or could become part of occupational therapy practice; (5) the dependent variable involved an outcome of therapy, not a short-term indicator of motivation or some aspect of the therapeutic process such as exercise repetitions; (6) the trial involved human participants; (7) the trial was reported as a full-size article in a peer-reviewed journal peer-reviewed journal Refereed journal Academia A professional journal that only publishes articles subjected to a rigorous peer validity review process. Cf Throwaway journal. ; and (8) only one study per sample could be included (thus eliminating follow-up and related studies of previously reported participants). No limits were placed on dates of publication. Twenty-one studies were found in OTseeker on November 5, 2003, using the limitation of controlled trials and the key phrase "occupational therapy." One study was then excluded because it used a counterbalanced design, three studies were excluded because they did not involve therapy outcomes, and three studies were excluded because they used the same sample as another study to report follow-up or related results. The 14 articles are indicated by * in the reference list. After identifying the 14 RCTs, the evaluators independently used NMECI to systematically analyze the articles from June 2004 to September 2004. The evaluators had no contact with each other during the evaluation. Plan for Data Analysis The main indicator of compliance with CONSORT was planned as the average (mean or median, depending on skewness Skewness A statistical term used to describe a situation's asymmetry in relation to a normal distribution. Notes: A positive skew describes a distribution favoring the right tail, whereas a negative skew describes a distribution favoring the left tail. ) NMECI score when all of the 201 sub-items were summed (a possible range from 0 to 201). Also planned was computation of average compliance scores for each of the 22 CONSORT items and each of the 201 sub-items. DN's scores were used as primary data, with EM's scores used for interrater reliability purposes. The reason for using DN's scores was his relative level of research experience. The main test for interrater reliability dealt with the source of the main indicator of compliance, the total score. The intraclass correlation In statistics, the intraclass correlation (or the intraclass correlation coefficient[1]) is a measure of correlation, consistency or conformity for a data set when it has multiple groups. coefficient for generalization gen·er·al·i·za·tion n. 1. The act or an instance of generalizing. 2. A principle, a statement, or an idea having general application. to a future single rater rat·er n. 1. One that rates, especially one that establishes a rating. 2. One having an indicated rank or rating. Often used in combination: a third-rater; a first-rater. was used, formula 2,1, as described in Shrout and Fleiss (1979). The same statistic was used to evaluate each of the 22 CONSORT items. Simple kappa Kappa Used in regression analysis, Kappa represents the ratio of the dollar price change in the price of an option to a 1% change in the expected price volatility. Notes: Remember, the price of the option increases simultaneously with the volatility. was then used test interrater agreement on 197 of the 201 items, with four of the items (sub-items 193-196) tested by the ICC ICC See: International Chamber of Commerce because they involved percentages, not categorical That which is unqualified or unconditional. A categorical imperative is a rule, command, or moral obligation that is absolutely and universally binding. Categorical is also used to describe programs limited to or designed for certain classes of people. judgments. We used the guidelines proposed by Landis and Koch (1977) for interpreting the strength of kappa and guidelines proposed by Eliasziw, Young, Woodbury, and Fryday-Field (1994) for interpreting the ICC. Results Interrater Reliability The main test of agreement yielded an ICC of .95 , indicating "almost perfect" strength of agreement (Eliasziw et al., 1994). Next an ICC was computed (testing the level of agreement on the Nelson-Moberg sub-items reflecting each CONSORT item). ICC results for each of the 22 CONSORT items are presented in Table 1. This table indicates perfect or "almost perfect" strength of agreement for items 1, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 20, and 21; "substantial" strength of agreement for items 2 and 3; "moderate" strength of agreement for item 22; "fair" agreement for item 11; "slight" strength of agreement for item 18; and "poor" strength of agreement for item 19. ICCs were computed for sub-items 193 (ICC = .43), 194 (ICC = .83), 195 (ICC = .59), and 196 (ICC = 1). In addition, kappa values could not be computed for 21 sub-items because these sub-items had no variance (there was perfect agreement that either all 14 articles met the criterion, or that none met the criterion). Please see Table 2. The mean kappa on the remaining sub-items was .85, with an inter-quartile range from .58 to 1. According to Landis and Koch (1977), this indicates a high degree of agreement overall. Fully 126 of 176 kappas were perfect or were in the substantial or almost perfect ranges, according to Landis and Koch. However, 29 of the 176 kappas ranged from -.08 to .34, in the ranges termed poor, slight, and fair by Landis and Koch. Table 3 lists these sub-items and also includes information concerning percentage of agreement. Percentage of agreement is given because kappa can be low or even 0 when the two raters agree at high rates. For example, when the two raters agree in 13 of 14 cases (93%) that the criterion is met, but disagree in the 14th case, kappa is 0. Degree of Compliance of the Articles to CONSORT Overall, the articles met about half of the sub-items of the NMECI based on DN's ratings (M = 104.2, SD = 32.9, skewness = -.09). See Table 1 for the degree of compliance in each of the 22 CONSORT items. Eighty-one of the 201 NMECI sub-items were below a mean of .5, indicating that less than half of the articles met the criteria of these sub-items. Seven of the items were not met by any of the articles. For example, sub-item 29, which states, "describe the types of interventionists (duration of experience)," and sub-item 55, which states, "identify the estimated outcomes in each group (not just the expected difference between groups)," were not reported in any of the articles. Additional Analyses A t test was computed to compare evaluators in terms of positive versus negative ratings. DN's ratings (M = 104.2, SD = 32.9) were significantly lower than EM's ratings (M = 112.0, SD = 31.0) (t = 3.75, p = .002). A moderate yet statistically significant positive correlation Noun 1. positive correlation - a correlation in which large values of one variable are associated with large values of the other and small with small; the correlation coefficient is between 0 and +1 direct correlation was found between year of publication and CONSORT compliance, (rho = .55, p = .04). Years of publication ranged from 1975 to 2001, with a median between 1996 and 1997. Discussion Degree of Compliance of the Articles to CONSORT Independent raters agreed that RCTs relevant to occupational therapy published in peer-reviewed journals met slightly more than half (52% and 56%) of the requirements outlined by the CONSORT criteria. A few of the criteria identified in the CONSORT Statement are impossible to comply with in most occupational therapy research, such as blinding of interventionists and subjects (NMECI sub-items within CONSORT item 11). However, most of the sub-items are possible to report as recommended by CONSORT. Full reporting helps the reader of the article to make decisions about the internal and external validity External validity is a form of experimental validity.[1] An experiment is said to possess external validity if the experiment’s results hold across different experimental settings, procedures and participants. of the study, ultimately leading to improved clinical decision making. CONSORT item 5 (stating the specific objectives and hypotheses) was the only item judged to be in full compliance by all 14 articles. Other areas of relative strength include CONSORT items 3 (reporting characteristics of participants) and 16 (reporting number of participants analyzed). Relatively weak areas were CONSORT items 9 (methods to conceal randomization), 10 (methods to implement randomization), 11 (blinding), and 19 (description of adverse events). These are areas that are not particularly difficult to comply with in occupational therapy research. For example, although it is impossible to blind interventionists and participants in most occupational therapy research, it is easy to report that they were not blinded. We suggest that researchers, reviewers, and editors develop the habit of addressing these issues. For example, CONSORT will be complied with if it is simply stated that no adverse events occurred, if that is the case. It is interesting to note that a recent CONSORT Group initiative has resulted in new guidelines for reporting adverse events, as a supplement to the main CONSORT criteria (Moher, Altman, Schulz, & Elbourne, 2004). The ratings in this report should not be interpreted as a rationale for disregarding the results of the research reported in these 14 articles. Many of the articles used in this study were written before publication of CONSORT and its adoption in peer-reviewed journals. Because the articles analyzed in this study use experimental design with randomization, they are most likely among the strongest research studies conducted in the field of occupational therapy. In fact, six of the articles met over half of the criteria. One article (Walker et al., 1999) met 150.8 of the 201 sub-items, and another (Parker et al., 2001) met 146.1 of the 201 sub-items. It is also important to point out that parallel studies of adherence to CONSORT criteria have indicated parallel problems of reporting in fields as diverse as pediatric pediatric /pe·di·at·ric/ (pe?de-at´rik) pertaining to the health of children. pe·di·at·ric adj. Of or relating to pediatrics. psychology (Stinson et al., 2003) and obstetrical anesthesia (Halpern, Darani, Douglas, Wight wight 1 n. Obsolete A living being; a creature. [Middle English, from Old English wiht; see wekti- in Indo-European roots. , & Yee, 2004). A particularly weak area across all fields has been the description of randomization procedures. The quality of reporting is not the same thing as quality of research (Huwiler-Muntener, Juni, Junker, & Egger, 2002). For example, a highly flawed study with a small sample could score quite high (but not perfectly) on the CONSORT or the NMECI as long as the reporting of the flaws was thorough. Conversely con·verse 1 intr.v. con·versed, con·vers·ing, con·vers·es 1. To engage in a spoken exchange of thoughts, ideas, or feelings; talk. See Synonyms at speak. 2. , an excellent trial might be reported incompletely because of the authors' training or because of editorial conventions. A practical problem in reporting RCTs is that many journals, particularly those oriented to medical specialties Medical Specialties See also anatomy; disease and illness; drugs; health; remedies; surgery. adenography the science of the description of glands. — adenographic, adj. , have stringent page requirements, resulting in loss of important detail. Specifically, introductions tend to be brief, and therefore the state of science in the area of concern and the theoretical background of the study tend to be underreported. Systematic study of the relationship between quality of reporting and quality of research is difficult, given the inherent problem of judging the quality of research that is poorly reported. The moderately positive correlation between year and score on the NMECI suggests an improvement of RCT reporting in occupational therapy (the more recent the article, the higher the degree of CONSORT compliance). With the growth of the field, there has been an increase in the quality of research. As the field of occupational therapy expands and as investigative efforts increase, research results must be reported in such a way that practitioners and fellow researchers can determine their veracity (Holm, 2000). In order to provide the most transparent information about "why a study was undertaken and how it was conducted and analyzed" (Moher et al., 2001, p. 657), the CONSORT can be used to guide authors in writing thorough research reports forming a basis for evidence-based practice. Development of the Nelson-Moberg Expanded CONSORT Instrument When considering the overall score as the principal indicator of quality, the interrater reliability of the NMECI is excellent with "almost perfect strength of agreement." This may have been due to the fact that the authors met before the analyses to discuss the meaning for each item based on The Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration (Altman et al., 2002). The descriptions presented for each item also may have contributed to the high overall agreement. It is important to note that the CONSORT was not modified by the authors, but rather expanded into a more detailed checklist based solely on the CONSORT elaboration document. Moher et al. (2001) stated that an "iterative it·er·a·tive adj. 1. Characterized by or involving repetition, recurrence, reiteration, or repetitiousness. 2. Grammar Frequentative. Noun 1. process" makes the CONSORT Statement a continually evolving instrument (p. 658). However, 3 of the 22 ICCs reflecting agreement on the original CONSORT items and 29 of the kappas on NMECI sub-items were unacceptably low (fair, slight, or poor agreement). Sub-item 10, "Describe the broad approach taken to study the problem," requires subjective analysis to receive credit. The term "broad" does not define what is necessary to report in describing the process of the intervention. In the future, this term could be more clearly defined to increase the reliability of the item, by naming specific aspects to be included in the report, such as a general description of the intervention and participants. Sub-item 15 also uses general terms, lacking specific instruction; hence raters relied on subjective judgments. Sub-items 45, 46, 77, 189, and 197 may have had low kappas and poor agreement because of a difference in training and knowledge between the two raters in regard to statistical methods. A limitation of this study is that EM did not fully understand certain statistical procedures or methodologies. NMECI sub-items were taken directly from the CONSORT elaboration document, which assumes users' prior knowledge of principles of research design. Future Research Although the total score of the NMECI proved to be highly reliable, sub-items with low kappa values should be revised prior to future research. In some cases, the wording of the sub-items should be changed to increase clarity. In other cases, the explanation of the sub-item should be amplified to cover previously unanticipated problems of interpretation. Also, RCT experts could offer systematic feedback on the content validity content validity, n the degree to which an experiment or measurement actually reflects the variable it has been designed to measure. of the NMECI sub-items and the procedures used in scoring. Once refined, the NMECI could be used for analyzing the quality of reporting in occupational therapy RCTs not cataloged in the OTSeeker database with "occupational therapy" in the title; for comparing quality of reporting across fields; and for comparing quality of reporting across time in the future. The NMECI could also be used to investigate whether incompleteness of reporting is associated with large effect sizes in occupational therapy research, as was done by Stinson et al. (2003) in psychology RCTs. A final recommendation is the development of supplementary sub-items dealing with issues commonly arising in occupational therapy and related fields involving complex interactions between interventionists and research participants. For example, issues dealing with intervention fidelity differ greatly between drug studies and occupational therapy intervention studies intervention studies, n.pl the epidemiologic investigations designed to test a hypothesized cause and effect relation by modifying the supposed causal factor(s) in the study population. (Nelson & Mathiowetz, 2004). Implications for Occupational Therapy The main significance of this study is that the reporting of occupational therapy RCTs can and should be improved in the future, in support of evidence-based practice. Tickle-Degnen (1999, p. 538) pointed out that evidence-based practice depends in part upon understanding research procedures and upon evaluating the validity of the study. Incompletely described procedures hamper the practitioner's search for evidence by casting doubt on the research conclusions. For example, the practitioner cannot know to whom the results can be generalized if the sample is described incompletely. For another example, the practitioner cannot trust the findings if bias might have been introduced by randomization measurement procedures that are described incompletely. Incompletely described procedures leave the practitioner in a state of doubt. Does the lack of detail reflect design flaws or simply errors of writing? There is no way of knowing the validity of the procedures without knowing what the procedures were. This study alerts researchers as well as consumers of research to past problems and the need for future improvement. The usefulness of a detailed interpretation of the CONSORT Statement such as the NMECI depends on one's perspective. Those designing RCTs would be well advised to use such a list as a double-check to their memories. Also, authors have special access to information concerning a trial, but often struggle with editors as to inclusion of details. A detailed interpretation such as the NMECI could help authors and editors negotiate what is critical information, and what is not. This could be helpful even if a 201-sub-item checklist is judged too detailed for a standard review. Although it is unlikely that clinicians would have time to apply such a detailed analysis, it might be helpful to them if such an analysis was provided by a Web site. So much work goes into conducting an RCT that the extra time involved in reporting it well is worthwhile. References (14 studies analyzed in this paper are marked by *) Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., et al. (2002). The revised CONSORT Statement for reporting randomized trials: Explanation and elaboration. Annals of Internal Medicine Annals of Internal Medicine (Ann Intern Med) is an academic medical journal published by the American College of Physicians (ACP). It publishes research articles and reviews in the area of internal medicine. Its current editor is Harold C. Sox. , 134, 663-694. Retrieved February 11, 2003, from http://consortstatement.org American Occupational Therapy Association. (1998). Standards of practice for occupational therapy. American Journal of Occupational Therapy, 52, 866-869. * Bach, D., Bach, M., Bohmer, F., Fruhwald, T., & Grilc, B. (1995). Reactivating occupational therapy: A method to improve cognitive performance in geriatric geriatric /ger·i·at·ric/ (jer?e-at´rik) 1. pertaining to elderly persons or to the aging process. 2. pertaining to geriatrics. ger·i·at·ric adj. 1. patients. Age and Ageing, 24, 222-225. Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D., Olkin, I., et al. (1996). Improving the quality of reporting of randomized controlled trials. The CONSORT Statement. JAMA, 276, 637-639. * Clark, F., Azen S. P., Zemke, R., Jackson, J., Carlson, M., Mandel D., et al. (1997). Occupational therapy for independent-living older adults. A randomized controlled trial A randomized controlled trial (RCT) is a scientific procedure most commonly used in testing medicines or medical procedures. RCTs are considered the most reliable form of scientific evidence because it eliminates all forms of spurious causality. . JAMA, 278, 1321-1326. Eliasziw, M., Young, S. L., Woodbury, M. G., & Fryday-Field, K. (1994). Statistical methodology for the concurrent assessment of interrater and intrarater reliability: Using goniometric go·ni·om·e·ter n. 1. An optical instrument for measuring crystal angles, as between crystal faces. 2. A radio receiver and directional antenna used as a system to determine the angular direction of incoming radio signals. measurements as an example. Physical Therapy, 74, 777-788. Esplen, M. J, Garfinkel, P. E., Olmsted, M., Gallop, R. M., & Kennedy, S. (1998). A randomized controlled trial of guided imagery Guided Imagery Definition Guided imagery is the use of relaxation and mental visualization to improve mood and/or physical well-being. Purpose in bulimia nervosa bulimia nervosa Eating disorder, mostly in women, in which excessive concern with weight and body shape leads to binge eating followed by compensatory behaviour such as self-induced vomiting or the excessive use of laxatives or diuretics. . Psychological Medicine, 28, 1347-1357. Friedman, L. M., Furberg, C. D., & DeMets, D. L. (1998). Fundamentals of clinical trials (3rd ed.). New York New York, state, United States New York, Middle Atlantic state of the United States. It is bordered by Vermont, Massachusetts, Connecticut, and the Atlantic Ocean (E), New Jersey and Pennsylvania (S), Lakes Erie and Ontario and the Canadian province of : Springer-Verlag. * Gauthier, L., Dalziel, S., & Gauthier, S. (1987). The benefits of group occupational therapy for patients with Parkinson's Disease Parkinson's disease or Parkinsonism, degenerative brain disorder first described by the English surgeon James Parkinson in 1817. When there is no known cause, the disease usually appears after age 40 and is referred to as Parkinson's disease. . American Journal of Occupational Therapy, 41, 360-365. * Gilbertson, L., Langhorne, P., & Walker, A. (2000). Domiciliary domiciliary pertaining to a household. domiciliary calls professional veterinary calls made to patients at their owners' residences. Called also house calls. occupational therapy for patients with stroke discharged from hospital: Randomised Adj. 1. randomised - set up or distributed in a deliberately random way randomized irregular - contrary to rule or accepted order or general practice; "irregular hiring practices" controlled trial. BMJ BMJ n abbr (= British Medical Journal) → vom BMA herausgegebene Zeitschrift , 320, 603-606. Halpern, S. H., Darani, R., Douglas, M. J., Wight, W., & Yee, J. (2004). Compliance with the CONSORT checklist in obstetric anaesthesia anaesthesia anesthesia. randomised controlled trials. International Journal of Obstetric Anesthesia, 13, 207-214. * Helewa, A., Goldsmith, C., Lee, P., Bombardier, C., Hanes, B., Smythe, H., et al. (1991). Effects of occupational therapy home service on patients with rheumatoid arthritis rheumatoid arthritis Chronic, progressive autoimmune disease causing connective-tissue inflammation, mostly in synovial joints. It can occur at any age, is more common in women, and has an unpredictable course. . Lancet lancet /lan·cet/ (lan´set) a small, pointed, two-edged surgical knife. lan·cet n. , 337, 1453-1456. Herbison, P. (2005). The reporting quality of abstracts of randomized controlled trials submitted to the ICS (1) (Internet Connection Sharing) A Windows feature that enables two or more computers to share one Internet connection. First introduced in Windows 98 Second Edition, sharing is accomplished with network address translation (NAT), which is the common method. meeting in Heidelberg. Neurourology and Urodynamics urodynamics /uro·dy·nam·ics/ (-di-nam´iks) the dynamics of the propulsion and flow of urine in the urinary tract.urodynam´ic urodynamics the dynamics of the propulsion and flow of urine in the urinary tract. , 24, 21-24. Holm, M. B. (2000). 2000 Eleanor Clarke Slagle lecture--Our mandate for the new millennium: Evidence-based practice. American Journal of Occupational Therapy, 54, 575-585. Huwiler-Muntener, K., Juni, P., Junker, C., & Egger, M. (2002). Quality of reporting of randomized trials as a measure of methodologic quality. JAMA, 287, 2801-2804. Kielhofner, G., Hammel, J., Finlayson, M., Helfrich, C., & Taylor, R. R. (2004). Documenting outcomes of occupational therapy: The Center for Outcomes Research and Education. American Journal of Occupational Therapy, 58, 15-23. Landis, J. R., & Koch G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363-374. * Liddle, J., March, L., Carfrae, B., Finnegan, T., Druce, J., Schwarz, J., et al. (1996). Can occupational therapy intervention play a part in maintaining independence and quality of life in older people? A randomised controlled trial. Australian and New Zealand New Zealand (zē`lənd), island country (2005 est. pop. 4,035,000), 104,454 sq mi (270,534 sq km), in the S Pacific Ocean, over 1,000 mi (1,600 km) SE of Australia. The capital is Wellington; the largest city and leading port is Auckland. Journal of Public Health, 20, 574-578. * Logan, P. A., Ahern, J., Gladman, J. R. F., & Lincoln, N. B. (1997). A randomized controlled trial of enhanced social service occupational therapy for stroke patients. Clinical Rehabilitation rehabilitation: see physical therapy. , 11, 107-113. MacPherson, H., White, A., Cummings, M., Jobst, K., Rose, K., & Niemtzow, R. (2001). Standards for reporting interventions in controlled trials of acupuncture: The STRICTA recommendations. Medical Acupuncture Medical acupuncture is a simplified version of traditional Chinese acupuncture that is learned by Western medical practitioners. History of medical acupuncture Medical acupuncture was created for Western practitioners such as medical doctors, physiotherapists, 13(3). Retrieved March 3, 2003, from http://www.medicalacupuncture.org/aama_marf/journal/ vol13_3/stricta.html Mann, W. C., Ottenbacher, K. J., Fraas, L., Tomita, M., & Granger, C. V. (1999). Effectiveness of assistive technology Hardware and software that help people who are physically impaired. Often called "accessibility options" when referring to enhancements for using the computer, the entire field of assistive technology is quite vast and even includes ramp and doorway construction in buildings to support and environmental interventions in maintaining independence and reducing home care costs for the frail elderly frail elderly, n.pl older persons (usually over the age of 75 years) who are afflicted with physical or mental disabilities that may interfere with the ability to independently perform activities of daily living. . Archives of Family Medicine, 8, 210-217. Moher, D., Altman, D. G., Schulz, K. F., & Elbourne, D. R. (2004). Opportunities and challenges for improving the quality of reporting clinical research: CONSORT and beyond. Canadian Medical Association Journal The Canadian Medical Association Journal (CMAJ) is a general medical journal that is published biweekly by the Canadian Medical Association (CMA). It is considered to be one of the top six general medical journals; the others being the , 171, 349-350. Moher, D., Jones, A. J., & Lepage, L. for the CONSORT Group. (2001). Use of CONSORT Statement and quality of reports of randomized trials: A comparative before-and after evaluation. JAMA, 285, 1992-1995. Moher, D., Schulz, K. F., & Altman, D. G. (2001). The CONSORT Statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. Annals of Internal Medicine, 134, 657-662. National Institutes of Health, Division of Research Grants, Research Analysis and Evaluation Branch. (1979). NIH "Not invented here." See digispeak. NIH - The United States National Institutes of Health. inventory of clinical trials: Fiscal year 1979 (Vol. 1). Bethesda, MD: Author. * Neistadt, M. E. (1992). Occupational therapy treatments for constructional deficits. American Journal of Occupational Therapy, 46, 141-148. Nelson, D. L., & Mathiowetz, V. (2004). Randomized controlled trials to investigate complex occupational therapy research questions. American Journal of Occupational Therapy, 58, 24-34. * Nuse Carlsen, P. (1975). Comparison of two occupational therapy approaches for treating the young cerebral palsied pal·sied adj. 1. Affected with palsy. 2. Trembling or shaking. Adj. 1. palsied - affected with palsy or uncontrollable tremor; "palsied hands" child. American Journal of Occupational Therapy, 29, 267-272. * Oerlemans, H. M., Oostendorp, R. A. B., de Boo, T., van der Laan, L., Severens, J. L., & Goris, J. S. (2000). Adjuvant adjuvant /ad·ju·vant/ (aj?dbobr-vant) (a-joo´vant) 1. assisting or aiding. 2. a substance that aids another, such as an auxiliary remedy. 3. physical therapy versus occupational therapy in patients with reflex sympathetic dystrophy/complex regional pain syndrome type I. Archives of Physical Medicine & Rehabilitation, 81, 49-56. OTseeker: Occupational Therapy Systematic Evaluation of Evidence. (2003). Retrieved November 5, 2003, from http://www.otseeker.com * Parker, C., Gladman, J., Drummond, A., Dewey, M., Lincoln, N., Barer, D., et al. (2001). A multicentre randomized controlled trial of leisure therapy and conventional occupational therapy after stroke. Clinical Rehabilitation, 15, 42-52. * Przybylski, B., Dumont, E., Watkins, M., Warren, S., Beaulne, A., & Lier, D. (1996). Outcomes of enhanced physical and occupational therapy service in a nursing home setting. Archives of Physical Medicine & Rehabilitation, 77, 554-561. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 2, 420-428. Stern, P. (2005). A holistic approach holistic approach A term used in alternative health for a philosophical approach to health care, in which the entire Pt is evaluated and treated. See Alternative medicine, Holistic medicine. to teaching evidence-based practice.American Journal of Occupational Therapy,59,157-164. Stinson, J. N., McGrath, P. J., & Yamada, J. T. (2003). Clinical trials in the Journal of Pediatric Psychology: Applying the CONSORT Statement. Journal of Pediatric Psychology, 28, 159-167. * Strong, J. (1998). Incorporating cognitive-behavioral therapy Cognitive-Behavioral Therapy Definition Cognitive-behavioral therapy is an action-oriented form of psychosocial therapy that assumes that maladaptive, or faulty, thinking patterns cause maladaptive behavior and "negative" emotions. with occupational therapy: A comparative study with patients with low back pain. Journal of Occupational Rehabilitation, 8, 61-71. Tickle-Degnen, L. (1999). Organizing, evaluating, and using evidence in occupational therapy practice. American Journal of Occupational Therapy, 53, 537-539. * Walker, M. F., Gladman, J. R. F., Lincoln, N. B., Siemonsoma, P., & Whiteley. T. (1999). Occupational therapy for stroke patients not admitted to hospital: A randomized controlled trial. Lancet, 354, 278-280. Emily Moberg-Mogren, MOT, OTR/L OTR/L Occupational Therapist, Registered, Licensed , is Staff Occupational Therapist, Rainbow and Children's Hospital A children's hospital is a hospital which offers its services exclusively to children. The number of children's hospitals proliferated in the 20th century, as pediatric medical and surgical specialties separated from internal medicine and adult surgical specialties. , Cleveland, Ohio "Cleveland" redirects here. For the Cleveland metropolitan area, see . For other uses, see Cleveland (disambiguation). Cleveland is a city in the U.S. state of Ohio and the county seat of Cuyahoga County, the most populous county in the state. . David L. Nelson, PhD, OTR/L, FAOTA FAOTA Fellow of the American Occupational Therapy Association , is Professor, Department of Occupational Therapy, Collier Building, School of Allied Health, Medical College of Ohio, 3015 Arlington Avenue, Toledo, Ohio
Table 1. Mean Compliance With CONSORT Items and Overall CONSORT
(Based on DN's Ratings), and Intraclass Correlation Coefficients
Reflecting Levels of Interrater Agreement
CONSORT NMECI
Section/Topic Item Sub-Items Mean
Title/Abstract 1 1-3 1.79
Background 2 4-10 3.71
Participants 3 11-23 9.93
Interventions 4 24-30 3.50
Objectives 5 31-32 2.00
Outcomes 6 33-52 13.00
Sample size 7 53-66 6.64
Randomization (a) 8 67-79 8.00
Randomization (b) 9 80-83 0.43
Randomization (c) 10 84-89 0.57
Blinding 11 90-102 1.07
Statistics 12 103-115 4.93
Participant flow 13 116-132 10.50
Recruitment 14 133-138 3.14
Baseline data 15 139-145 4.14
Numbers analyzed 16 146-153 6.29
Outcomes 17 154-166 6.93
Ancillary analyses 18 167-175 8.57
Adverse events 19 176-180 0.00
Interpretation 20 181-189 5.00
Generalizability 21 190-197 3.08
Overall evidence 22 198-201 1.00
Overall Score 1-201 104.23 32.93
Section/Topic SD ICC
Title/Abstract 1.12 1.00
Background 1.59 .78
Participants 2.02 .73
Interventions 1.56 .88
Objectives 0.00 -- (d)
Outcomes 3.98 .90
Sample size 3.41 .95
Randomization (a) 5.46 .96
Randomization (b) 1.16 1.00
Randomization (c) 1.09 1.00
Blinding 0.92 .31
Statistics 4.80 .88
Participant flow 5.75 .98
Recruitment 1.88 .98
Baseline data 2.57 .91
Numbers analyzed 2.49 .96
Outcomes 5.14 .95
Ancillary analyses 0.85 .12
Adverse events 0.00 0.00
Interpretation 1.66 .88
Generalizability 1.33 .89
Overall evidence 1.24 .53
Overall Score .95
(a) Sequence generation
(b) Allocation concealment
(c) Implementation
(d) Perfect agreement that all articles met this criterion; ICC
cannot be calculated (no variance).
Table 2. Kappa Statistics Checking Levels of Agreement Between
Raters on 197 NMECI Sub-Items. Judgments Concerning Levels
are Based on Guidelines Provided by Landis and Koch (1977).
Four NMECI Sub-Items Were Tested by Intraclass Correlation
Coeffcients.
Number of
Range of NMECI
Kappa Sub-Items
Levels of Agreement
Poor <0 2
Slight 0-.20 24
Fair .20-.40 3
Moderate .40-.60 21
Substantial .60-.80 25
Almost perfect or perfect .80-1 101
Perfect agreement that all 14 Cannot be
articles met the criteria computed 15
Perfect agreement that all 14 Cannot be
articles did not meet the criteria computed 6
Table 3. Kappa Values and Percentages of Agreement for the
29 Nelson-Moberg Sub-Items With Kappa Values Less Than .40
(Less Than Moderate Agreement)
Nelson-Moberg Sub-Items Kappa % Agree
10 Broad approach taken to studying the problem .34 64%
14 Types of settings for recruitment 0 86%
15 Description of trial participants -.08 86%
18 Types of settings where interventions
tool place 0 93%
26 How the placebo was described 0 86%
29 Types of interventionists 0 86%
45 Properties of scales used in primary
outcomes .14 57%
46 Properties of scales used in planned .11 50%
secondary outcomes
65 Group statistical methods for interim
analyses 0 71%
77 Methods used for blocking .31 64%
93 Whether or not data analysts were blinded 0 93%
94 Procedures for blinding participants 0 93%
95 Procedures for blinding care providers 0 93%
96 Procedures for blinding outcome evaluators 0 93%
98 Mechanism for blinding interventions 0 93%
106 Basis for data analysis for primary outcomes 0 43%
110 Basis for data analysis for secondary
outcomes 0 64%
145 Whether or not the trial protocol called
for statistical adjustment .38 71%
153 Refrain from intention to treat analyses
for adverse effects 0 93%
169 Why each subgroup was examined 0 93%
171 Confidence interval for interactions 0 93%
173 Refrain from selective reporting of
subgroup analyses 0 93%
176 Information about harm in intervention 0 93%
177 Estimates of the frequency of sever
adverse events 0 93%
178 Number of affected participants and
adverse events 0 93%
179 Reasons for discontinuation secondary to
adverse events 0 93%
180 Operational definitions for adverse events 0 93%
189 Statements of caution if multiplicity exists .14 57%
197 Control group risk -.08 86%
Figure 1. Nelson-Moberg Expanded CONSORT Instrument elaboration of
CONSORT item #3 into 13 sub-items.
Nelson-Moberg Expanded CONSORT
Section Instrument Elaboration Sub-Items Based
Topic Item on CONSORT Explanation Document
METHODS 3 A. Eligibility criteria for
Participants participants
11) -- define eligibility criteria
12) -- describe the method of
recruitment (e.g., referral
or self-selection through
advertisements)
13) -- describe the geographical
locations) for recruitment
(e.g., city/country as
needed)
14) -- describe the types) of
setting, (e.g., community,
office practice, hospital
clinic, hospital)
15) -- describe the trial
participants
Sub-total -- /5
B. The settings and locations where the
data were collected
16) -- report the number of
intervention settings
17) -- report the number of data
collection sites
18) -- report the types of
settings where intervention
took place (e.g.,
community, office practice,
hospital clinic, hospital
inpatient)
19) -- report the geographical
locations of the
intervention sites
(e.g., city/country)
20) -- report the types) of
settings where data
collection took place
21) -- report the geographical
locations of the data
collection sites
22) -- report the number of care
providers
23) -- report the types of care
providers (e.g.,
credentials)
Sub-total -- / 8
Total -- /13
Section
Topic Interpretation/Explanation
METHODS A. overlaps with part of CONSORT 15,
Participants and is scored in both sections.
(11) need not distinguish between
exclusion and inclusion criteria
(13) if used, advertisements must
imply the geographical area (e.g., by
identifying the publications or
posting sites)
(15) may be reported in Results section
(overlap with CONSORT # 15)
A. (13)-(14), B. (18)-(21) may be
reported in one place if the settings
were same
(16)-(17) credit if clearly only
one site
(22)-(23) credit if the only care
providers are the interventionists
and if the interventionists are
described in CONSORT # 4 (27)-(28)
|
|
||||||||||||||||||||||
Printer friendly
Cite/link
Email
Feedback
Reader Opinion