Printer Friendly

Statistical Literacy Among Academic Pathologists: A Survey Study to Gauge Knowledge of Frequently Used Statistical Tests Among Trainees and Faculty.

The practice of both clinical and anatomic pathology requires familiarity with statistical concepts. The use of sophisticated statistical methods is vital for the modern academic pathologist from nearly all pathology disciplines, ranging from the critical appraisal of evidence and interpretation of results as dictated by evidence-based medicine (EBM), research (eg, genomics, biomarker studies, translational research, and health services research), and for laboratory operations, quality assurance, and improvement initiatives. All pathologists are also now required to complete a quality improvement project, which requires statistical knowledge for maintenance of certification. Additionally, reviewing manuscripts for journals requires some basic level of statistical familiarity to ensure that the data analysis methods presented in manuscripts are appropriate to address a research question and support the inferences made by authors.

Statistical literacy can be defined as understanding the statistical tests and terminology needed for the design, analysis, and conclusions of original research or laboratory testing. (1) Several studies have found that physicians from a variety of medical disciplines have a poor understanding of statistics, (2-7) even while believing such knowledge is important for their work. (2,8) Evidence-based medicine training has been integrated into all levels of medical education and is mandated by the Accreditation Council for Graduate Medical Education (ACGME) for residents, but the ACGME has not provided specific guidelines regarding the content or methods for EBM training, (9) and no published EBM curricula have been directed specifically to pathology. Although there are general concepts that underlie all statistics, different disciplines may emphasize particular techniques, and the use of statistical methods changes over time. (10) Knowledge of how frequently statistical tests are used in published studies from a particular medical discipline could be useful in designing curricula and assessment tools specifically tailored for EBM in pathology.

Several studies have investigated how frequently different statistical methods are used in research from a variety of medical disciplines (2,6,11,12); however, little is known about the understanding and use of statistics by pathology trainees and practicing pathologists. The specifics of statistical knowledge required to be an effective pathologist is also not known and probably varies by scope of practice. The objective of this study was 2-fold: (1) to determine the frequency of different statistical tests most commonly used in research studies published in pathology journals, and (2) to assess the degree of understanding pathologists profess to have of the most common statistical methods.

METHODS

Literature Review on Reporting Frequency of Statistical Tests

We conducted a focused literature review of 1100 research articles to determine the types of statistical tests commonly reported in research articles published in pathology journals. We obtained and audited a random sample of 100 original articles per journal published during 2015 in 11 pathology journals: American Journal of Clinical Pathology (AJCP), The American Journal of Pathology (AJP), American Journal of Surgical Pathology (AJSP), Archives of Pathology & Laboratory Medicine (Archives), Cancer Cytopathology (CAC), Clinica Chimica Acta (CCA), Clinical Chemistry (CC), Diagnostic Cytopathology (DC), Journal of Pathology (JP), Laboratory Investigation (LI), and Modern Pathology (MP). These journals provide broad coverage of pathology and laboratory medicine. Small case series (<20 cases), reviews, and commentaries were excluded. Data were extracted by one author (R.L.S.). In some cases, we included articles from 2014 if the journal did not publish 100 original articles in 2015. Each article was reviewed to determine whether statistical tests were used and, if so, the specific techniques were recorded. Articles were classified as using no statistical methods, using descriptive methods, or using inferential methods. Articles were classified as using inferential methods if they reported P values but did not report the statistical test used. In such cases, we inferred the statistical test from the type of data; these cases were relatively rare (~1%). An article was identified as using descriptive statistical methods if it used either graphical displays of data (eg, histograms, box and whisker charts, survival curves, or scatterplots) and/or calculated confidence intervals or interquartile ranges. Tabulation of numerical data, calculation of percentages, means, or medians (without distributional information) did not qualify as statistical analysis. We recorded the statistical tests used in each article as well as the statistical packages used. We classified journals into 5 categories: general (AJCP, Archives), basic science (AJP, JP, LI), translational (AJSP, MP), cytopathology (CAC, DC), and chemistry (CC, CCA).

Pathologist Survey on Knowledge of Statistical Tests

The Knowledge of Statistical Methods Survey was developed by 2 pathologists and reviewed by an educational expert to assess knowledge level of statistical techniques identified in the literature review by pathology faculty and trainees. We solicited pathology programs to participate in the survey by placing an announcement on a listserv for residency program directors. The following pathology programs agreed to participate in the study: Cleveland Clinic (Cleveland, Ohio), Icahn School of Medicine at Mount Sinai Medical Center (New York, New York), Thomas Jefferson University Hospital (Philadelphia, Pennsylvania), University of Missouri (Columbia), University of Utah (Salt Lake City), East Carolina University (Greenville, North Carolina), University of California at San Francisco (San Francisco), Loyola University (Maywood, Illinois), and Baylor College of Medicine (Houston, Texas). The survey was administered from the University of Utah by using Qualtrics software (Qualtrics LLC, Provo, Utah). The survey questionnaire is presented in the Supplemental Digital Content (supplemental digital content, containing the survey and 3 tables, is available for this article at www.archivesofpathology. org in the February 2017 table of contents). Potential respondents were provided by site coordinators at each site. Potential respondents were sent an invitation email with a Web link to the survey in January/February of 2016. Nonresponders were sent 2 reminder invitations at 1-week intervals. The study was approved by the Institutional Review Board at the University of Utah.

Statistical Training by Program

Admission requirements for statistics and calculus for all US medical schools were obtained from the Web site of the American Association of Medical Colleges (AAMC). (13) Data on statistics training in medical school programs were also obtained from the AAMC. Program directors for each pathology residency program were surveyed regarding statistics and EBM training at each of the participating institutions.

Assessment of Statistical Knowledge

Knowledge of statistical methods was evaluated on a 3-point scale: 0 if the respondent had no useful knowledge of the test (had never heard of a method or only recognized the name), 1 if the respondent had basic knowledge (could interpret the test but was unable to perform it himself/herself), or 2 if the respondent had advanced knowledge (could perform and interpret the test himself/herself). A respondent statistical knowledge (RSK) variable was created by averaging knowledge of statistical methods ratings for each individual. The RSK score provided a measure of the statistical knowledge of an individual respondent averaged over all tests. The RSK score could range from 0 (no knowledge of any test) to 2 (able to perform and interpret all tests). Intermediate RSK scores are more difficult to interpret, but higher RSK scores indicate that a respondent has more skill with more tests. For example, an RSK score of 1.0 could mean that a respondent had basic knowledge (could interpret but not perform) of all tests, or that he/she had a range of knowledge that averaged out to 1.0 over all tests. We also created a test statistical knowledge (TSK) variable. The TSK score was calculated by averaging the responses for an individual test. The TSK score could range from 0 (all respondents indicated no knowledge of the test) to 2 (all respondents could perform the test and interpret the results without assistance).

Statistical Analysis

Respondent statistical knowledge scores were compared by using the Kruskal-Wallis test. The [chi square] test was used to evaluate the association between categorical variables. The correlation between TSK and test usage was evaluated by using the Spearman rank correlation coefficient. Statistical calculations were performed by using Stata 13 (Stata LLC, College Station, Texas).

RESULTS

Statistics Methods Used in the Research Literature

We abstracted data from 1100 original research articles (Table 1), of which 888 (81%) reported the use of statistical methods. Of the studies reporting statistical analysis, 770 of 888 (86%) used statistical tests (inferential statistics), and the remainder used descriptive statistics. The use of statistical methods ranged from 140 of 200 (70%) in general journals to 282 of 300 (94%) in basic science journals. Data comparing statistical test use in individual journals are presented in Supplemental Material Table 1. Among individual journals, statistical method use (descriptive or inferential) ranged from 61 of 100 (Modern Pathology) to 98 of 100 (American Journal of Pathology). There were significant differences in the pattern of test use among journal categories ([[chi square].sub.48] = 313, P < .001). That is, particular tests were used more frequently in different categories of journals. There was no significant difference in test usage among the journals within the general, translational, or cytology categories ([chi square] test, P > .43 for each category). There was a significant difference in test use by journals in the basic science and chemistry categories (P < .01).

We identified 18 statistical tests that appeared in at least 1% of studies (Table 1). The tests are described in Supplemental Material Table 2. The most frequently used tests were the t test (used in 297 of 1100 articles, or 27%), regression or analysis of variance (209 of 1100 articles, or 19%), [chi square] test (198 of 1100 articles, or 18%), Mann-Whitney test (167 of 1100 articles, 15.2%), Fisher exact test (143 of 1100 articles, 13%), and tests for survival analysis (154 of 1100, or 14%).

Use of Statistical Software

Seven hundred twenty-six of 1100 articles (66%) reported the type of software used for statistical analysis. Twenty-nine different software packages were used. Statistical Program for the Social Sciences (SPSS; IBM Corp, Armonk, New York), GraphPad (GraphPad Software Inc, La Jolla, California), and Statistical Analysis System (SAS; SAS Institute Inc, Cary, North Carolina) were the most commonly used software programs (Table 2). Data comparing statistical software use in individual journals are presented in Supplemental Material Table 3.

Survey Response

We surveyed faculty and trainees from 9 pathology residency programs regarding their knowledge of statistical methods. Program directors at each site provided an email list of potential targets for the survey (faculty and trainees). This produced a list of 879 potential respondents (580 faculty and 299 trainees) who were sent the survey. The overall response rate to the survey was 365 of 879 (42%). The trainee response rate (136 of 299, 45%) was not significantly greater ([[chi square].sub.1] = 1.2, P = .28) than the faculty response rate (229 of 580, 39%) (Table 3).

Characteristics of Respondents

Ninety-seven percent of respondents indicated that they were MD/DO trained. Of the 365 who responded to the survey, 229 (63%) were faculty, and 136 (37%) were trainees (Table 3). One hundred twelve (56%) were primarily focused on anatomic pathology, 64 (32%) were focused on clinical pathology, and 25 (12%) indicated that their work was balanced between the two. One hundred twenty-seven respondents (41%) indicated that they had earned an advanced degree in addition to the MD degree. Of these, 95 (72%) had earned a PhD degree. Faculty respondents were evenly distributed across the categories for years in practice. Two hundred sixty-four of 321 respondents (82%) indicated that they conducted research studies for publication; 223 of 309 (73%) said that publication was required for professional advancement. Respondents differed in their use of statistics. Of 314 respondents, 208 (66%) indicated that they mainly needed statistics for research, 57 (18%) primarily used statistics for clinical work or operations, and 49 (16%) indicated that they had no need for statistics in their work. More than half of respondents (157 of 265, 59%) said that they rely on a professional statistician or a knowledgeable colleague to perform statistical analyses, while 92 (35%) perform statistical analyses themselves. The respondents had a wide range of research experience. For trainees, the median number of published articles was 5 (range: 0-45, interquartile range: 2-10). For faculty, the median number of published articles was 32 (range: 2-380, interquartile range: 16-91). Almost all respondents (302 of 310, 97%) indicated that a better understanding of statistics would be helpful.

Statistical Training

Statistical training can occur at several points during training (before medical school, during medical school, during residency or fellowship, or from other graduate training) or after training through self-study and experience. Fifty-nine percent of respondents had taken a statistics course at some point in their career. Having had a statistics course was strongly associated with having an advanced degree other than an MD degree ([[chi square].sub.1] = 14.6, P < .001). Only 6 of 148 medical schools (4%) list a course in statistics as an admission requirement, and only 34 of 148 (23%) recommend a course in statistics before enrollment. The distribution of requirements for calculus (14 schools require, 31 recommend) was not statistically different from the distribution of requirements for statistics ([[chi square].sub.1] = 3.5, P = .18). In 2013, most medical schools reported that they required their medical students to complete a biostatistics course (140 of 154), a course in clinical and translational research (134 of 154), an epidemiology course (139 of 154), a course in literature evaluation (137 of 154), a course in EBM (140 of 154), or a course in research methods (140 of 154). Our sample of residency programs showed wide variation in statistical training (Table 4). Statistical training is mostly informal (journal club, rotations), but some programs provide formal lectures.

Knowledge of Statistical Methods

The average TSK score (average knowledge for all respondents for a particular test) was 0.70 and ranged from 0.3 (Kruskal-Wallis test) to 1.33 (Student t test). The TSK scores data are summarized in Table 5. The median RSK score was 0.56 (interquartile range: 0.25-1.06; Figure 1, A through C). Overall, respondents' knowledge of tests correlated with the frequency with which tests are used in journal articles (Spearman [rho] = 0.74, P = .003; Figure 2). For example, respondents reported the greatest knowledge of the [chi square] test and Student t test, which were also the most widely used methods in the examined journals (Table 5, Figure 2).

Factors Associated With Knowledge of Statistical Tests

The factors associated with statistical knowledge are summarized in Table 6. There was no significant difference in the RSK scores between trainees and faculty (P = .14), and there was no association between RSK scores and years in practice (P = .20). Greater statistical knowledge was associated with a focus in clinical pathology rather than anatomic pathology (P < .001), having an advanced degree other than an MD degree (P < .001), having taken a statistics course (P < .01), and conducting research studies for publication (P < .001). The type of advanced degree (PhD versus master's level) had a borderline association with statistical knowledge (P = .06). Trainees at programs with formal statistics training had higher RSK scores than those without (Kruskal-Wallis test, P = .03).

DISCUSSION

A list of statistical tests that are frequently used in the pathology literature provides a starting point for understanding the type of statistical knowledge that is useful for pathologists, and provides a direction for how to train pathologists in statistics and EBM. Our study found that a small number of statistical tests are frequently used in pathology studies. A working knowledge of approximately 12 tests would enable a pathologist to understand the statistical methods used in most research published in pathology journals.

The frequency of test use that we report here is similar to what was reported in a previous study focused on family and emergency medicine, (11) as well as a recent study of cytopathology journals. (14) However, pathology is a varied discipline, and our study discovered that different categories of pathology journals show significant differences in the type of statistical tests that they commonly report. For example, basic science and chemistry journals tend to use analysis of variance and regression, translational journals tend to use survival methods and [kappa] statistics, and cytopathology journals tend to use categorical methods. This suggests that optimal statistical training of pathologists may need to be focused on different areas of specialization.

Our survey found that familiarity of tests by pathology faculty and trainees highly correlated with the frequency of tests commonly reported in the literature. Our RSK scale is relative and does not provide an absolute measurement of statistical literacy, but our findings suggest that statistical literacy is lacking for trainees and faculty alike. As indicated in Table 5, most pathologists can interpret but not perform simple tests such as the [chi square] and t tests. Our survey found that most pathologists (59%) solicit the help of a knowledgeable colleague or a professional statistician to perform statistical tests. While pathologists may only need basic knowledge because they have the benefit of working with a statistician, the fact that 41% of respondents did not indicate that they work with a statistician suggests that pathologists may not have access to statisticians at some academic institutions. This is worth exploring further, because not only is there a lack of knowledge of statistics, but also potentially a lack of resources for pathologists to use statistical experts. Moreover, 96% of the respondents indicated that having a greater knowledge of statistics would be beneficial, so while a basic knowledge of statistics may be sufficient for most work roles, it may not be sufficient for particular roles such as teaching EBM, reviewing papers, conducting studies, or designing quality improvement experiments.

Statistical knowledge is not limited to understanding methods. It also requires an appreciation for broader statistical ideas such as hypothesis testing, statistical power, key assumptions, and differences between parametric and nonparametric methods. At a broader level, there are key concepts related to study design and good statistical practice (eg, reporting guidelines, reproducibility). Pathologists do not need to know a long list of methods; however, knowledge of a few provides a bridge to understanding the broader conceptual issues that could help pathologists work effectively with statisticians or colleagues who have statistical knowledge. Pathologists need a basic-level understanding of general statistical concepts, and this is enhanced by an advanced-level knowledge of at least a few tests.

We found that pathologists do not acquire statistical skills through lifelong learning. Statistical knowledge was not associated with years of experience or with research experience (estimated by number of publications). This suggests that statistical knowledge is mostly developed during the training years. When we evaluated statistics requirements for medical school, we found that relatively few schools require statistics for admission. Our survey did not assess the percentage of medical students who took an undergraduate course in statistics, an area for further research. Almost all medical schools provide training in EBM and statistics; however, the format, extent, and effectiveness of medical school training are not standardized. Our survey showed that statistical knowledge is highly associated with an advanced degree in another discipline, and that statistical knowledge is associated with taking a formal course in statistics. Thus, the statistical knowledge of pathologists obtained only in medical school and residency is likely to be inadequate.

Almost all survey respondents indicated that they would benefit from more statistical training. For residents, statistics training is often delivered via journal clubs. Although journal clubs may be an effective mechanism for developing high-level conceptual knowledge such as critical appraisal and study design, it may not be an ideal method for developing statistical knowledge and skills. Our results suggest that formal lectures may be more effective than informal (and nonstandardized) avenues such as journal club or discussion on rotations.

Given that RSK scores of faculty did not differ from those of trainees, it is likely that many programs would not have faculty with the expertise to provide training in statistics. Developing statistical training material would require substantial duplication of effort if each training program developed such materials independently. Although online courses are available (eg, via Coursera, edX, Udacity, statistics.com), these courses may not be tailored to the needs of pathology trainees. Thus, it would be helpful if pathology organizations (eg, American Society for Clinical Pathology, College of American Pathologists) pooled resources to develop training materials that could be shared among programs. A Web-based pathology journal club, such as the General Surgery Journal Club, might also provide a cost-effective way to pool resources and avoid duplication of effort. (15) Such training materials could benefit both trainees and practicing pathologists.

To our knowledge, ours is the first study to investigate statistical literacy of pathologists by using a survey. It should be noted that this method has inherent limitations. Our survey of statistical knowledge was based on self-reported knowledge rather than an objective test, which we felt would be practically infeasible. Our list of statistical tests may have been limited by the journals we selected, though we attempted to capture a wide range of journals providing broad coverage of pathology and laboratory medicine. Pathology is a large and varied discipline, and we realize the methods reported in research journals may not reflect the statistical methods required for routine clinical work or operations improvement. Similarly, pathologists do read clinical medical journals, and our study did not assess the methods used in such journals. We also only assessed methods used in original research articles, though many other types of articles (eg, reviews, case studies, commentaries) are useful to pathologists. Our study did not evaluate whether methods were used appropriately; we only tabulated reported methods. Some tests may have been used but not reported, so our estimate of frequency is most likely an underestimate. Our rank order of tests is probably accurate unless the number of unreported tests is high and the distribution of unreported tests is quite different from reported tests. Our response rate was 42%; thus, the respondents to the survey may not reflect the overall population of pathologists and residents. And lastly, we only surveyed pathologists at academic institutions, so we cannot gauge the knowledge of statistics by pathologists in private practice.

Finally, the need for statistical literacy among pathologists is likely to grow. New types of research (genomics, health services research, computational pathology) will require greater statistical knowledge, and statistical knowledge is required for quality improvement and assurance initiatives, which are now required for maintenance of certification. Overall, there appears to be an unmet need for statistical training among pathologists.

In conclusion, despite its limitations, our survey has pinpointed a number of key findings that warrant further investigation. In particular, even those pathologists who professed to have some understanding of statistics responded that having greater knowledge of statistics would be beneficial. Because medical training is so long and often circuitous for those acquiring additional degrees, residency may be a good place to start with the foundations of statistical literacy, but there is also a need to offer additional training and reinforcement of statistical concepts to practicing pathologists.

References

(1.) Anderson BL, Gigerenzer G, Parker S, Schulkin J. Statistical literacy in obstetricians and gynecologists. J Healthc Qual. 2014;36(1):5-17.

(2.) Windish DM, Huot SJ, Green ML. Medicine residents' understanding of the biostatistics and results in the medical literature. J Am Med Assoc. 2007;298(9): 1010-1022.

(3.) Novack L, Jotkowitz A, Knyazer B, Novack V. Evidence-based medicine: assessment of knowledge of basic epidemiological and research methods among medical doctors. Postgrad Med J. 2006;82(974):817-822.

(4.) Oliveri RS, Gluud C, Wille-Jrgensen PA. Hospital doctors' self-rated skills in and use of evidence-based medicine: a questionnaire survey. J Eval Clin Pract. 2004;10(2):219-226.

(5.) Hamdan A. Medical students still lack skills needed to practise evidence-based medicine. J R Soc Med. 2012;105(8):324.

(6.) Weiss ST, Samet JM. An assessment of physician knowledge of epidemiology and biostatistics. J Med Educ. 1980;55(8):692-697.

(7.) Wulff HR, Andersen B, Brandenhoff P, Guttler F. What do doctors know about statistics? Stat Med. 1987;6(1):3-10.

(8.) Miles S, Price GM, Swift L, Shepstone L, Leinster SJ. Statistics teaching in medical school: opinions of practising doctors. BMC Med Educ. 2010;10:75.

(9.) The pathology milestone project. J Grad Med Educ. 2014;6(1 suppl 1):182-203.

(10.) Yi D, Ma D, Li G, et al. Statistical use in clinical studies: is there evidence of a methodological shift? PLoS One. 2015;10(10):e0140159.

(11.) Reed JF III, Salen P, Bagher P. Methodological and statistical techniques: what do residents really need to know about statistics? J Med Syst. 2003;27(3): 233-238.

(12.) Horton NJ, Switzer SS. Statistical methods in the journal. New Engl J Med. 2005;353(18):1977-1978.

(13.) American Association of Medical Colleges. Required premedical coursework and competencies. 2015. https://students-residents.aamc.org/applyingmedical-school/ applying-medical-school-process/deciding-where-apply/ medical-school-admission-requirements/. Accessed August, 23, 2015.

(14.) Bahar B, Pambuccian S, Barkan G, Akdas Y. The use of statistical methods in studies published in cytopathology journals. J Am Soc Cytopathol. 2015;4(6): S4.

(15.) McMaster University. General Surgery Journal Club. 2016. http://fhs. mcmaster.ca/generalsurgery/journal_club.html. Accessed June 4, 2016.

Robert L. Schmidt, MD, PhD, MBA; Deborah J. Chute, MD; Jorie M. Colbert-Getz, PhD; Adolfo Firpo-Betancourt, MD, MPA; Daniel S. James, MSIS; Julie K. Karp, MD; Douglas C. Miller, MD, PhD; Danny A. Milner Jr, MD, MSc; Kristi J. Smock, MD; Ann T. Sutton, MD; Brandon S. Walker, MS; Kristie L. White, MD, MAEd; Andrew R. Wilson, MStat, PhD; Eva M. Wojcik, MD; Marwan A. Yared, MD; Rachel E. Factor, MD, MHS

Accepted for publication July 1, 2016.

Published as an Early Online Release December 13, 2016.

Supplemental digital content is available for this article at www. archivesofpathology.org in the February 2017 table of contents.

From the Department of Pathology, University of Utah Health Sciences Center, and ARUP Laboratories, Salt Lake City, Utah (Drs Schmidt, Smock, and Factor); the Department of Pathology, Cleveland Clinic Lerner College of Medicine, Cleveland, Ohio (Dr Chute); the Department of Internal Medicine, University of Utah, Salt Lake City (Dr Colbert-Getz);the Department of Pathology, Icahn School of Medicine at Mount Sinai, New York, New York (Dr FirpoBetancourt);the Departments of Marketing (Mr James) and Informatics (Mr Walker), ARUP Laboratories, Salt Lake City, Utah; the Department of Pathology, Anatomy, and Cell Biology, Thomas Jefferson University Hospital, Philadelphia, Pennsylvania (Dr Karp); the Department of Pathology & Anatomical Sciences, University of Missouri, Columbia (Dr Miller); the Department of Pathology, Brigham and Women's Hospital, Boston, Massachusetts (Dr Milner); the Department of Pathology and Laboratory Medicine, Brody School of Medicine, East Carolina University, Greenville, North Carolina (Dr Sutton); the Department of Laboratory Medicine, University of California at San Francisco, San Francisco (Dr White); College of Nursing, Health Sciences Center, University of Utah, Salt Lake City (Dr Wilson); the Department of Pathology, Loyola University, Chicago, Illinois (Dr Wojcik); and the Department of Pathology, Baylor College of Medicine, Houston, Texas (Dr Yared).

The authors have no relevant financial interest in the products or companies described in this article.

Reprints: Robert L. Schmidt, MD, PhD, MBA, Department of Pathology, 15 N Medical Drive East, University of Utah, Salt Lake City, UT 84112 (email: Robert.Schmidt@hsc.utah.edu).

Please Note: Illustration(s) are not available due to copyright restrictions.

Caption: Figure 1. A through C, Distribution of RSK scores. Respondent statistical knowledge was calculated by averaging the respondents' knowledge for 16 statistical tests (0 =no knowledge, 1 =able to interpret test but cannot perform test without help, 2 = can perform and interpret the test without assistance). A score of zero would indicate no knowledge of all 16 tests. A score of 2 would indicate that the respondent was able to perform and interpret all 16 tests without assistance. A, Distribution of trainee RSK scores. B, Distribution of faculty RSK scores. C, Distribution of combined (trainee and faculty) RSK scores. Abbreviation: RSK, respondent statistical knowledge.

Caption: Figure 2. Comparison of respondent statistical knowledge and use of statistical tests in journal articles. Percentage test use (horizontal axis) refers to the percentage of articles that use statistical tests and that used this particular test. For example, the [chi square] test was used in 25% of the articles that used statistical tests. Abbreviations: ANOVA, analysis of variance; reg, regression; ROC, receiver operating characteristic.
Table 1. Use of Statistical Tests by Journal Category (a)

                                                 Journal Category

     Item                                     General (b)   Basic (c)

     No. of journals evaluated                     2            3
     No. of articles evaluated                    200          300
     Articles using no statistics, %              30            6
     Descriptive methods only, %                  12            5
     Articles using statistical tests, %          58           90

1    Student t test                               19           59
2    Regression/ANOVA                             14           42
3    [chi square] test                            15            9
4    Mann-Whitney test (rank sum)                  9           22
5    Fisher exact test                            14            6
     Survival analysis                            13           14
6      Kaplan-Meier/log-rank                      13           13
7      Cox regression                              9            8
     Multiple comparison adjustment                1           29
8      Tukey                                       1           11
9      Bonferroni                                  1            8
10     Newman-Keuls                                0            4
11   [kappa] Statistic                             4            1
12   ROC analysis                                  5            2
13   Logistic regression                           4            1
14   Spearman rank correlation                     3            5
15   Kruskal-Wallis test                           4            6
16   Pearson correlation statistic                 4            3
17   Normality test                                2            2
18   McNemar test                                  1            0
     Other tests                                   4            5
     Avg tests per article (g)                    1.3          2.1
     Avg tests in articles using tests (h)        2.2          2.3

                                               Journal Category

     Item                                     Trans (d)   Cyto (e)

     No. of journals evaluated                    2          2
     No. of articles evaluated                   200        200
     Articles using no statistics, %             36          26
     Descriptive methods only, %                  7          12
     Articles using statistical tests, %         59          62

1    Student t test                              20          10
2    Regression/ANOVA                             6          4
3    [chi square] test                           23          28
4    Mann-Whitney test (rank sum)                12          9
5    Fisher exact test                           20          21
     Survival analysis                           29          5
6      Kaplan-Meier/log-rank                     27          5
7      Cox regression                            17          2
     Multiple comparison adjustment               1          2
8      Tukey                                      0          1
9      Bonferroni                                 1          1
10     Newman-Keuls                               0          0
11   [kappa] Statistic                           10          9
12   ROC analysis                                 2          4
13   Logistic regression                          5          6
14   Spearman rank correlation                    4          2
15   Kruskal-Wallis test                          1          3
16   Pearson correlation statistic                3          2
17   Normality test                               2          1
18   McNemar test                                 1          4
     Other tests                                  4          8
     Avg tests per article (g)                   1.6        1.2
     Avg tests in articles using tests (h)       2.7        1.9

                                                  Journal
                                                 Category

     Item                                     Chem (f)   Avg

     No. of journals evaluated                   2
     No. of articles evaluated                  200      220
     Articles using no statistics, %             7       21.0
     Descriptive methods only, %                 23      11.8
     Articles using statistical tests, %         70      67.7

1    Student t test                              25      26.6
2    Regression/ANOVA                            30      19.2
3    [chi square] test                           19      18.8
4    Mann-Whitney test (rank sum)                24      15.2
5    Fisher exact test                           4       13.0
     Survival analysis                           10      14.2
6      Kaplan-Meier/log-rank                     5       12.6
7      Cox regression                            7       8.6
     Multiple comparison adjustment              4       7.4
8      Tukey                                     1       2.8
9      Bonferroni                                2       2.6
10     Newman-Keuls                              1       1.0
11   [kappa] Statistic                           5       5.8
12   ROC analysis                                14      5.4
13   Logistic regression                         10      5.2
14   Spearman rank correlation                   10      4.8
15   Kruskal-Wallis test                         8       4.4
16   Pearson correlation statistic               8       4.0
17   Normality test                              7       2.8
18   McNemar test                                1       1.4
     Other tests                                 10      6.2
     Avg tests per article (g)                  1.9      1.6
     Avg tests in articles using tests (h)      2.7      2.4

Abbreviations: ANOVA, analysis of variance; Avg, average;
ROC, receiver operating characteristic.

(a) One hundred articles were randomly selected from each
journal (1100 total articles). Table entries are the average
number of times a test was used; for example, the [chi square]
test was used an average of 15 times per 100 articles in general
journals. Other table entries are percentages (the number of
journals in each category is different);for example, 30% of
articles in general journals used no statistical analysis.
Nineteen percent of articles in general journals used the
Student t test. Comparison of individual journals is
available in supporting material.

(b) General journals: American Journal of Clinical Pathology,
Archives of Pathology & Laboratory Medicine.

(c) Basic science journals: Laboratory Investigation,
Journal of Pathology, American Journal of Pathology.

(d) Translational journals: Modern Pathology,
American Journal of Surgical Pathology.

(e) Cytopathology journals: Cancer Cytopathology,
Diagnostic Cytopathology.

(f) Chemistry journals: Clinical Chemistry,
Clinica Chimica Acta.

(g) Sum of numbered rows divided by 100 articles.

(h) Sum of numbered rows divided by number of
articles using statistical tests.

Table 2. Statistical Software Usage
by Journal Category (a)

                         Journal Category

Item            General (b)   Basic (c)   Trans (d)

Not specified       28           49          26
SPSSg               14           15          16
GraphPad (h)         4           24           3
SAS (i)             13            1           6
R (j)                2            1           7
Stata (k)            3            0           2
JMP (l)              1            1           3
Other                7            4           4
  (N = 23)

                      Journal Category

Item            Cyto (e)   Chem (f)   Average

Not specified      33         36        34
SPSSg              19         21        17
GraphPad (h)       2          3          7
SAS (i)            9          8          7
R (j)              1          9          4
Stata (k)          4          5          3
JMP (l)            3          2          2
Other              5          10         6
  (N = 23)

(a) One hundred articles were randomly selected from each journal
(1100 total articles). Table entries are the percentage of articles
that used a particular software package. For example, 14% of general
journals used SPSS. Comparison of individual journals is available in
supporting material.

(b) General journals: American Journal of Clinical Pathology,
Archives of Pathology & Laboratory Medicine.

(c) Basic science journals: Laboratory Investigation,
Journal of Pathology, American Journal of Pathology.

(d) Translational journals: Modern Pathology, American Journal
of Surgical Pathology.

(e) Cytopathology journals: Cancer Cytopathology,
Diagnostic Cytopathology.

(f) Chemistry journals: Clinical Chemistry, Clinica Chimica Acta.

(g) Statistical Program for the Social Sciences (IBM Corp, Armonk,
New York).

(h) GraphPad Software (GraphPad Software Inc,
La Jolla, California).

(i) Statistical Analysis System (SAS Institute
Inc, Cary, North Carolina).

(j) R Development Core Team (R Foundation for Statistical
Computing, Vienna, Austria).

(k) Stata Statistical Software (StataCorp LP,
College Station, Texas).

(l) JMP Software (SAS Institute Inc, Cary, North Carolina).

Table 3. Response Rate by Responder Type and Institution (a)

                             Faculty

Institution            Inv    Resp       Rate (b)

Baylor (c)              55      21         0.38
Cleveland Clinic (d)    97      54 (e)     0.56
East Carolina (f)       17       2         0.12
Thomas Jefferson (g)    55      11         0.20
Loyola (h)              36      19         0.53
Mount Sinai (i)         85      36         0.42
Missouri (j)             1       1         1.00
UCSF (k)                94      44         0.47
Utah (l)               140      41         0.29
Total                  580     229         0.39

                             Trainees

Institution            Inv     Resp      Rate

Baylor (c)              28      16       0.57
Cleveland Clinic (d)    49      28 (e)   0.56
East Carolina (f)       13       2       0.15
Thomas Jefferson (g)    19       5       0.26
Loyola (h)              22      17       0.77
Mount Sinai (i)         54      28       0.52
Missouri (j)            11       2       0.18
UCSF (k)                59      16       0.27
Utah (l)                44      22       0.50
Total                  299     136       0.45

                             Overall

Institution            Inv    Resp     Rate

Baylor (c)              83      37     0.45
Cleveland Clinic (d)   146      82     0.56
East Carolina (f)       30       4     0.13
Thomas Jefferson (g)    74      16     0.22
Loyola (h)              58      36     0.62
Mount Sinai (i)        139      64     0.46
Missouri (j)            12       3     0.25
UCSF (k)               153      60     0.39
Utah (l)               184      63     0.34
Total                  879     365     0.42

Abbreviations: Inv, invited, Resp, responded.

(a) The responses for Cleveland Clinic were anonymous. We estimated
the response for faculty and trainees by dividing the total responses
(82) in proportion to the number of faculty and trainees that
were invited.

(b) Rate = response rate (responded/invited). All responses from each
institution were from faculty and trainees in the Department of
Pathology.

(c) Baylor College of Medicine (Houston, Texas).

(d) Cleveland Clinic Lerner College of Medicine (Cleveland, Ohio).

(e) Estimate. We were blinded to the responders from Cleveland Clinic.

(f) Brody School of Medicine, East Carolina University
(Greenville, North Carolina).

(g) Thomas Jefferson University (Philadelphia, Pennsylvania).

(h) Stritch School of Medicine, Loyola University Medical Center
(Maywood, Illinois).

(i) Icahn School of Medicine at Mount Sinai (New York, New York).

(j) University of Missouri School of Medicine (Columbia).

(k) University of California, San Francisco
(San Francisco, California).

(l) University of Utah Health Sciences Center (Salt Lake City).

Table 4. Statistical Training in Sampled Residency Programs (a)

                            Statistical Training

                        Journal Club and
                       Informal Training      Formal
Program                   on Rotations       Lectures

Baylor (b)                    Yes           2
Cleveland Clinic (c)          Yes           0
East Carolina (d)             Yes           0
Jefferson (e)                 Yes           0
Loyola (f)                    Yes           2, repeated
                                            PGY1-PGY3
Missouri (g)                  Yes           4
Mount Sinai (h)               Yes           0
UCSF (i)                      Yes           8
Utah (j)                      Yes           2

                          Statistical Training

                            Other Methods of
Program                    Teaching Statistics

Baylor (b)             Research course
Cleveland Clinic (c)   None
East Carolina (d)      None
Jefferson (e)          None
Loyola (f)             Statistics exercises during
                         chemistry rotation
Missouri (g)           None
Mount Sinai (h)        None
UCSF (i)               None
Utah (j)               None

Abbreviation: PGY, post graduate year.

(a) All responses each institution were from faculty
and trainees in the Department of Pathology.

(b) Baylor College of Medicine (Houston, Texas).

(c) Cleveland Clinic Lerner College of Medicine
(Cleveland, Ohio).

(d) Brody School of Medicine, East Carolina University
(Greenville, North Carolina).

(e) Thomas Jefferson University (Philadelphia, Pennsylvania).

(f) Stritch School of Medicine, Loyola University
Medical Center (Maywood, Illinois).

(g) University of Missouri School of Medicine (Columbia).

(h) Icahn School of Medicine at Mount Sinai (New York, New York).

(i) University of California, San Francisco
(San Francisco, California).

(j) University of Utah Health Sciences Center (Salt Lake City).

Table 5. Summary of Statistical Knowledge by Test (a)

                                   Knowledge Level, %

Method or Test                   None   Basic   Advanced

Student t test                    17     33        50
Regression/ANOVA                  40     34        26
[chi square] test                 17     41        42
Mann-Whitney test                 66     20        14
Fisher exact test                 40     33        27
Kaplan-Meier/log-rank             19     59        22
Cox regression                    61     32        7
[kappa] Statistic                 60     31        9
ROC analysis                      43     40        17
Logistic regression               58     31        11
Spearman rank correlation         69     21        10
Kruskal-Wallis test               79     12        9
Pearson correlation statistic     57     27        16
Average                           48     32        20

                                                         Rank

Method or Test                   TSK Score   Journal Usage   TSK Score

Student t test                      1.3            1             1
Regression/ANOVA                    0.9            2             5
[chi square] test                   1.3            3             2
Mann-Whitney test                   0.5            4            10
Fisher exact test                   0.9            5             4
Kaplan-Meier/log-rank               1.0            6             3
Cox regression                      0.5            7            11
[kappa] Statistic                   0.5            8             9
ROC analysis                        0.7            9             6
Logistic regression                 0.5           10             8
Spearman rank correlation           0.4           11            13
Kruskal-Wallis test                 0.3           12            14
Pearson correlation statistic       0.6           14             7
Average                            0.72

Abbreviations: ANOVA, analysis of variance;ROC, receiver
operating characteristic; RSK, respondent statistical
knowledge; TSK, test statistical knowledge score.

(a) Knowledge levels were designated as none (never heard
of test or recognized name only), basic (can interpret test
results but need help to perform the test), and advanced (can
perform the test and interpret it without help). The TSK score
is calculated by assigning a score to each knowledge category
(none = 0, basic = 1, advanced = 2) and calculating the weighted
average (score X percentage) for each test. The rank shows the
relative rank at which methods are reported in journals
(journal usage) or with respect to respondent knowledge (RSK).
For example, ROC analysis was ranked ninth in the methods used
in journals but was ranked sixth with respect to respondents'
knowledge.

Table 6. Characteristics of Respondents and Statistical
Knowledge Characteristics of Participants

                                                            P
     Characteristic        No (%)    Median RSK (IQR)   Value (a)

Position
  Trainee                123 (36)      0.5 (0.25-1)        .14
  Faculty                209 (63)    0.63 (0.31-1.13)

Primary focus
  AP                     112 (56)    0.5 (0.25-0.94)      <.001
  CP                      64 (32)    0.91 (0.56-1.25)
  Both                    25 (12)     0.53 (0.31-1)

Advanced degree
other than MD
  Yes                    127 (41)    0.86 (0.44-1.25)     <.001
  No                     182 (59)    0.39 (0.19-0.88)

Type of advanced
degree
  PhD                     95 (72)    0.91 (0.56-1.25)      .06
  MS                      37 (28)    0.45 (0.31-1.28)

Years in practice
  Still training         114 (36)      0.5 (0.25-1)        .20
  <5                      51 (16)    0.63 (0.25-1.25)
  5-10                    49 (15)     0.56 (0.25-1)
  11-20                   52 (16)    0.63 (0.38-0.94)
  [greater than or        51 (16)    0.75 (0.32-1.19)
    equal to] 20

Do you conduct
research studies
for publication?
  Yes                    264 (82)    0.63 (0.31-1.13)     <.001
  No                      57 (18)    0.25 (0.13-0.56)

Need to publish
studies for
professional
advancement?
  Yes                    223 (73)    0.63 (0.27-1.13)     .049
  No                      86 (27)    0.44 (0.25-0.94)

Have taken a
statistics course?
  Yes                    186 (59)    0.75 (0.38-1.25)     <.001
  No                     129 (41)    0.44 (0.19-0.89)

Where do you most
often use statistics?
  For research           208 (66)    0.66 (0.31-1.1)      <.001
  To support clinical     57 (18)    0.75 (0.25-1.25)
    work and
    operations
  Rarely use              49 (16)     0.28 (0-0.57)
    statistics
    in my work

How do you approach
statistical work
in your job?
  Help from a            157 (59)    0.43 (0.25-0.88)     <.001
    colleague or
    statistician
  I do the analysis       92 (35)    1.19 (0.88-1.56)
    myself
  Not applicable           16 (6)    0.28 (0.22-0.38)

Having a better
understanding of
statistics would
be helpful
  Agree                  302 (96)     0.56 (0.25-1)        .58
  Disagree                  8 (3)    1.09 (0.19-1.25)

Abbreviations: AP, anatomic pathology; CP, clinical pathology; IQR,
interquartile range; RSK, respondent statistical knowledge score.

(a) P values are reported for the Kruskal-Wallis test.
COPYRIGHT 2017 College of American Pathologists
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Schmidt, Robert L.; Chute, Deborah J.; Colbert-Getz, Jorie M.; Firpo-Betancourt, Adolfo; James, Dani
Publication:Archives of Pathology & Laboratory Medicine
Article Type:Report
Date:Feb 1, 2017
Words:6893
Previous Article:Pathology's Stepchild.
Next Article:RAI1 Alternate Probe Identifies Additional Breast Cancer Cases as Amplified Following Equivocal HER2 Fluorescence In Situ Hybridization Testing:...
Topics:

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters