A prospective, multi-institutional diagnostic trial to determine pathologist accuracy in estimation of percentage of malignant cells.
Few methods exist for determining the percentage of malignant nuclei. (1) Counting the total nuclei in a tissue section is extremely time consuming and not practical for clinical use. Usually, a pathologist reviews the tissue section to be analyzed and roughly estimates the percentage of malignant nuclei. (2,3,6) Previous studies have evaluated pathologist agreement when estimating percentage of malignant cells (concordance) and have illustrated the difficulty in using visual assessment to quantify absolute numbers of cells. (1,2,17,18) These studies were relatively limited and included small numbers of cases, and in particular small numbers of participants. Therefore, we designed a statistically powered study to evaluate pathologist accuracy in estimating percentage of malignant cells as part of the College of American Pathologists (CAP) KRAS-B 2011 Survey. Here, we report the results of this prospective, multi-institutional diagnostic trial.
MATERIALS AND METHODS
Ten x20 magnification images of colon adenocarcinoma specimens stained with hematoxylin-eosin (Figure 1) were selected by the CAP Molecular Oncology Committee to represent a range of specimen appearances encountered in routine diagnostic practice, and the images were distributed as an educational component of the KRAS-B 2011 Survey (a survey that the CAP offers for clinical laboratories to assess their proficiency at detection of mutations in KRAS). Included in the survey were carefully detailed instructions as well as an example of how lab personnel should calculate the percentage of malignant cells. The instructions included a brief description regarding the importance of accurate determination of the percentage, and participating laboratories were asked to identify who was responsible for determining the percentage of malignant cells and the method used for achieving that number. Data from 194 labs were received and analyzed for accuracy through comparison of estimated percentages to a criterion standard achieved by counting all malignant and nonmalignant nuclei in the survey images and calculating the percentage of malignant cells. The criterion standard for the correct percentage of malignant cells in each image was determined by subdividing the images into 9 parts for better detail resolution and marking each nucleus with a red or green dot corresponding to neoplastic and nonneoplastic cells, respectively, using Photoshop (Adobe, San Jose, California). The number of red and green dots was counted for each part of each image and totaled. A sample image, one of its subdivided parts, and the same part with marked nuclei are shown (Figure 2, A through C). All dotting and counting was done by technicians (H.V. and K.L.) in the Rimm lab, and designations of benign versus malignant nuclei were reviewed by a pathologist (D.L.R.). We believe this approach accurately represents a criterion standard for evaluating pathologist estimation.
None of the institutions surveyed in this study reported counting the total nuclei in a tissue section to obtain the percentage of tumor cells in the sample. When asked to describe their method for determining the percentage of malignant cells in a tissue section, the vast majority of laboratories reported using pathologist estimation. Survey results indicated a high level of interlaboratory variation and the presence of a wide range of estimated percentages on multiple study images. The mean estimate for 8 of 10 images was within 10% of the criterion standard, suggesting that, on average, pathologists perform well. For example, image KRAS-90 (Figure 2, D) contained the highest percentage of malignant cells (66.2%), and the mean estimate was 64.4% malignant cells. On the other end of the scale, KRAS-91 (Figure 2, E) had the lowest percentage of malignant cells (6.6%), and the mean estimate was within 1% of the count. Other examples suggested less accurate pathologist estimates, with the potential to impact patient care. For example, KRAS-89 (Figure 2, F) yielded the most discordant responses, ranging from 10% to 95% malignant cells and with the mean estimate of percentage of malignant cells differing by 24.4% from the criterion standard.
The frequency distribution of all estimated percentages for each image is shown in Figure 3, A through J. The range of responses indicated a low level of interlaboratory precision in estimating the percentage of malignant cells. The mean estimate for each image in the survey was also plotted versus the criterion standard (Figure 3, K). Linear regression showed a statistically significant correlation between the mean estimate and criterion standard ([R.sup.2] = 0.8236; P < .001). The greatest variation occurred in images containing higher percentages of malignant cells (>40%).
Patient care could be affected by pathologist overestimation of malignant cells, which could generate a false-negative test result for a mutation. Based on the generally accepted ranges of analytic sensitivity for various diagnostic assays, we categorized overestimation errors of greater than 20% as having the potential to change patient outcome because of false-negative DNA testing results (although this figure is highly dependent on the analytic sensitivity of the assay selected by each laboratory). For each image, we graphed the percent of survey responders that overestimated the percentage of malignant cells by greater than 20% (Figure 3, L). Although some cases showed low rates of overestimation, others were more problematic. The most discrepant image was KRAS-89, in which 57% of survey participants overestimated the percentage of malignant cells by greater than 20%.
With respect to mean pathologist estimation, the data indicate that pathologists more accurately estimate the percentage of malignant nuclei in cases containing low amounts of tumor cells. Based on our linear regression plot of mean pathologist estimates versus the counted malignant cellularity (Figure 3, K), that accuracy seems to decrease as a function of increasing tumor cell content, suggesting overestimation affecting patient care may be less likely. Pathologist concordance, however, remains low for all study images except KRAS-91 and appears to be unrelated to malignant cellularity. Other specimen factors, such as staining intensity or section thickness, may also affect pathologist estimation accuracy, although these factors were unable to be assessed in this study. The study also suggests that pathologists are more likely to overestimate than underestimate the percentage of malignant cells. This is of some concern, because overestimation is more likely to affect patient care than underestimation. We averaged each pathologist's error across 10 images and found that 5.4% of survey participants overestimated by more than 20% on average. This is particularly concerning because overestimation errors greater than 20% are most likely to affect molecular testing results because of insufficient malignant DNA.
Although the results of this study were powered based on the number of participants and pilot data on estimation of percent neoplastic cellularity, they are limited by inclusion of only 10 representative images for review and the exclusive use of x20 images for evaluation of malignant cellularity, rather than whole-slide examination at multiple magnifications, as is done in routine clinical practice. Finally, this study does not address how errors in the estimation of percentage of malignant cells affect patient outcome. Nonetheless, the data demonstrate that most pathologists adequately estimate the percentage of malignant cells. However, they also illustrate the potential for a false-negative rate of at least 5%. As molecular testing continues to become a routine part of pathologic evaluation, pathologists should be conservative in their estimations of the percentage of malignant cells and consider the importance of tissue macrodissection or microdissection to increase malignant cellularity. In addition, pathologists should have an understanding of the analytic sensitivity of the assay selected by their lab (Sanger sequencing, pyrosequencing, real-time polymerase chain reaction, or next-generation sequencing). Finally, new automated objective methods should be sought that can quickly and accurately determine the percentage of malignant cells.
Although all committee members participated in discussions, only committee members directly involved in the study are listed as authors. The authors would like to thank the committee members Joel Moncur, MD; Jeff Chang, MD; Greg Tsongalis, PhD; and Cherie Paquette, MD, for editorial comments on the manuscript prior to submission.
Caption: Figure 1. The complete set of 10 images distributed as part of the College of American Pathologists (CAP) KRAS-B 2011 Survey for the Photo Challenge. A through J, The 10 images are presented in order, each representing a random field from a random colon cancer specimen (hematoxylin-eosin, original magnifications x20).
Caption: Figure 2. College of American Pathologists (CAP) KRAS-B 2011 Survey images. A, A sample survey image, KRAS-92. B, An electronically magnified image from KRAS-92. C, The same image from KRAS-92 (B) with red and green dots objectively quantifying malignant and benign cells, respectively. D, KRAS-90 contained the highest percentage of malignant cells (66.2%). E, KRAS-91 had the lowest percentage of malignant cells (6.6%) and was the least discrepant case for estimating the percentage of malignant cells at x20 magnification. F, KRAS-89 was the most discrepant case for the estimating percentage of malignant cells, containing 41.2% malignant cells (hematoxylin-eosin, original magnifications x20 [A, D, E, F]).
Caption: Figure 3. Precision and accuracy of pathologist estimation. A through J, All estimates of percentage of malignant cells were plotted in a histogram for each survey image (KRAS-86 to KRAS-95). The criterion standard obtained by manually counting nuclei and the mean estimate of survey participants are shown for each image. K, The mean estimated percentage of malignant cells was plotted versus the criterion standard for all images. Error bars show the standard deviation of responses. L, Errors in estimation were calculated as the percentage difference between the criterion standard and the estimated percentage of malignant cells. The percent of survey responders who overestimated the percentage of malignant cells by greater than 20% is shown for each image.
(1.) Warth A, Penzel R, Brandt R, et al. Optimized algorithm for Sanger sequencing-based EGFR mutation analyses in NSCLC biopsies. Virchows Arch. 2012;460(4):407-414.
(2.) Bellon E, Ligtenberg MJ, Tejpar S, et al. External quality assessment for KRAS testing is needed: setup of a European program and report of the first joined regional quality assessment rounds. Oncologist. 2011;16(4):467-478.
(3.) Lamy A, Blanchard F, Le Pessot F, et al. Metastatic colorectal cancer KRAS genotyping in routine practice: results and pitfalls. Mod Pathol. 2011;24(8):1090-1100.
(4.) Pont-Kingdon G, Gedge F, Wooderchak-Donahue W, et al. Design and analytical validation of clinical DNA sequencing assays. Arch Pathol Lab Med. 2011;136(1):41-46.
(5.) Anderson S, Bloom KJ, Vallera DU, et al. Multisite analytic performance studies of a real-time polymerase chain reaction assay for the detection of BRAF V600E mutations in formalin-fixed paraffin-embedded tissue specimens of malignant melanoma. Arch Pathol Lab Med. 2012;136(11):1385-1391.
(6.) Angulo B, Garcia-Garcia E, Martinez R, et al. A commercial real-time PCR kit provides greater sensitivity than direct sequencing to detect KRAS mutations: a morphology-based approach in colorectal carcinoma. J Mol Diagn. 2010;12(3): 292-299.
(7.) Allegra CJ, Jessup JM, Somerfield MR, et al. American Society of Clinical Oncology provisional clinical opinion: testing for KRAS gene mutations in patients with metastatic colorectal carcinoma to predict response to anti-epidermal growth factor receptor monoclonal antibody therapy. J Clin Oncol. 2009;27(12):2091-2096.
(8.) Weichert W, Schewe C, Lehmann A, et al. KRAS genotyping of paraffin-embedded colorectal cancer tissue in routine diagnostics: comparison of methods and impact of histology. J Mol Diagn. 2010;12(1):35-42.
(9.) Symmans WF, Peintinger F, Hatzis C, et al. Measurement of residual breast cancer burden to predict survival after neoadjuvant chemotherapy. J Clin Oncol. 2007;25(28):4414-4422.
(10.) Peintinger F, Anderson K, Mazouni C, et al. Thirty-gene pharmacogenomic test correlates with residual cancer burden after preoperative chemotherapy for breast cancer. Clin Cancer Res. 2007;13(14):4078-4082.
(11.) Nahleh Z, Sivasubramaniam D, Dhaliwal S, Sundarajan V, Komrokji R. Residual cancer burden in locally advanced breast cancer: a superior tool. Curr Oncol. 2008;15(6):17-24.
(12.) Peintinger F, Buzdar AU, Kuerer HM, et al. Hormone receptor status and pathologic response of HER2-positive breast cancer treated with neoadjuvant chemotherapy and trastuzumab. Ann Oncol. 2008;19(12):2020-2025.
(13.) Meldrum C, Doyle MA, Tothill RW. Next-generation sequencing for cancer diagnostics: a practical perspective. Clin Biochem Rev. 2011;32(4):177-195.
(14.) Natrajan R, Reis-Filho JS. Next-generation sequencing applied to molecular diagnostics. Expert Rev Mol Diagn. 2011;11(4):425-444.
(15.) Talkowski ME, Ernst C, Heilbut A, et al. Next-generation sequencing strategies enable routine detection of balanced chromosome rearrangements for clinical diagnostics and genetic research. Am J Hum Genet. 2011;88(4):469-481.
(16.) Smeets SJ, Harjes U, van Wieringen WN, et al. To DNA or not to DNA? That is the question, when it comes to molecular subtyping for the clinic! Clin Cancer Res. 2011;17(15):4959-4964.
(17.) Dequeker E, Ligtenberg M, Vander Borght S, van Krieken J. Mutation analysis of KRAS prior to targeted therapy in colorectal cancer: development and evaluation of quality by a European external quality assessment scheme. Virchows Arch. 2011;459(2):155-160.
(18.) Thunnissen E, Bovee JVMG, Bruinsma H, et al. EGFR and KRAS quality assurance schemes in pathology: generating normative data for molecular predictive marker analysis in targeted therapy. J Clin Pathol. 2011 ;64(10):884-892.
Hollis Viray, BS; Kevin Li; Thomas A. Long, MPH; Patricia Vasalos, BS; Julia A. Bridge, MD; Lawrence J. Jennings, MD; Kevin C. Halling, MD; Meera Hameed, MD; David L. Rimm, MD, PhD
Accepted for publication February 5, 2013.
From the Department of Pathology, Yale University School of Medicine, New Haven, Connecticut (Dr Viray, Mr Li, Dr Rimm); the Biostatistics (Mr Long) and Surveys (Ms Vasalos) Departments, College of American Pathologists, Northfield, Illinois; the Department of Pathology & Microbiology, University of Nebraska Medical Center, Omaha (Dr Bridge); the Department of Pathology & Laboratory Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois (Dr Jennings); the Department of Pathology, Mayo Clinic, Rochester, Minnesota (Dr Halling); and the Department of Pathology, Memorial Sloan-Kettering Cancer Center, New York, New York (Dr Hameed).
The authors have no relevant financial interest in the products or companies described in this article.
Reprints: David L. Rimm, MD, PhD, Department of Pathology, BML 11 6, Yale University School of Medicine, 310 Cedar St, PO Box 208023, New Haven, CT 06520-8023 (e-mail: firstname.lastname@example.org).
Please note: Illustration(s) are not available due to copyright restrictions.
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||CAP Laboratory Improvement Programs|
|Author:||Viray, Hollis; Li, Kevin; Long, Thomas A.; Vasalos, Patricia; Bridge, Julia A.; Jennings, Lawrence J|
|Publication:||Archives of Pathology & Laboratory Medicine|
|Date:||Nov 1, 2013|
|Previous Article:||Lack of standards for the detection of melanoma in sentinel lymph nodes: a survey and recommendations.|
|Next Article:||Immunostaining for rapid diagnosis of acute promyelocytic leukemia with the tetramethylrhodamine 5-isothiocyanate--conjugated anti-promyelocytic...|