Printer Friendly

The Effects of Computerized Clinical Decision Support Systems on Laboratory Test Ordering: A Systematic Review.

After years of persistent increase, laboratory test ordering has become the highest-volume medical act. In the United States (1) and Europe, (55) the annual increase in the use of laboratory tests averaged around 5% during the last decade. Medicare (2) spending for clinical laboratory testing has peaked at almost $10 billion, accounting for 1.7% of the total health care budget. However, it has been estimated that when including non-Medicare spending, the figures for the United States are up to 7 times higher. (3) In Europe, laboratory spending accounted for 1.8% of total health care spending in 2012, totaling 17 billion [euro]. Overuse of laboratory test ordering has been shown to be at 20%. (4) Even though reducing the overuse of laboratory testing may not seem very important in relation to the whole of health care spending, the true benefits are to be found in reducing downstream costs. (5) These downstream activities include additional diagnostics such as repeat testing or imaging, prescriptions, surgeries, and hospital stays. Besides reducing costs, improving appropriate laboratory test ordering improves the quality of care by avoiding false-positive results and accidental findings of unknown significance, (6) and is not associated with an increase in adverse effects, morbidity, or mortality. (7,8)

Computerized clinical decision support systems (CCDSSs) are believed to be effective in reducing unnecessary diagnostic testing. Computerized clinical decision support systems are information technology-based systems that use patient-specific characteristics and match these to a knowledge base using rule-based algorithms. (9) These systems can then generate patient-specific reminders or recommendations prompting more appropriate care. A recent systematic review comparing strategies for changing laboratory testing behavior of physicians concluded that several interventions, such as educational strategies, feed back, changing test order forms, and reminders (a form of CCDSS) may be effective. (10) The influence of CCDSSs on diagnostic test ordering has been evaluated in 3 previous systematic reviews. (11-13) These reviews concluded that CCDSSs have not yet been able to show an effect on clinical outcomes but appear to have a positive effect on process outcomes for diagnostic test ordering. One of the practical barriers to measuring the effect of CCDSSs is inadequate compliance by health care professionals with their recommendations, which has been shown to be as low as 1%. (14) However, reviews evaluating CCDSSs on diagnostic test ordering have included a very wide variety of systems aimed at diverse types of diagnostic tests or procedures. To date, no review has been undertaken to evaluate CCDSSs aimed specifically at laboratory test ordering. We conducted this review to evaluate the effect of CCDSSs on the appropriateness of laboratory test ordering of health care professionals.


The details of our protocol have been published elsewhere. (15) Our methods are summarized here, including post hoc details of our review process.

Research Question

What is the effect of CCDSSs integrated in the electronic health record (EHR) on appropriate laboratory test ordering?

Search Strategy

We searched the MEDLINE, CINAHL, Embase, MEDLINE InProcess and Other Non-Indexed Citations,, Cochrane Library, and Inspec databases through December 18, 2015, for relevant citations. We included terms for the following key concepts: decision making, computer-assisted, decision support systems, order sets, laboratory, and diagnosis. We combined these search terms with terms for the following study designs: systematic reviews, meta-analyses, randomized clinical trials, cluster randomized trials, nonrandomized controlled trials, controlled before-after studies, and time series. Details on our search strategy can be found in the supplemental digital content (see supplemental digital content at in the April 2017 table of contents). Post hoc we also searched the National Institute for Health and Clinical Excellence Evidence Search database for relevant systematic reviews and health technology assessments. We also hand searched the reference lists of included articles and included systematic reviews and health technology assessments.

Study Selection

Each study was independently assessed for eligibility by 2 reviewers and a third reviewer was consulted in case of disagreement. Initially, studies were assessed on title and abstract and subsequently full-text screening for eligibility was performed. We included studies that assessed a CCDSS providing patient-specific information, delivered in the form of an on-screen management option, reminder, or suggestion through a computerized physician order entry using a rule-based or algorithm-based system relying on an evidence-based knowledge resource. We excluded CCDSSs that were not primarily focused on laboratory test ordering and did not directly communicate with the EHR. This meant that we excluded systems where patient characteristics needed to be manually introduced and systems that used paper-based reminders even if the data were processed electronically. We also limited our review to systems studied in real-life settings, used by clinicians (not students) in hospital, outpatient, or primary care settings. We excluded studies that focused uniquely on laboratory testing in a preventive care setting, such as Papanicolaou tests or fecal occult blood tests. We did not include papers studying systems that provided reminder messages for attendance at appointments or examinations. We included the following study designs as defined by the Cochrane Effective Practice and Organization of Care Group: randomized clinical trials, nonrandomized controlled trials, controlled before-after studies, and interrupted time series. (16)

Assessment of Study Quality

Only randomized clinical trials were assessed on quality using the Cochrane checklist for risk of bias instrument because all other included designs were considered to have a high risk of bias. (17) For each trial 7 areas for potential risk of bias were assessed, including random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, addressing of incomplete outcome data, selective reporting, and other potential sources of bias. Each item was scored as high, low, or unclear as proposed by the Cochrane risk of bias assessment tool.

Data Extraction

Data from all eligible trials were extracted by one reviewer and assessed by a second reviewer. Disagreements were resolved by consulting a third reviewer. The data extraction included elements such as study design, various study characteristics, study setting, various intervention characteristics, involvement of the software developers in the evaluation of the CCDSS, outcome types, and various outcome characteristics.

Data Synthesis and Analysis

We inventoried all the extracted data and classified these according to CCDSS type and outcome type. Most studies could be classified as using a system aimed at improving appropriateness of testing for drug monitoring (eg, suggesting next international normalized ratio examination for maintaining anticoagulant therapy), disease monitoring (eg, recommending glycated hemoglobin [A.sub.1c] [[HbA.sub.1c]] tests in patients overdue for this examination) or diagnosis (eg, recommendations on laboratory tests for the assessment of patients with a sore throat). For each study we inventoried the effect on the primary outcome and evaluated effectiveness using a strategy for dichotomizing this outcome used in previous studies. (9,11) If a statistically significant effect was found for the primary outcome, we considered the CCDSS to be effective. If more than one primary outcome was measured, we considered the CCDSS to be effective if a statistically significant effect was found for half or more of the primary outcomes. Post hoc, we defined that if the interventions, setting, and outcomes were deemed homogenous, a meta-analysis using a random-effects Mantel-Haenszel risk ratio was to be performed using Review Manager. (18) Pooled results with signs of statistical heterogeneity, defined as [I.sup.2] > 50%, (17) were not reported. Other results were narratively synthesized and reported. The quality of the evidence was appraised using GRADE. (19)


Our search yielded 5455 citations, of which, based on title and abstract, 377 were selected for full-text screening. We excluded another 334 papers and performed a quality assessment of 43 papers. With the high yield of high-quality studies and in contrast with our protocol, we chose to exclude those studies without a randomized controlled design because of their intrinsic high risk of bias. On this basis we excluded 15 studies and ultimately included 28 papers reporting on 23 studies in our review. Figure 1 shows the flow of our process and the Table lists the included studies.

Included Studies

Risk of Bias in Included Studies.--Of the 23 studies, 7 (30%) were randomized clinical trials with randomization on the patient level and 16 (70%) were randomized clinical trials with cluster randomization on the level of the health care provider or service. Figure 2 summarizes the risk of bias assessment for each included paper. Almost all studies could not sufficiently blind the health care providers to the intervention. Four studies (17%) attempted to blind the participants by introducing a CCDSS in both the intervention and the control group, but aimed at a different pathology (20-22) or with a different design. (23,24) Trials with randomization on the patient level could not guarantee the absence of performance bias because health care providers were not fully blinded to the intervention and cared for patients in both the intervention and control groups.

Setting and Participants.--Fourteen studies (61%) were conducted in the United States and Canada. (20-22,24-37) Eight studies (35%) were conducted in Europe. (23,38-45) One study (4%) was conducted in Europe and Australia. (46,47) Thirteen studies (57%) evaluated CCDSSs in primary care,* 5 (22%) in hospital outpatient ambulatory care, (26,31,33,36,44) and 5 (22%) in hospital inpatient care. (22,25,32,34)

Intervention.--Fifteen different CCDSSs were studied, of which 6 (40%) were focused solely on laboratory testing ([dagger]) 5 (33%) also included other reminders such as treatment options, (27-29,31,39,41-43) and 4 (27%) were not stand-alone systems, but were developed by an EHR software vendor. ([double dagger])

Of the CCDSSs focused solely on laboratory testing, 3 (50%) were targeted at regulating blood clotting in patients using anticoagulants. (32,33,38,40,44,46,47) Other targeted diseases or situations included diabetes, (27-29,31,39,42,43) hyperlipid emia, (39,45) human immunodeficiency virus (HIV) infection, (36) initiation or management of specific drug therapies, (20,21,26,30,35) sore throat, (41) urinary tract infections, (41) recommendations regarding multiple conditions, (23) and redundancy of laboratory tests. (25,34) All studies targeted at anticoagulation used either the PARMA 5 (Instrumentation Laboratory SpA, Milan, Italy), DAWN AC (4S Information Systems Ltd, Cumbria, England), or the Anticoagulation Management Support System (Softop Information Systems, Warwick, England).

Noteworthy is the finding that the 8 studies ([section]) reporting on a CCDSS integrated within the EHR used 1 of only 3 systems: the Longitudinal Medical Record as implemented at Brigham and Women's Hospital (Boston, Massachusetts), the Regenstrief Medical Record System (Indianapolis, Indiana), or the Kaiser Permanente EHR (Oakland, California). Besides these 3 early adopters that have integrated CCDSSs during the last 2 decades, no other systems with integrated CCDSSs were identified.

In 14 studies,([parallel]) the developer of the intervention was involved in the evaluation of its effect. In the studies where the developer was involved, 8 (57%) showed a significant effect on outcomes. In the 9 studies where the developers were not involved, only 2 (22%) showed a positive effect. Albeit not statistically significant, there appears to be a tendency that involving the evaluator of the CCDSS will generate a positive effect (risk ratio, 11.33; 95% CI, 0.73-175.10]).


Clinical Outcomes.--Diabetes.--Three studies reported [HbA.sub.1c] as a primary outcome for glycemic control in their studies. MacLean et al (27,28) studied the Vermont Diabetes Information System (Burlington, Vermont), which generates reminders based on the results from independent laboratories and sends these to the primary care physicians either through the EHR, by fax, or by mail, depending on their network capabilities. They could not show a significant effect on [HbA.sub.1c] values. The Diabetes Disease Management Application implemented in the Harvard University Adult Medicine Clinic (Cambridge, Massachusetts) includes reminders on [HbA.sub.1c] and cholesterol tests. During their 12-month trial, Meigs et al (31) could not show a significant effect on glycemic control. Of the 3 studies reporting on glycemic control, not a single one could demonstrate a significant effect on [HbA.sub.1c] levels. Hetlevik et al (43) studied a CCDSS service for the implementation of guidelines on diabetes and hypertension, including recommendations on diagnostics and laboratory tests. They noticed no significant effect on [HbA.sub.1c] values after 18 months. Overall, CCDSSs aimed at laboratory testing for diabetes appear not to have an effect on glycosylated hemoglobin, though this evidence is graded as weak because of important heterogeneity and imprecision (grade C).

Anticoagulation.--Of the studies evaluating a CCDSS service aimed at improving anticoagulation, six (32,33,38,40,44,46,47) reported the surrogate clinical outcome time in therapeutic range (TTR) and three (38,40,46,47) reported clinical (bleeding and thrombotic) events. The study by Poller et al, (46) the only study reporting clinical events as primary outcome, did not show any effect. The 2 other studies (38,40) reporting clinical events, albeit not as a primary outcome, were also unable to show a significant effect on clinical events (see Figure 3 for the pooled results). Computerized clinical decision support systems aimed at anticoagulation testing have not been able to show a significant effect on adverse effects (grade B).

Of the 5 studies reporting TTR as a primary outcome, only the studies by Manotti et al (44) and Mitra et al (32) showed a significant effect. Poller et al (46) reported TTR as a secondary outcome and found a small but statistically significant change of 0.7% difference (P = .02) in favor of the CCDSS arm. We did not attempt to meta-analyze these results despite similar CCDSS services and outcomes because of large heterogeneity in settings and means of reporting outcomes. Computerized clinical decision support systems aimed at anticoagulation testing may have a positive, but small, effect on TTR, but this evidence is considered of low quality because of heterogeneity of results and imprecision of results (grade C).

Human Immunodeficiency Virus.--We identified 1 study (36) that used a CCDSS for guiding laboratory testing in patients with HIV. This large study, following 1001 HIV-infected patients, reported the mean CD4 white blood cell count as primary outcome. This study demonstrated a positive effect on clinical control of HIV with an increase of the CD4 cell count by 2.0 cells/mm3/mo (95% CI, 0.1-4.0; P = .04). These results were downgraded to moderate quality (grade B) because of imprecision of results.

Process Outcomes: Compliance.--Thirteen studies reported compliance with recommendations as a primary outcome.([paragraph]) Eleven studies reported compliance as the percentage of appropriate tests or reported data from which this statistic could be calculated. In the study by Bates et al, (25) this was the rate of cancelled tests after a reminder was triggered signaling that the physician was about to order a redundant test. In all other cases, compliance was derived from the number of tests performed after a test was suggested by the CCDSS system. We did not meta-analyze the results because of heterogeneity of the populations studied and the types of intervention. Fifty-four percent (n = 7) reported a positive effect on compliance with recommendations. The scope of the CCDSS tended to influence the effect. Reminders aimed at influencing the laboratory test ordering behavior of 1 or 2 tests tended to have no effect or a smaller effect than reminders aimed at 3 or more conditions (see Figure 4). Computerized clinical decision support systems aimed at laboratory testing may have a positive effect on compliance with recommendations, but this evidence is considered of low quality (grade C) because of important heterogeneity of the results.

Economic Outcomes.--Three studies reported economic outcomes, namely Bates et al, (25) Smith et al, (21) and Khan et al (29) Bates et al (25) reported the possible charge savings when implementing a CCDSS that reminds a physician upon ordering a new test that the same test had already been ordered within certain test-specific intervals. They calculated that the 24% decrease in test ordering correlated to a $35 000 annual savings for the clinic. These findings suggest that a direct cost savings of $6.14 per patient per year is possible. Smith et al (21) investigated the costs of various strategies to improve laboratory monitoring of medications, including one arm that used a CCDSS within the Kaiser Permanente EHR aimed at primary care physicians. Based on costs for patient identification, chart reviews, resources required to contact patients, laboratory testing, result review of normal and abnormal tests, follow-up visits for abnormal tests, and service cost of letters, a cost for each arm was calculated and compared with the usual care group. They concluded that CCDSSs are probably not cost-effective. These conclusions were made based on the actual expenditures, not on a cost-benefit analysis including the cost of avoided events. Khan et al (29) studied the effects of the Vermont Diabetes Information System in ambulatory care on the use of emergency room and hospital resources. They showed that patients cared for with the Vermont Diabetes Information System had 11% lower hospital charges, stayed fewer days in hospitals, and had 25% lower emergency room charges than the control patients, resulting in a $24-26 (95% CI, 205-4647; P = .03) annual savings per patient. Current evidence suggests that CCDSSs may reduce direct and downstream costs of laboratory testing by amounts ranging from $6.14 to $2426 annually per patient.


This systematic review summarizes the evidence presented by randomized clinical trials on the effect of CCDSSs on laboratory test ordering of health care professionals in primary care, hospital outpatient care, and hospital inpatient care. There is no evidence that CCDSSs focused on changing laboratory test ordering behavior for diabetes or anticoagulation have an effect on clinical outcomes. Reported clinical outcomes were glycosylated hemoglobin for interventions targeting diabetes and clinical events and TTR for interventions targeting the management of anticoagulation. No study reporting clinical events as an outcome for the evaluation of a CCDSS focused on test ordering for anticoagulation therapy showed a significant effect, including a large multicenter trial conducted by Poller et al. (46,47) Although we were unable to pool the results for TTR, only a minority of studies reporting this outcome showed a significant effect. When implemented for the follow-up of HIV testing, Robbins et al (36) found that CCDSSs had a significant effect on improving CD4 cell count. Overall, CCDSSs aimed at improving laboratory test ordering behavior seem to have little or no effect on clinical outcomes. Studies reporting economic outcomes show an improvement on cost-effectiveness, but it must be noted that only studies that demonstrated an effect on clinical outcomes conducted an additional cost-effectiveness evaluation. This could be an important source of publication bias for this outcome measure.

We found an effect on compliance with recommendations in a majority of the studies reporting this process outcome. We defined compliance with recommendations as the percentage of recommended tests that were ordered or the percentage of redundant tests that were cancelled. The effect sizes varied strongly, ranging from the largest effect measured by van Wyk et al (45) (30% increase of appropriate ordering; risk ratio, 2.55; 95% CI, 2.26-2.87) to a negative effect measured by Hetlevik et al (42,43) (risk ratio, 0.9; 95% CI, 0.8-1.01). Bates et al (25) evaluated the effect of a CCDSS that notified when a potentially redundant laboratory test was ordered for 13 different tests. When triggered, the redundancy reminders reduced the percentage of redundant tests performed by half (27% of tests performed in the intervention group compared with 51% in the control group). Van Wijk et al (23) evaluated a CCDSS for the implementation of the Dutch College of General Practitioners guidelines on laboratory testing on laboratory test ordering behavior. This CCDSS included reminders based on 54 different guidelines and proposed laboratory tests based on a series of indications. When compared with a restricted laboratory test ordering form containing only the 15 most popular tests, the CCDSS reduced the average number of tests from 6.9 to 5.5 per order. A limitation to this study is that no formal evaluation of appropriateness was made. The authors assumed that reducing the number of tests implied a reduction in the number of inappropriate tests. Positive effects were also found by Feldstein et al (20) evaluating the effect of CCDSSs reminding physicians of 10 possible drug-laboratory interactions in Kaiser Permanente primary care practices. That this trend is not absolute is illustrated by the study by Matheny et al (30) studying a similar CCDSS aimed at 14 drug-laboratory interactions within the Longitudinal Medical Record at Brigham & Women's Hospital. They could not report a significant effect on appropriateness of testing, but an important note to be made is that the baseline rates of overdue testing were already very low before the implementation of the CCDSS, hence leaving very little margin for improvement.

We also found that when the developers of the CCDSS were included in the evaluation of their system, this tended to lead to a higher chance of finding positive effects. Roshanov et al (11) previously pointed out that involving the developers in the evaluation process may be a potential source of bias. Our findings add strength to this conclusion.

Strengths and Limitations of This Review

Our review has some important strengths. First, our literature review was very thorough, screening more than 5000 articles and hand searching reference lists of previously conducted systematic reviews for additional citations. Second, we limited the intervention specifically to CCDSSs aimed at changing the laboratory testing behavior of health care professionals, thus assuming that findings may be more generalizable than in previous systematic reviews. Finally, this study provides a comprehensive and clear summary of randomized clinical trials and explores several tendencies in results.

An important limitation to this study is that we were unable to report an overall effect because of various sources of heterogeneity. First, we observed large heterogeneity in settings due to the inclusion of multiple health care settings and countries with different baseline testing behaviors. Second, even though the functionality of all the studied CCDSSs may be very similar, because they are targeted at different conditions, their effectiveness varies largely and we were unable to make generalizable conclusions. In our protocol we specified that, if possible, we would meta-analyze results using extracted data for patient and process outcomes. When this strategy was not deemed appropriate because of heterogeneity, we chose to summarize the effect by dichotomizing the results as being positive if they showed a significant effect for at least half of the primary outcomes. This strategy for reporting has been used previously (11,48) but poses limitations to the interpretation of the results. We reported data from studies with process outcomes as a primary outcome and from studies with process outcomes as a secondary outcome if they showed a significant effect on their primary (clinical) outcomes. We refrained from reporting on every secondary outcome because we feared that including positive effects for secondary outcomes from studies that failed to show an effect on their primary outcome could be a potential source of bias. As a result, some studies reporting positive effects for some secondary outcomes were considered ineffective in our review. This may have led to an underestimation of the effect; however, our results appear consistent with other reviews on CCDSSs. Our review was not designed as a health technology assessment; hence, our findings regarding cost-effectiveness must be interpreted with care. We found some evidence suggesting that CCDSSs aimed at improving appropriateness of laboratory testing may be cost-effective. This evidence supports earlier conclusions that CCDSSs are potentially cost-effective, but lacking strong evidence. (12,49)

Evaluating the trustworthiness of the knowledge base used to design the reminders or recommendations in the CCDSS proved very difficult. Some authors clearly described the guidelines or recommendations used; however, in a large majority of the studies, this was not mentioned. Roshanov et al (50) evaluated multiple factors that may influence the effectiveness of CCDSSs and found that whether or not the advice was evidence-based did not significantly influence their effectiveness; hence, we chose not to exclude any studies on this basis.

Almost all studies on CCDSSs had difficulties in blinding the participants in their design. This poses a problem in studying the effects of CCDSSs in a randomized controlled design and analyzing its results. Simply by knowing that they are being studied, participants in both intervention and control groups tend to improve their clinical practice. This effect is also known as the Hawthorne effect. (51) We identified only one study (40) that was designed to correct for a possible Hawthorne effect by introducing a cluster randomized design on the physician level (with interpractice controls) and additionally randomizing patients within the cluster to the intervention or usual care (intrapractice controls). The interpractice control group was used to evaluate any Hawthorne effect in the intrapractice controls. Albeit Fitzmaurice et al (40) did not see a Hawthorne effect in their control group, it is to date unclear what the influence of this effect could be in trials investigating complex interventions like CCDSSs, especially when process outcomes are being investigated. Additionally, there is a risk that participants in the intervention groups may discuss the information provided by the CCDSS with participants in the control group, resulting in performance bias. That the inability to properly blind the participants and the Hawthorne effect may have a significant effect on results was painfully demonstrated in the study by Holmes et al (22) wherein a single nurse was responsible for a significant improvement in guideline implementation for the whole control group and subsequently cancelled a difference in effect with the intervention group. To correctly observe a true effect of CCDSSs, it remains imperative that researchers set up sound research designs taking into account the complexity of CCDSSs in order to minimize any risk of bias.


Our results are consistent with other reviews on the effect of CCDSSs on diagnostic testing. (11-13) Roshanov et al (11) concluded that, based on data from randomized controlled trials, CCDSSs can modify test-ordering behavior of health care professionals, but also concluded that generalizable results were impossible because of differences in implementation, system features, system design, and study design. Based on data from controlled studies and time series, Main et al (12) concluded that, when combining effects on primary and secondary outcomes, CCDSSs significantly improve process or practitioner performance outcomes without increasing harm or adverse effects. They also stressed the need for a framework defining types of CCDSS and recommendations on study design including reporting on relevant outcomes and incorporating process, patient, and economic evaluations. Based on controlled studies, Bright et al (13) found that CCDSSs can significantly change the clinical study ordering behavior of practitioners (odds ratio, 1.72; 95% CI, 1.47-2.00). The authors stated that the studies were heterogeneous but did not report a measure for heterogeneity; however, they concluded that the pooled effect was based on moderate evidence. Bright et al (13) also found statistical evidence for publication bias in their review. We could not reproduce this finding; instead, we found that the most recent studies reported little or no effect. Contrary to the results by Roshanov et al (50) in their review on factors of effective CCDSSs, we found that increasing the number of reminders tended to increase effectiveness. They suggested that increasing the number of alerts may result in practitioners' starting to ignore them as a result of "alert fatigue." We found that systems focusing on only 1 or 2 conditions tended to be less effective, implying that there is a subtle balance between underalerting and overalerting.

The patient and the context in which the clinician works, including the various processes involved in ordering laboratory tests, also influence physicians' testing behavior. Computerized clinical decision support systems interact in a complex process of decision making involving the physician, the EHR, and a rule-based knowledge base, providing information during multiple points in time during this process. Currently, studies on CCDSSs barely address these issues. We looked at the effect of including the software developers in the implementation and study process as a measure of integration, but this is not a sufficient measure of effective implementation. Computerized clinical decision support systems address only a physician's possible knowledge gap; they do not offer a solution for other reasons why recommended care is not adhered to, such as attitudes or behavioral constraints. (52,53) Cointerventions such as educational strategies or resources assuring the trustworthiness of the CCDSS could also influence their effectiveness. A more comprehensive framework evaluating the features of successful CCDSS implementation would help to guide those involved in developing, implementing, and evaluating CCDSSs. The GUIDES project (54) aims to develop such a framework and may prove to be an important tool for CCDSS implementation.


Computerized clinical decision support systems aimed at improving the laboratory test ordering of health care professionals have shown little or no effect on clinical outcomes and may have an effect on some process outcomes; however, data remain sparse and not generalizable. A sound framework defining various system designs, allowing comparison of effects, and avoiding implementation errors, as well as recommendations on study designs and relevant outcomes to avoid evaluation errors, would greatly improve reporting on CCDSSs.

The authors would like to acknowledge and thank Marleen Michels for her aid and contribution to the systematic search strategy.


(1.) Health, United States, 2014: With Special Feature on Adults Aged 55-64. Hyattsville, MD: National Center for Health Statistics;2015.

(2.) Health Care Spending and the Medicare Program. Washington, DC: Medicare Payment Advisory Commission;2014:1-199.

(3.) Wolcott J, Schwartz A, Goodman C. Laboratory Medicine: A National Status Report. Prepared for Centers for Disease Control and Prevention. Falls Church, VA: The Lewin Group;2008.

(4.) Zhi M, Ding EL, Theisen-Toupal J, Whelan J, Arnaout R. The landscape of inappropriate laboratory testing: a 15-year meta-analysis. PLoS One. 2013;8(11): e78962. doi:10.1371/journal.pone.0078962.

(5.) Rang M. The Ulysses syndrome. Can Med Assoc J. 1972;106(2):122-123.

(6.) Koch H, van Bokhoven MA, ter Riet G, et al. Ordering blood tests for patients with unexplained fatigue in general practice: what does it yield?: results of the VAMPIRE trial. Br J Gen Pract. 2009;59(561):e93-e100. doi:10.3399/bjgp09X420310.

(7.) Neilson EG, Johnson KB, Rosenbloom ST, et al. The impact of peer management on test-ordering behavior. Ann Intern Med. 2004;141(3):196-204.

(8.) Attali M, Barel Y, Somin M, et al. A cost-effective method for reducing the volume of laboratory tests in a university-associated teaching hospital. Mt Sinai J Med. 2006;73(5):787-794.

(9.) Haynes RB, Wilczynski NL. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: methods of a decision-maker-researcher partnership systematic review. Implement Sci. 2010; 5(1):12. doi:10.1186/1748-5908-5-12.

(10.) Cadogan SL, Browne JP, Bradley CP, Cahill MR. The effectiveness of interventions to improve laboratory requesting patterns among primary care physicians: a systematic review. Implement Sci. 2015;10(1):167. doi:10.1186/s13012-015-0356-4.

(11.) Roshanov PS, You JJ, Dhaliwal J, etal. Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior?: a decision-maker-researcher partnership systematic review. Implement Sci. 2011;6: 88. doi:10.1186/1748-5908-6-88.

(12.) Main C, Moxham T, Wyatt JC, Kay J, Anderson R, Stein K. Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effectiveness of systems. Health Technol Assess. 2010;14(48):1-227. doi:10.3310/hta14480.

(13.) Bright TJ, Wong A, Dhurjati R, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29-43. doi:10.7326/0003-4819-157-1-201207030-00450.

(14.) Lugtenberg M, Pasveer D, van der Weijden T, Westert GP, Kool RB. Exposure to and experiences with a computerized decision support intervention in primary care: results from a process evaluation. BMC Fam Pract. 2015;16:141. doi:10.1186/s12875-015-0364-0.

(15.) Delvaux N, Van Thienen K, Heselmans A, Van de Velde S, Raemaekers D, Aertgeerts B. The effectiveness of computerized clinical decision support systems in improving appropriate laboratory test ordering. PROSPERO. 2015: CRD42015015970. doi:10.15124/CRD4201 5015970.

(16.) Effective Practice and Organisation of Care (EPOC). EPOC resources for review authors. Published 2015. Accessed March 1, 2016.

(17.) Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. The Cochrane Collaboration;2011. Accessed March 17, 2016.

(18.) Review Manager (RevMan) [computer program]. Version 5.3. Copenhagen, Denmark: The Nordic Cochrane Centre, The Cochrane Collaboration;2014.

(19.) Guyatt GH, Oxman AD, Vist GE, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008; 336(7650):924-926. doi:10.1136/bmj.39489.470347.AD.

(20.) Feldstein AC, Smith DH, Perrin N, et al. Improved therapeutic monitoring with several interventions: a randomized trial. Arch Intern Med. 2006;166(17): 1848-1854. doi:10.1001/archinte.166.17.1848.

(21.) Smith DH, Feldstein AC, Perrin NA, et al. Improving laboratory monitoring of medications: an economic analysis alongside a clinical trial. Am J Manag Care. 2009;15(5):281-289.

(22.) Holmes JF, Freilich J, Taylor SL, Buettner D. Electronic alerts for triage protocol compliance among emergency department triage nurses: a randomized controlled trial. Nurs Res. 2015;64(3):226-230. doi:10.1097/NNR. 0000000000000094.

(23.) van Wijk MAM, van der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH. Assessment of decision support for blood test ordering in primary care: a randomized trial. Ann Intern Med. 2001;134(4):274-281. doi:10.7326/00034819-134-4-200102200-00010.

(24.) Zera CA, Bates DW, Stuebe AM, Ecker JL, Seely EW. Diabetes screening reminder for women with prior gestational diabetes: a randomized controlled trial. Obstet Gynecol. 201 5;126(1):109-1 14. doi:10.1097/AOG. 0000000000000883.

(25.) Bates DW, Kuperman GJ, Rittenberg E, et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med. 1999;106(2):144-150.

(26.) Lo HG, Matheny ME, Seger DL, Bates DW, Gandhi TK. Impact of noninterruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc. 2009;16(1):66-71. doi:10.1197/jamia.M2687.

(27.) MacLean CD, Gagnon M, Callas P, Littenberg B. The Vermont diabetes information system: a cluster randomized trial of a population based decision support system. J Gen Intern Med. 2009;24(12):1303-1310. doi:10.1007/s11 606009-1147-x.

(28.) MacLean CD, Littenberg B, Gagnon M. Field action report. Diabetes decision support: initial experience with the Vermont Diabetes Information System. Am J Public Health. 2006;96(4):593-595. doi:10.2105/AJPH.2005. 065391.

(29.) Khan S, MacLean CD, Littenberg B. The effect of the Vermont Diabetes Information System on inpatient and emergency room use: results from a randomized trial. Health Outcomes Res Med. 2010;1(1):e61-e66. doi:10.1016/j. ehrm.2010.03.002.

(30.) Matheny ME, Sequist TD, Seger AC, et al. A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008;15(4):424-429. doi:10.1197/jamia.M2602.

(31.) Meigs JB, Cagliero E, Dubey A, et al. A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project. Diabetes Care. 2003;26(3):750-757.

(32.) Mitra R, Marciello MA, Brain C, Ahangar B, Burke DT. Efficacy of computer-aided dosing of warfarin among patients in a rehabilitation hospital. Am J Phys Med Rehabil. 2005;84(6):423-427.

(33.) Nieuwlaat R, Hubers LM, Spyropoulos AC, et al. Randomised comparison of a simple warfarin dosing algorithm versus a computerised anticoagulation management system for control of warfarin maintenance therapy. Thromb Haemost. 2012;108(6):1228-1235. doi:10.1160/TH12-06-0433.

(34.) Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of "corollary orders" to prevent errors of omission. J Am Med Inform Assoc. 1997; 4(5):364-375. d10.1136/jamia.1997.0040364

(35.) Palen TE, Raebel M, Lyons E, Magid DM. Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders. Am J Manag Care. 2006;12(7):389-395.

(36.) Robbins GK, Lester W, Johnson KL, et al. Efficacy of a clinical decision-support system in an HIV practice: a randomized trial. Ann Intern Med. 2012; 157(11):757-766. doi:10.7326/0003-4819-157-11-201212040-00003.

(37.) Sequist TD, Gandhi TK, Karson AS, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc. 2005;12(4):431-437. doi:10.1197/jamia. M1788.

(38.) Claes N, Buntinx F, Vijgen J, et al. The Belgian Improvement Study on Oral Anticoagulation Therapy: a randomized clinical trial. Eur Heart J. 2005;26(20): 2159-2165. doi:10.1093/eurheartj/ehi327.

(39.) Eccles M, McColl E, Steen N, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325(7370):941.

(40.) Fitzmaurice DA, Hobbs FD, Murray ET, Holder RL, Allan TF, Rose PE. Oral anticoagulation management in primary care with the use of computerized decision support and near-patient testing: a randomized, controlled trial. Arch Intern Med. 2000;160(15):2343-2348.

(41.) Flottorp S, Oxman AD, Havelsrud K, Treweek S, Herrin J. Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ. 2002;325(7360):367.

(42.) Hetlevik I, Holmen J, Kruger O. Implementing clinical guidelines in the treatment of hypertension in general practice: evaluation of patient outcome related to implementation of a computer-based clinical decision support system. Scand J Prim Health Care. 1999;17(1):35-40.

(43.) Hetlevik I, Holmen J, Kruger O, Kristensen P, Iversen H, Furuseth K. Implementing clinical guidelines in the treatment of diabetes mellitus in general practice: evaluation of effort, process, and patient outcome related to implementation of a computer-based decision support system. Int J Technol Assess Health Care. 2000;16(1):210-227.

(44.) Manotti C, Moia M, Palareti G, Pengo V, Ria L, Dettori AG. Effect of computer-aided management on the quality of treatment in anticoagulated patients: a prospective, randomized, multicenter trial of APROAT (Automated PRogram for Oral Anticoagulant Treatment). Haematologica. 2001;86(10):1060-1070.

(45.) van WykJT, van Wijk MA, Sturkenboom MC, Mosseveld M, Moorman PW, van der Lei J. Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation. 2008; 117(3):371-378. doi:10.1161/CIRCULATIONAHA.107.697201.

(46.) Poller L, Keown M, Ibrahim S, et al. A multicentre randomised clinical endpoint study of PARMA 5 computer-assisted oral anticoagulant dosage. Br J Haematol. 2008;143(2):274-283. doi:10.1111/j.1365-2141.2008.07337.x.

(47.) Poller L, Keown M, Ibrahim S, et al. An international multicenter randomized study of computer-assisted oral anticoagulant dosage vs. medical staff dosage. J Thromb Haemost. 2008;6(6):935-943. doi:10.1111/j.1538-7836. 2008.02959.x.

(48.) Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765. doi:10.1136/bmj. 38398.500764.8F.

(49.) Lobach D, Sanders GD, Bright TJ, et al. Enabling Health Care Decisionmaking Through Clinical Decision Support and Knowledge Management. Rockville, MD: Agency for Healthcare Research and Quality; 2012:1-784.

(50.) Roshanov PS, Fernandes N, Wilczynski JM, et al. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346:f657. doi:10.1136/bmj.f657.

(51.) McCarney R, Warner J, Iliffe S, van Haselen R, Griffin M, Fisher P. The Hawthorne Effect: a randomised, controlled trial. BMC Med Res Methodol. 2007; 7:30. doi:10.1186/1471-2288-7-30.

(52.) Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines?: a framework for improvement. JAMA. 1999;282(15): 1458-1465.

(53.) Lugtenberg M, Zegers-van SchaickJM, Westert GP, Burgers JS. Why don't physicians adhere to guideline recommendations in practice?: an analysis of barriers among Dutch general practitioners. Implement Sci. 2009;4:54. doi:10. 1186/1748-5908-4-54.

(54.) Van de Velde S, Roshanov P, Kortteisto T, et al. Tailoring implementation strategies for evidence-based recommendations using computerised clinical decision support systems: protocol for the development of the GUIDES tools. Implement Sci. 2016;11:29. doi:10.1186/s13012-016-0393-7.

(55.) State of Health in the EU. European Commission, Public Health Web site. Accessed May 25, 2016. Nicolas Delvaux, MD; Katrien Van Thienen, MD; Annemie Heselmans, PhD; Stijn Van de Velde, PhD; Dirk Ramaekers, MD, PhD; Bert Aertgeerts, MD, PhD

Accepted for publication August 10, 2016.

Supplemental digital content is available for this article at www. in the April 2017 table of contents.

From the Department of Public Health and Primary Care, KU Leuven, Leuven, Belgium (Drs Delvaux, Heselmans, Ramaekers, and Aertgeerts);the Department of Public Health, Vrije University Brussels, Brussels, Belgium (Dr Van Thienen); the GUIDES project, Norwegian Institute of Public Health, Oslo, Norway (Dr Van de Velde); and the Centre for Evidence-Based Medicine (CEBAM), Belgian Branch of the Dutch Cochrane Collaboration, Leuven, Belgium (Drs Ramaekers and Aertgeerts).

The authors have no relevant financial interest in the products or companies described in this article.

Reprints: Nicolas Delvaux, MD, Department of Public Health and Primary Care, KU Leuven, Kapucijnenvoer 33, Blok J PB 7001, B3000 Leuven, Belgium (email: * References 20, 21, 23, 24, 27-30, 35, 37-43, 45.

([dagger]) References 23, 32, 33, 36, 38, 40, 44-47.

([double dagger]) References 20-22, 24-26, 30, 34, 35, 37.

([section]) References 20, 21, 25, 26, 30, 34, 35, 37.

([parallel]) References 20, 21, 23-31, 34, 36, 37, 41-43, 45.

([paragraph]) References 20, 22-26, 30, 34, 35, 37, 39, 41, 42, 45.

Please Note: Illustration(s) are not available due to copyright restrictions

Caption: Figure 1. Flow diagram of included and excluded studies.

Caption: Figure 2. Summary of risk of bias of all included studies. The risk of bias assessment of MacLean 2009 was the same as for their 2006 paper. Green indicates low risk of bias, orange indicates moderate risk of bias, and red indicates high risk of bias. Caption: Figure 4. Forest plot summarizing studies reporting compliance as outcome. The x-axis shows the effect on risk ratio, where results more than 1 favor the intervention group and results less than 1 favor the control group. Studies above the line report on a computerized clinical decision support system (CCDSS) targeting 1 or 2 conditions; studies below the line report on a CCDSS targeting 3 or more conditions.
Characteristics of Included Studies

Source, y        Setting       No. of Patients/     Intervention
                               No. of               (a)

Bates et         Hospital      11 586/NR            LMR: reminds
al, (25)         inpatient                          physician that a
1999                                                test has previously
                                                    been ordered
                                                    within test-
                                                    specific interval

Claes et         Primary       834/96               DAWN AC:
al, (38)         care                               calculates dosage
2005                                                anticoagulant
                                                    based on INR
                                                    values and
                                                    proposes next test

Eccles et        Primary       2335/279b            PRODIGY: triggers
al, (39)         care                               guidelines,
2002                                                presents patient
                                                    scenarios, and
                                                    offers suggestions
                                                    for management

Feldstein        Primary       961/253              KP EHR: electronic
et al, (20)      care                               messages
2006; Smith                                         recommending
et al, (21)                                         laboratory
2009                                                monitoring

Fitzmaurice      Primary       367/NR               AMSS: calculates
et al, (40)      care                               dosage
2000                                                anticoagulant
                                                    based on INR
                                                    values and
                                                    proposes next test

Flottorp et      Primary       16 939 (sore         Mediata: reminders
al, (41)         care          throat), 9887        on laboratory tests
2002                           (UTI)/620            and management
                                                    of sore throat and

Hetlevik         Primary       2239                 InfoTech: guides in
et al,           care          (hypertension),      diagnostics,
(42,43)                        1034                 history taking,
1999, 2000                     (diabetes)/53        physical
                                                    additional tests,
                                                    and treatment

Holmes et        Primary       19 157/28            Alert provided to
al, (22)         care                               triage nurse with
2015                                                recommendations
                                                    to follow triage
                                                    protocol on
                                                    radiology and
                                                    laboratory test

Lo et al,        Primary       2757/366             LMR: reminders on
(26) 2009        care                               baseline tests for

MacLean et       Hospital      7412/132             VDIS: reminders for
al, (27,28)      outpatient                         overdue
2009, 2006;                                         laboratory tests
Khan et al,                                         recommended for
(29) 2010                                           monitoring

Manotti et       Hospital      1358/NR              PARMA 5:
al, (44)         outpatient                         calculates dosage
2001                                                anticoagulant
                                                    based on INR
                                                    values and
                                                    proposes next test

Matheny et       Primary       1922/303             LMR: reminders
al, (30)         care                               on laboratory
2008                                                monitoring of

Meigs et         Hospital      598/66               DDMA: reminders
al, (31)         outpatient                         on diagnosis
2003                                                and
                                                    management of

Mitra et         Hospital      30/NR                DAWN AC:
al, (32)         inpatient                          calculates
2005                                                dosage
                                                    based on INR
                                                    values and
                                                    proposes next
                                                    test date

Nieuwlaat        Hospital      1068/NR              DAWN AC:
et al, (33)      outpatient                         calculates
2012                                                dosage
                                                    based on INR
                                                    values and
                                                    proposes next
                                                    test date

Overhage et      Hospital      2181/89              RMRS: system
al, (34)         inpatient                          suggests
1997                                                corollary orders
                                                    based on patient

Palen et         Primary       26 586/207           KP EHR:
al, (35)         care                               reminders on
2006                                                laboratory
                                                    monitoring of
                                                    drugs at
                                                    initiation of

Poller et        Hospital      13 219/NR            DAWN AC and
al, (46,47)      outpatient                         PARMA 5:
2008                                                calculates
                                                    based on INR
                                                    values and
                                                    proposes next
                                                    test date

Robbins et       Hospital      1011/33              Virology FasTrack:
al, (36)         outpatient                         generates alerts
2012                                                of potential
                                                    adverse events

Sequist et       Primary       6243/194             LMR: reminders
al, (37)         care                               on diagnosis
2005                                                and
                                                    management of
                                                    diabetes and
                                                    coronary heart

van Wijk et      Primary       NR/60                Bloodlink:
al, (23)         care                               proposes order
2001                                                sets based on
                                                    guidelines on

van Wyk et       Primary       92 054/80            CholGate: reminders
al, (45)         care                               on primary and
2008                                                secondary
                                                    prevention of
                                                    disease with
                                                    regard to lipid

Zera et          Primary       6439/841             LMR: reminds
al, (24)         care                               screening for
2015                                                diabetes in
                                                    women with
                                                    history of


Source, y         Software    Process    Clinical    Economic

Bates et            Yes         Yes         Yes         Yes
al, (25)

Claes et             No          No         Yes         No
al, (38)

Eccles et            No         Yes         No          No
al, (39)

Feldstein           Yes         Yes         No          Yes
et al, (20)
2006; Smith
et al, (21)

Fitzmaurice          No          No         Yes         No
et al, (40)

Flottorp et         Yes         Yes         No          No
al, (41)

Hetlevik            Yes         Yes         Yes         No
et al,
1999, 2000

Holmes et            No         Yes         No          No
al, (22)

Lo et al,           Yes         Yes         No          No
(26) 2009

MacLean et          Yes         Yes         Yes         Yes
al, (27,28)
2009, 2006;
Khan et al,
(29) 2010

Manotti et           No         Yes         Yes         No
al, (44)

Matheny et          Yes         Yes         No          No
al, (30)

Meigs et            Yes         Yes         No          No
al, (31)

Mitra et             No         Yes         Yes         No
al, (32)

Nieuwlaat            No         Yes         Yes         No
et al, (33)

Overhage et         Yes         Yes         No          No
al, (34)

Palen et             No         Yes         No          No
al, (35)

Poller et            No          No         Yes         No
al, (46,47)

Robbins et          Yes          No         Yes         No
al, (36)

Sequist et          Yes         Yes         No          No
al, (37)

van Wijk et         Yes         Yes         No          No
al, (23)

van Wyk et          Yes         Yes         No          No
al, (45)

Zera et             Yes         Yes         No          No
al, (24)

Abbreviations: AMSS, Anticoagulation Management Support System;
DDMA, Diabetes Disease Management Support System; INR,
international normalized ratio; KP EHR, Kaiser Permanente
Electronic Health Record; LMR, Longitudinal Medical Record; NR, not
reported; PRODIGY, Prescribing Rationally With Decision Support in
General Practice Study; RMRS, Regenstrief Medical Record System;
UTI, urinary tract infection; VDIS, Vermont Diabetes Information

(a) Longitudinal Medical Record, Boston, Massachusetts; DAWN AC, 4S
Information System Ltd, Cumbria, England; Kaiser Permanente
Electronic Health Record, Oakland, California; Diabetes Disease
Management Support System, Cambridge, Massachusetts; Regenstrief
Medical Record System, Indianapolis, Indiana; Vermont Diabetes
Information System, Burlington, Vermont; PARMA 5, Instrumentation
Laboratory SpA, Milan, Italy; Anticoagulation Management Support
System, Softop Information Systems, Warwick, England; Prescribing
Rationally With Decision Support in General Practice Study,
Newcastle upon Tyne, England.

(b) Calculated from number of
practices and mean or median amount of whole/full-time-equivalent
general practitioners.

Figure 3. Forest plot of studies (Claes et al, (38) Fitzmaurice et
al, (40) and Poller et al (46,47)) reporting adverse (bleeding and
thrombotic) effects as outcome for computerized clinical decision
support systems (CCDSSs) aimed at anticoagulation testing. The
x-axis shows the effect on risk ratio, where results more than 1
favor the intervention group and results less than 1 favor the
control group. Heterogeneity: [chi square] = 1.26, df = 2, P =.53,
[I.sup.2] = 0%. Test for overall effect: Z =.66 P = .51.
Abbreviation: M-H, Mantel-Haenszel.

Caption: Figure 3. Forest plot of studies (Claes et al,
38 Fitzmaurice et al, (40) and Poller et al (46,47) reporting
adverse (bleeding and thrombotic) effects as
outcome for computerized clinical decision support systems
(CCDSSs) aimed at anticoagulation testing. The x-axis shows
the effect on risk ratio,
where results more than 1 favor the intervention group and
results less than 1 favor the control group. Heterogeneity:
[chi square] = 1.26, df =2, P = .53, [I.sup.2] =
0%. Test for overall effect: Z = .66 P = .51. Abbreviation:
M-H, Mantel-Haenszel.

                             CCDSS              Control

Study or               Events    Total     Events    Total

Claes 2005                  5       70          7       60
Fitzmaurice 2000            3       87         10      166
Poller 2008               260     9353        267     9264

Total (95% CI)                    9510                9490
Total events              268                 284

                      Control          Risk Ratio

Study or               Weight      M-H, Random, 95% CI

Claes 2005               2.3%       0.61 [0.20, 1.83]
Fitzmaurice 2000         1.7%       0.57 [0.16, 2.03]
Poller 2008             96.0%       0.96 [0.82, 1.14]

Total (95% CI)         100.0%       0.95 [0.80, 1.12]
Total events
COPYRIGHT 2017 College of American Pathologists
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2017 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Delvaux, Nicolas; Van Thienen, Katrien; Heselmans, Annemie; Van de Velde, Stijn; Ramaekers, Dirk; Ae
Publication:Archives of Pathology & Laboratory Medicine
Date:Apr 1, 2017
Previous Article:New-Generation Thromboelastography: Comprehensive Evaluation of Citrated and Heparinized Blood Sample Storage Effect on Clot-Forming Variables.
Next Article:Acquired Cystic Disease-Associated Renal Cell Carcinoma: Review of Pathogenesis, Morphology, Ancillary Tests, and Clinical Features.

Terms of use | Privacy policy | Copyright © 2018 Farlex, Inc. | Feedback | For webmasters