Printer Friendly

Control outcomes and exposures for improving internal validity of nonrandomized studies.

There is an ongoing debate about the evidentiary value of nonrandomized study results. Much of this has stemmed from the controversy surrounding the Women's Health Initiative trial of hormone replacement therapy and risk for coronary heart disease (Manson et al. 2003). Although other efforts have demonstrated that well-conducted nonrandomized studies can generate results equivalent with randomized trials (Benson and Hartz 2000; Concato, Shah, and Horwitz 2000; Furlan et al. 2008; Hernan et al. 2008), this perspective is far from universally shared because several meta-analyses have reported substantial discordance between randomized and nonrandomized studies (Ioannidis et al. 2001; Ioannidis 2005).

Efforts have been made in recent years to address concerns regarding the internal validity of nonrandomized studies, including the widespread adoption of new-user designs (Ray 2003), use of sample restriction (Schneeweiss et al. 2007), and use of statistical methods such as propensity scores, instrumental variables, and marginal structural models. These statistical tools are often a last resort, because internal validity threats of selection bias (also known as unobserved confounding) were not addressed via study design or measurement strategies. Further, any assessment of causality in a nonrandomized study relies on assumptions about statistical models and their specification that must be guided by subject-matter knowledge (Robins 2001; Hernan et al. 2002). Researchers using secondary data for comparative effectiveness research have to address limitations regarding a lack of information on known confounders (since known confounders may be unmeasured) and potential gaps in subject-matter knowledge that reduce the likelihood of estimating causal effects without bias (Brookhart et al. 2010b).

Nevertheless, researchers may have adequate subject-matter knowledge to be able to identify outcomes that are not expected to change in response to the intervention of interest, which have been referred to as control outcomes or nonequivalent outcomes. Inclusion of control outcomes in nonrandomized studies can be a potentially useful strategy for detecting selection bias by expanding the measurement set beyond outcomes expected to change in response to the exposure or treatment of interest. Researchers may also be able to identify treatments that are known to be unrelated to the primary outcome (analogous to placebos). These control exposures are additional tools that researchers could use to assess selection bias, as is routinely done in the economics literature (Basker 2005; Holmlund, McNally, and Viarengo 2010; Rothstein 2010). Unfortunately, control outcomes and control exposures are under-utilized in nonrandomized comparative effectiveness research, although their use and advantages in this area have been described previously (Brookhart et al. 2010a; Lipsitch, Tchetgen Tchetgen, and Cohen 2010; Prasad and Jena 2013).

The purpose of this paper is to introduce control outcomes and exposures to a wider audience by summarizing illustrative examples from prior studies, which may provide readers with insights for identification of control outcomes and exposures in their own work. We conclude with recommendations for the conduct and reporting of studies that employ control outcomes or exposures, and present a framework for identifying them using Sir Austin Bradford Hill's factors for assessing causation (Hill 1965). We expect this overview to be of interest to researchers, manuscript reviewers, and grant reviewers seeking to improve the rigor and internal validity of comparative effectiveness research studies using nonrandomized study designs.


To identify published studies that employed control outcomes or exposures, we searched MEDLINE (via PubMed) and Google Scholar for manuscripts that included terms related to control outcomes, control exposures, falsification endpoints, falsification tests, and nonequivalent outcomes or exposures. We also included articles previously known or produced by the study team that were not identified in our prior search since there are no Medical Subject Heading (MeSH) terms for control outcomes or control exposures. We then manually searched bibliographies and works citing selected articles and consulted with colleagues to guide further study selection.

Identified studies were retained if they clearly identified the control outcome or exposure on the basis of its stated purpose for inclusion in the analysis. From our search we identified 11 studies that utilized control outcomes, control exposures, or both (Table 1). For each of these studies, we abstracted the following information: year of publication, control outcome, control exposure, primary study endpoint, primary study exposure/treatment, primary study effect measure, control outcome effect, control exposure effect, the proposed source of bias (if named), and the causal criteria (if known). We used this information to summarize results from eight studies that used control outcomes, two studies that used control exposures, and one study that used both.


Studies Using Control Outcomes

Control outcomes can be either negative (i.e., outcomes known to be unaffected by the treatment under study) or positive (i.e., outcomes known to be affected by treatment). The nine studies that used negative control outcomes chose outcomes that were not anticipated to be related to exposure (McClellan, McNeil, and Newhouse 1994; Redelmeier, Scales, and Kopp 2005; Jackson et al. 2006; Brookhart et al. 2007; Rasmussen, Chong, and Alter 2007; Mauri et al. 2008; Maciejewski et al. 2010; Patrick et al. 2011; Jena, Sun, and Goldman 2013) and were sorted into two groups. Four studies found that the association of the treatment with the negative control outcome was null (as expected) and provided greater confidence that the treatment effect/association with the primary outcome was not biased by unobserved confounding (Redelmeier, Scales, and Kopp 2005; Rasmussen, Chong, and Alter 2007; Mauri et al. 2008; Maciejewski et al. 2010). Five studies found that the treatment was unexpectedly associated with the negative control outcomes, which raised concerns about residual bias in the treatment effect/association on the primary outcome (McClellan, McNeil, and Newhouse 1994; Jackson et al. 2006; Brookhart et al. 2007; Patrick et al. 2011;Jena, Sun, and Goldman 2013).

Studies by Redelmeier, Mauri, Rasmussen, and Maciejewski were represented by this first group. Redelmeier, Scales, and Kopp (2005) compared the risk of acute myocardial infarction (AMI) or death among patients receiving either atenolol or metoprolol following elective surgery. They included several postsurgical noncardiac complications (wound infection, ileus, pneumonia, aspiration, respiratory failure, renal failure, delirium) as negative control outcomes and found no differences in these negative controls by beta-blocker received. Mauri et al. (2008) compared mortality, myocardial infarction (MI), and target-vessel revascularization within 2 years among patients receiving a drug-eluding stent or a bare metal stent. Mortality during the first 2 days following stent placement was the negative control outcome, since benefits of using one therapy over the other would not be expected in the immediate postsurgery period. They found no differences in 2-day mortality as expected, so concluded that the association between treatment and primary outcome was unconfounded.

Rasmussen, Chong, and Alter (2007) compared long-term mortality postAMI among patients with high, intermediate, and low levels of adherence to statins or beta-blockers. They included cancer-related hospital admissions as a control outcome as adherence to statins or beta-blockers was not hypothesized to increase this risk. As expected, they found no association between treatment adherence and cancer hospitalizations, suggesting that healthy adherer bias was not likely to be influencing their findings. Finally, Maciejewski et al. (2010) evaluated the impact of a value-based insurance design scheme (lower prescription drug copayments) on refill adherence to angiotensin-converting-enzyme (ACE) inhibitors, beta-blockers, calcium channel blockers, statins, and diuretics, comparing patients in plans that did and did not implement copayment changes. Angiotensin receptor blockers and cholesterol absorption inhibitors were the negative control outcomes because copayment changes for these drugs were comparable between patients in the two arms. As expected, they found no difference in adherence to these drugs, so concluded that the association between copayment reduction and increased refill adherence to the other drugs was unconfounded.

The studies by McClellan, Jackson, Brookhart, Patrick, and Jena were represented by the second group in which the treatment was unexpectedly associated with the negative control outcomes. The study by McClellan, McNeil, and Newhouse (1994) evaluated the impact of cardiac catheterization on mortality within 4 years following an AMI. They utilized mortality at 1 day post-AMI as a negative control outcome, hypothesizing that effects appearing on the first day post-AMI were unlikely to be related to catheterization and revascularization but to other aspects of treatment that correlated with the procedures. Unlike the null finding of a similar "early effect of treatment" from the Mauri study, they found a significant difference in 1-day mortality, indicating residual confounding.

Next, Jackson et al. (2006) evaluated the impact of influenza vaccination on mortality and influenza/pneumonia-related hospitalizations during influenza season. They used hospitalizations for injury or trauma during influenza season as their primary negative control outcomes. In a novel use of timing as a negative control outcome, they also used mortality and influenza/pneumonia-related hospitalizations in the pre-influenza season as a negative control outcome because influenza vaccination was not expected to affect these outcomes before influenza season started. They found that influenza vaccination was negatively associated with (i.e., protective against) hospitalizations for injuries and trauma during influenza season and with mortality and influenza/ pneumonia-related hospitalizations in the pre-influenza season. These unexpected findings suggested that the association between influenza vaccination and mortality and influenza/pneumonia-related hospitalizations during influenza season was likely confounded.

The two papers by Brookhart et al. (2007) and Patrick et al. (2011) used the occurrence of burns, asthma, and gastrointestinal bleeding as negative control outcomes when evaluating the effect of statins and statin adherence. They argued that there is no biologically plausible rationale or causal pathway through which statins would impact the likelihood of these negative control outcomes (Patrick et al. 2011). Thus, significant associations between statins and these negative control outcomes could suggest residual confounding in the relationship between statins and other outcomes of interest (e.g., mortality) in prior studies (Aronow et al. 2001; Stenestrand and Wallentin 2001). They found significant differences in multiple control outcomes, including increased preventative services use (bone mineral density testing, fecal-occult blood tests, mammography, and influenza and pneumonia vaccinations) and clinical outcomes (asthma, burns, falls, fractures, motor vehicle accidents, wounds, gastrointestinal bleeding, skin infections) for patients who were adherent to statins, suggesting that other outcomes were likely confounded via healthy adherer/healthy user bias.

Finally, Jena, Sun, and Goldman (2013) evaluated the incidence of community acquired pneumonia among patients who did and did not use proton pump inhibitors (PPIs). They selected several negative control outcomes that had no biologically plausible relationship with PPIs, including osteoarthritis, chest pain, urinary tract infections, deep vein thrombosis, skin infections, and rheumatoid arthritis. They found associations between each of the selected outcomes and PPIs, suggesting that there was possible confounding by indication or disease severity that was unaccounted for in their study.

It should be noted that these five papers, in which the treatment was unexpectedly associated with the negative control outcomes did not then attempt to statistically adjust for this evidence of unobserved confounding. Instead, each of these papers noted that the treatment effect on the primary outcome was likely biased, and that future work was needed to improve upon these estimates given significant treatment effects/associations in the primary outcome.

Studies Using Control Exposures

An additional approach to improving the internal validity of nonrandomized studies is to use control exposures, which are treatments that are expected to have no effect on the outcome of interest (analogous to a placebo). We identified three studies in our review that used negative control exposures (Table 2). Zaadstra et al. (2008) evaluated childhood infections that could be possible causes of multiple sclerosis. To address the possibility of recall bias in their patient survey, they used several negative control exposures, including broken arms, concussions, and tonsillectomy. Two of the negative control exposures (concussions and tonsillectomy) were associated with the later development of multiple sclerosis, which they interpreted as evidence of recall bias since neither childhood event was plausibly related to multiple sclerosis.

Dusetzina et al. (2013) used aromatase inhibitor initiators as the negative control exposure when evaluating the effect of an FDA label change targeting drug interaction risks between strong CYP2D6 inhibitor antidepressants and tamoxifen. In this example, the drug interaction risk exists only for tamoxifen-treated patients as aromatase inhibitors are metabolized outside of the CYP2D6 pathway. Changes in the use of strong inhibitor antidepressants (the primary outcome) among tamoxifen users could then be estimated while controlling for broader changes in antidepressant use over time among women using endocrine therapy. The authors observed greater decreases in strong CYP2D6 inhibitor antidepressant use among individuals prescribed tamoxifen, suggesting that the label change related to CYP2D6 risk resulted in selective prescribing of therapies and not general reductions in strong inhibitor antidepressants.

Finally, Rasmussen, Chong, and Alter (2007) evaluated the impact of varying levels of adherence to statins and beta-blockers on long-term mortality following AMI. The authors used adherence to calcium channel blockers as a control exposure as these treatments have no biologically plausible effect on post-AMI mortality. They compared three levels of adherence to each therapy and found patients with intermediate and lower adherence to statins were at increased risk of mortality as compared with patients with high adherence. Similar trends were observed for patients taking beta-blockers. They found no association between adherence to calcium channel blockers and mortality, suggesting that healthy user bias was minimized in their study and the protective effect of high statin adherence against mortality was unlikely to be confounded.


Control outcomes and exposures can improve the internal validity of nonexperimental studies because in certain situations they can detect confounding and selection bias via measurement of whether the treatment effect in the outcome of interest is confounded by unexpected factors (Brookhart et al. 2010a; Lipsitch, Tchetgen Tchetgen, and Cohen 2010). This direct assessment is an appealing feature of using control outcomes as measurement tools because they may be easier to implement in practice than statistical tools like instrumental variables. Of the 11 peer-reviewed studies we summarize that incorporated control outcomes or exposures to evaluate bias, six of them found a null association between treatment and the selected negative control outcomes or exposure. The null association where expected provided greater confidence in the validity of significant associations of interest (also where expected). However, five studies found unexpected associations for the selected control outcomes or exposures, which suggest that the primary association of interest may be biased.

Inclusion of control outcomes in nonrandomized studies is also appealing when there is interest in making causal claims because control outcomes impose a higher threshold for rejecting the null hypothesis than exists for non-randomized studies without them. In nonrandomized studies that include control outcomes or exposures, the null hypothesis of no association between treatment and primary outcome of interest can only be rejected if treatment is significant in the outcome equation of interest and one fails to reject the null in a control outcome or exposure. Control outcomes can also be used to assess the potential for recall bias in surveys, although we found few examples of this application. Given their utility, control outcomes have been recommended in guidance on the design of prospective nonrandomized studies (Berger et al. 2012, 2014).

Proposed Framework for Identifying Control Exposures or Outcomes

Given the theoretical utility of these measurement tools, one must identify a potentially valid control outcome or control exposure a priori. This requires subject-matter knowledge and consideration of the causal criteria in the specific analysis under consideration. Application of formal criteria to this task can aid in identification of valid control outcomes or control exposures, particularly when there is an interest in estimating the causal association between an exposure and primary outcome.

Some of the best known epidemiologic criteria for evaluating the cause and effect relationship were summarized by Hill (1965). The criteria outlined for identifying a causal relationship between an exposure and outcome can be readily adapted and used as a framework for identifying control exposures or outcomes. In particular, Hill's criteria for (biological) plausibility, temporality, specificity, consistency, and analogy seem most appropriate for this application (Table 3). For medical product evaluations, the criteria of temporality and biological plausibility may be the most familiar to researchers.

We found that these two criteria were most commonly employed by researchers for justification for the selection of controls, with the Jackson study providing a particularly thoughtful application of the temporality and biological plausibility criteria. If vaccination is expected to reduce influenza/pneumonia-related hospitalizations only during influenza season, then one might reasonably assume that vaccination should have no impact on these hospitalizations in the pre-influenza season due to biological implausibility and timing criteria. As a further robustness check for unobserved confounding, Jackson and colleagues also assessed a second set of outcomes that did not have biological plausibility for association with vaccination: hospitalizations for injury or trauma during influenza season. The only plausible way in which influenza/ pneumonia vaccination could affect hospitalizations for injury or trauma would be through unobserved confounding. However, Jackson and colleagues did not then re-analyze their primary outcomes and these control outcomes after accounting for unobserved confounding via covariate adjustment or statistical methods. Only through this additional step would it be possible to know whether the association between influenza/pneumonia vaccination and the control outcomes converged to the null after initial detection of unobserved confounding. Nonetheless, the analysis by Jackson was a creative application of the timing and biological plausibility criteria for identification of control outcomes.

Although not all criteria are appropriate for control identification in every setting, this framework may be a useful starting place for identifying controls. Importantly, proper control selection requires that the researcher understand the mechanism of potential confounding and that he or she selects a control that is subject to the same confounding mechanism but is not impacted by the treatment of interest. Suppose a researcher is studying the effect of statins on mortality following MI and is concerned that this treatment effect may be prone to healthy user bias. To test this possibility, the researcher should select a control outcome that has been associated with patient health behaviors but that is not influenced by statin use. Selecting a control outcome or exposure that is unrelated to patient health behaviors (for example, kidney stones or diverticulitis) (Dormuth et al. 2009) does not provide a robust test because these control outcomes are not likely impacted by the confounding mechanism (healthy user bias) that is potentially biasing the effect of statins on post-MI mortality. Thus, a null finding of statins on these control outcomes would provide the researcher with a false sense of confidence that the effect of statins on post-MI mortality was not subject to healthy user bias. Clinical judgment--particularly knowledge of the disease process--will often be critical for selection of control outcomes or control exposures that serve their stated purpose.

Recommendations for Reporting

We suggest that researchers using control outcomes and exposures should explicitly identify these measures in the methods sections of their manuscripts and include a rationale for their inclusion. Further, researchers should report results of all a priori selected controls, regardless of their consistency or lack thereof with the investigators' hypothesis.

Next, we recommend that terminology be standardized to improve the recognition of these measures. Terms that have been previously employed to describe these measures include "falsification endpoints," "non-equivalent controls/exposures," and "control outcomes/exposures." We recommend the use of "control outcomes" or "control exposures" with specification of the proposed direction of the effect (e.g., negative or positive). Further, we recommend the creation of a MeSH term for improving the identification of the use of these measurement tools in nonrandomized studies. This would benefit researchers since these measurement tools are likely to be used increasingly over time to improve the rigor and internal validity of comparative effectiveness research. Easier identification of studies using control outcomes and exposures will allow for further evaluation of the adoption of these tools and assessments of the quality of reporting of their use.

We identified a limited number of papers for inclusion in this review since these measures tend to be reported within the manuscript text and not in fields used for keyword identification within PubMed. As a result, we may have excluded other papers that faithfully applied control outcomes or control exposures to great effect. As these measurement tools become more widely used and easier to systematically identify, it will be useful to examine ways in which residual confounding has been addressed upon its identification in studies that find evidence of confounding via these tools.


Control outcomes and exposures are important tools for evaluating the internal validity of nonrandomized study findings. Routine use of controls will increase the rigor of studies by helping to identify studies where residual confounding is a concern (when control findings are not consistent with the researcher's hypothesis) or they may act as confirmation of study findings (when control findings are consistent with the researcher's hypothesis). Their use will create a higher threshold for rejecting the null hypothesis in an association of interest by requiring rejection of the null in the outcome equation of interest and failure to reject the null in a control outcome or exposure. Given the ongoing concern about clinical and policy inferences from nonrandomized studies, it seems reasonable to more widely employ these measurement tools.

This paper was developed for researchers conducting nonrandomized comparative effectiveness research who are unfamiliar with these nonequivalent outcomes. As these studies undergo continued scrutiny and investigators need to increase the validity of their nonrandomized studies, the use of tools to increase validity will be important.

DOI: 10.1111/1475-6773.12279


Joint Acknowledgment/Disclosure Statement: This work was supported by the Office of Research and Development, Health Services Research and Development Service, Department of Veterans Affairs. Dr. Dusetzina is supported by the NIH Building Interdisciplinary Research Careers in Women's Health (BIRCWH) K12 Program and the North Carolina Translational and Clinical Sciences Institute (UL1TR001111). Dr. Brookhart (MAB) receives investigator-initiated research funding from the National Institutes of Health (R01 AG042845, R21 HD080214, R01 AG023178) and through contracts with the Agency for Healthcare Research and Quality's DEcIDE program and the Patient Centered Outcomes Research Institute. Dr. Maciejewski was supported by a Research Career Scientist award from the Department of Veterans Affairs (RCS 10-391) and received investigator-initiated research funding from the Department of Veterans Affairs, the National Institutes of Health (R01 DK097165), the Agency for Healthcare Research and Quality (R01 HS023085, R01 HS023099), and the Robert Wood Johnson Health Care Financing and Organization Initiative (70922). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the University of North Carolina at Chapel Hill, the Department of Veteran Affairs, or Duke University. MAB has received research support from Amgen for unrelated projects and has served as an unpaid scientific advisor for Amgen, Merck, Pfizer, and GSK. He receives consulting fees from World Health Information Science Consultants and RxAnte, Inc.

Disclosure: None.

Disclaimer. None.


Aronow, H. D., E. J. Topol, M. T. Roe, P. L. Houghtaling, K. E. Wolski, A. M. Lincoff, R. A. Harrington, R. M. Califf, E. M. Ohman, N. S. Kleiman, M. Keltai, R. G. Wilcox, A. Vahanian, P. W. Armstrong, and M. S. Lauer. 2001. "Effect of Lipid-Lowering Therapy on Early Mortality after Acute Coronary Syndromes: An Observational Study." Lancet 357 (9262): 1063-8.

Basker, E. 2005. 'Job Creation or Destruction? Labor Market Effects of Wal-Mart Expansion." Review of Economics and Statistics 87 (1): 174-83.

Benson, K., and A. J. Hartz. 2000. "A Comparison of Observational Studies and Randomized, Controlled Trials." New England Journal of Medicine 342 (25): 1878-86.

Berger, M. L., N. Dreyer, F. Anderson, A. Towse, A. Sedrakyan, and S. L. Normand. 2012. "Prospective Observational Studies to Assess Comparative Effectiveness: The ISPOR Good Research Practices Task Force Report." Value Health 15 (2): 217-30.

Berger, M. L., B. C. Martin, D. Husereau, K. Worley, J. D. Allen, W. Yang, N. C. Quon, C. D. Mullins, K. H. Kahler, and W. Crown. 2014. "A Questionnaire to Assess the Relevance and Credibility of Observational Studies to Inform Health Care Decision Making: An ISPOR-AMCP-NPC Good Practice Task Force Report." Value Health 17 (2): 143-56.

Brookhart, M. A., A. R. Patrick, C. Dormuth, J. Avorn, W. Shrank, S. M. Cadarette, and D. H. Solomon. 2007. "Adherence to Lipid-Lowering Therapy and the Use of Preventive Health Services: An Investigation of the Healthy User Effect." American Journal of Epidemiology 166 (3): 348-54.

Brookhart, M. A., A. R. Patrick, W. H. Shrank, and C. R. Dormuth. 2010a. "Validating Studies of Adherence through the Use of Control Outcomes and Exposures." American Journal of Hypertension 23 (2): 110.

Brookhart, M. A., T. Sturmer, R.J. Glynn, J. Rassen, and S. Schneeweiss. 2010b. "Confounding Control in Healthcare Database Research: Challenges and Potential Approaches." Medical Care 48 (6 Suppl): S114-20.

Concato, J., N. Shah, and R. I. Horwitz. 2000. "Randomized, Controlled Trials, Observational Studies, and the Hierarchy of Research Designs." New England Journal of Medicine 342 (25): 1887-92.

Dormuth, C. R., A. R. Patrick, W. H. Shrank, J. M. Wright, R.J. Glynn, J. Sutherland, and M. A. Brookhart. 2009. "Statin Adherence and Risk of Accidents: A Cautionary Tale." Circulation 119 (15): 2051-7.

Dusetzina, S. B., G. C. Alexander, R. A. Freedman, H. A. Huskamp, and N. L. Keating. 2013. "Trends in Co-Prescribing of Antidepressants and Tamoxifen among Women with Breast Cancer, 2004-2010." Breast Cancer Research and Treatment 137 (1): 285-96.

Furlan, A. D., G. Tomlinson, A. A. Jadad, and C. Bombardier. 2008. "Methodological Quality and Homogeneity Influenced Agreement between Randomized Trials and Nonrandomized Studies of the Same Intervention for Back Pain." Journal of Clinical Epidemiology 61 (3): 209-31.

Hernan, M. A., S. Hernandez-Diaz, M. M. Werler, and A. A. Mitchell. 2002. "Causal Knowledge as a Prerequisite for Confounding Evaluation: An Application to Birth Defects Epidemiology." American Journal of Epidemiology 155 (2): 176-84.

Hernan, M. A., A. Alonso, R. Logan, F. Grodstein, K. B. Michels, W. C. Willett, J. E. Manson, and J. M. Robins. 2008. "Observational Studies Analyzed Like Randomized Experiments: An Application to Postmenopausal Hormone Therapy and Coronary Heart Disease." Epidemiology 19 (6): 766-79.

Hill, A. B. 1965. "The Environment and Disease: Association or Causation?" Proceedings of the Royal Society of Medicine 58: 295-300.

Holmlund, H., S. McNally, and M. Viarengo. 2010. "Does Money Matter for Schools?" Economics of Education Review 29 (6): 1154-64.

Ioannidis, J. P. 2005. "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research." Journal of the American Medical Association 294 (2): 218-28.

Ioannidis, J. P., A. B. Haidich, M. Pappa, N. Pantazis, S. I. Kokori, M. G. Tektonidou, D. G. Contopoulos-Ioannidis, and J. Lau. 2001. "Comparison of Evidence of Treatment Effects in Randomized and Nonrandomized Studies." Journal of the American Medical Association 286 (7): 821-30.

Jackson, L. A., M. L. Jackson, J. C. Nelson, K. M. Neuzil, and N. S. Weiss. 2006. "Evidence of Bias in Estimates of Influenza Vaccine Effectiveness in Seniors." International Journal of Epidemiology 35 (2): 337-44.

Jena, A. B., E. Sun, and D. P. Goldman. 2013. "Confounding in the Association of Proton Pump Inhibitor Use with Risk of Community-Acquired Pneumonia. "Journal of General Internal Medicine 28 (2): 223-30.

Lipsitch, M., E. Tchetgen Tchetgen, and T. Cohen. 2010. "Negative Controls: A Tool for Detecting Confounding and Bias in Observational Studies." Epidemiology 21 (3): 383-8.

Maciejewski, M. L.,J. F. Farley, J. Parker, and D. Wansink. 2010. "Copayment Reductions Generate Greater Medication Adherence in Targeted Patients." Health Affairs (Millwood) 29 (11): 2002-8.

Manson, J. E., J. Hsia, K. C. Johnson, J. E. Rossouw, A. R. Assaf, N. L. Lasser, M. Trevisan, H. R. Black, S. R. Heckbert, R. Detrano, O. L. Strickland, N. D. Wong, J. R. Crouse, E. Stein, and M. Cushman. 2003. "Estrogen Plus Progestin and the Risk of Coronary Heart Disease." New England Journal of Medicine 349 (6): 523-34.

Mauri, L., T. S. Silbaugh, R. E. Wolf, K. Zelevinsky, A. Lovett, Z. Zhou, F. S. Resnic, and S. L. Normand. 2008. "Long-Term Clinical Outcomes after Drug-Eluting and Bare-Metal Stenting in Massachusetts." Circulation 118 (18): 1817-27.

McClellan, M., B. J. McNeil, and J. P. Newhouse. 1994. "Does More Intensive Treatment of Acute Myocardial Infarction in the Elderly Reduce Mortality? Analysis Using Instrumental Variables." Journal of the American Medical Association 272 (11): 859-66.

Patrick, A. R., W. H. Shrank, R. J. Glynn, D. H. Solomon, C. R. Dormuth, J. Avorn, S. M. Cadarette, H. Mogun, and M. A. Brookhart. 2011. "The Association between Statin Use and Outcomes Potentially Attributable to an Unhealthy Lifestyle in Older Adults." Value Health 14 (4): 513-20.

Prasad, V., and A. B. Jena. 2013. "Prespecified Falsification end Points: Can They Validate True Observational Associations?" Journal of the American Medical Association 309 (3): 241-2.

Rasmussen, J. N., A. Chong, and D. A. Alter. 2007. "Relationship between Adherence to Evidence-Based Pharmacotherapy and Long-Term Mortality after Acute Myocardial Infarction." Journal of the American Medical Association 297 (2): 177-86.

Ray, W. A. 2003. "Evaluating Medication Effects Outside of Clinical Trials: New-User Designs." American Journal of Epidemiology 158 (9): 915-20.

Redelmeier, D., D. Scales, and A. Kopp. 2005. "Beta Blockers for Elective Surgery in Elderly Patients: Population Based, Retrospective Cohort Study." British Medical Journal 331 (7522): 932.

Robins, J. M. 2001. "Data, Design, and Background Knowledge in Etiologic Inference." Epidemiology 12 (3): 313-20.

Rothstein, J. 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement." Quarterly Journal of Economics 125 (1): 175-214.

Schneeweiss, S., A. R. Patrick, T. Sturmer, M. A. Brookhart J. Avorn, M. Maclure, K.J. Rothman, and R.J. Glynn. 2007. "Increasing Levels of Restriction in Pharmacoepidemiologic Database Studies of Elderly and Comparison with Randomized Trial Results." Medical Care 45 (10 Supl 2): 131-42.

Stenestrand, U., and L. Wallentin. 2001. "Early Statin Treatment Following Acute Myocardial Infarction and 1-Year Survival." Journal of the American Medical Association 285 (4): 430-6.

Zaadstra, B. M., A. M. Chorus, S. van Buuren, H. Kalsbeek, andj. M. van Noort. 2008. "Selective Association of Multiple Sclerosis with Infectious Mononucleosis." Multiple Sclerosis 14 (3): 307-13.


Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.

Address correspondence to Matthew L. Maciejewski, Ph.D., Center for Health Services Research in Primary Care, Department of Veterans Affairs; Division of General Internal Medicine, Department of Medicine, Duke University Medical Center, Durham, NC 27705; e-mail: mlm34@duke. edu. Stacie B. Dusetzina, Ph.D., is with the Division of Pharmaceutical Outcomes and Policy, UNC Eshelman School of Pharmacy; Department of Health Policy and Management, Gillings School of Global Public Health; Lineberger Comprehensive Cancer Center; Cecil G. Sheps Center for Health Services Research, all at University of North Carolina at Chapel Hill, Chapel Hill, NC. M. Alan Brookhart, Ph.D., is with the Department of Epidemiology, Gillings School of Global Public Health, University of North Carolina at Chapel Hill; Cecil G. Sheps Center for Health Services Research, University of North Carolina at Chapel Hill, Chapel Hill, NC.
Table 1: Summary of Published Studies Using Nonequivalent Outcomes

                    Nonequivalent      Nonequivalent
Citation               Outcome           Exposure

Redelmeier,      Postsurgical               N/A
  BMJ. 2005        noncardiac
                   failure, renal
Mauri,           Mortality within           N/A
  Circulation.     2 days of
  2008             stenting
Rasmussen,       Cancer-related        Calcium
  JAMA.            hospital              channel
  2007             admissions            blockers
Maciejewski,     ARBs, cholesterol          N/A
  Health           absorption
  Affairs.         inhibitors
McClellan,       Mortality within           N/A
  JAMA.            1 day of acute
  1994             myocardial
Jackson, Int J   Injury or trauma      Flu shot in
  Epidemiol.       hospitalization,      pre-flu
  2006             also possibly         season
                   admits for IHD,
                   CHF and CVD
                   (but not clearly
                   stated as such)
Brookhart,       Bone mineral              N/A
  AmJ              density test,
  Epidemiol.       screening
  2007             mammography,
                   FSA, FOBT, flu
                   shot, pneumo
Patrick,         Preventive                 N/A
  Value            services (bone
  Health.          min density,
  2011             PSA,
                   shot), clinical
                   (asthma, burns,
                   GI bleeds, skin
Jena, J Gen        Dx of                      N/A
  Intent Med.      ostecarthritis,
  2013             chest pain, UTI,
                   DVT, skin
                   infection, and

                   Primary Study        Exposure or
Citation             Endpoint            Treatment

Redelmeier,      Death or AMI        Atenolol versus
  BMJ. 2005                            metoprolol

Mauri,           Mortality within    Drug-eluding or
  Circulation.     2 years of          bare metal
  2008             stenting            stents
Rasmussen,       Long-term           Adherence to
  JAMA.            mortality post-     statins and
  2007             MI                  beta-blockers
Maciejewski,     Medication          VBID (lower
  Health           adherence to        copays)
  Affairs.         ACEIs, BBs,
  2010             CCBs, statins,
McClellan,       Mortality within    Cardiac
  JAMA.            4 years of          catheterization
  1994             AMI
Jackson, Int J   Mortality and       Flu shot
  Epidemiol.       flu/pneumonia
  2006             hospitalization
                   in flu season
Brookhart,              N/A
Patrick,         MI, death,          Statin
  Value            nursing home        initiation
  Health.          admit
Jenaj Gen        Community-          PPI fill
  Intent Med.      acquired
  2013             pneumonia

                   Study Effect         Nonequivalent
Citation              Measure           Outcome Effect

Redelmeier,      Relative risk       No differences in
  BMJ. 2005        from logistic       postsurgical
                   regression          noncardiac
                                       by beta-blocker
                                       (atenolol vs.
Mauri,           Percentage          Small absolute
  Circulation.     difference in       difference in 2-
  2008             mortality rates     day mortality
                                       (0.45% vs.
                                       0.68%, p = .10)
Rasmussen,       Hazard ratio        No increase in
  JAMA.            from Cox            cancer-related
  2007             model               admissions by
                                       statin or beta-
Maciejewski,     OR and              Null effect as
  Health           predicted           expected
  Affairs.         adherence
  2010             showed that
                   avoided a 2
                   3% drop in
                   adherence at
                   1 year
McClellan,       Percentage          Significant
  JAMA.            difference in       difference at
  1994             mortality rates     1 day, which
Jackson, Int J   Relative risk       Flu shot
  Epidemiol.       from Cox            significantly
  2006             model               protective
                   showed that         versus injury/
                   flu shot was        trauma admit
                   protective          in pre-flu
                   versus all-         season and in
                   cause               flu season
                   mortality and
                   admits in flu
Brookhart,       Hazard ratio
  AmJ              from Cox
  Epidemiol.       model
Patrick,         Rate ratio from     Significant
  Value            Cox model           difference in
  Health.          for                 preventive
  2011             preventive,         (mammogram,
                   various             flu shot) and
                   methods for         clinical
                   clinical            outcomes
                   outcomes            (asthma, burns,
                                       falls, fractures,
                                       motor vehicle
Jenaj Gen        Proportion in       PPI associated
  Intent Med.      quarter with        with higher
  2013             Dx of CAP           rates of all
                   showed that         unexpected Dx
                   PPI users had
                   rates than

                   Nonequivalent        Proposed
                     Exposure          Source of          Causal
Citation              Effect              Bias           Criteria

Redelmeier,             N/A          Unobserved       Unstated
  BMJ. 2005                            confounding
Mauri,                  N/A          Unobserved       Unstated
  Circulation.                         confounding
Rasmussen,       Adherence to        Healthy user     No
  JAMA.            calcium             bias             biological
  2007             channel                              plausibility
                   blockers was
                   not associated
                   endpoints, as
Maciejewski,            N/A          Unobserved       Unstated
  Health                               confounding
McClellan,              N/A          Unobserved       Access to
  JAMA.                                confounding      cath-ing
  1994                                 due to           hospitals
                                       access to
Jackson, Int J   Flu shot            Confounding      Unstated
  Epidemiol.       significantly       by health
  2006             protective          status
                   against all
                   cause mortality
                   and flu/
                   in pre-flu
Brookhart,              N/A          Healthy user     Unstated
  AmJ                                  bias
Patrick,                N/A          Healthy user     Unstated
  Value                                bias
Jenaj Gen               N/A          Confounding      No
  Intent Med.                          by               biological
  2013                                 indication,      plausibility

Table 2: Summary of Published Studies Using Nonequivalent Exposures

                Nonequivalent    Nonequivalent      Primary Study
Citation           Outcome          Exposure           Endpoint

Zaadstra,            N/A         Broken arm,      Multiple sclerosis
  Mult Scler.                      concussion,
  2008                             tonsilectomy
Dusetzina,           N/A         Aromatase        Co-prescribing of
  Breast                           inhibitors       antidepressants
  Cancer                                            and endocrine
  Res Treat.                                        therapy
Rasmussen,      Cancer-related   Calcium          Long-term
  JAMA.           hospital         channel          mortality post-
  2007            admissions       blockers         MI

                  Exposure        Primary
                     or        Study Effect      Nonequivalent
Citation         Treatment        Measure        Outcome Effect

Zaadstra,       Childhood      Odds ratios            N/A
  Mult Scler.     infections     from
  2008            (rubella,      logistic
                  chicken        regression
Dusetzina,      Tamoxifen      Risk ratios            N/A
  Breast                         from
  Cancer                         binomial
  Res Treat.                     regression
  2013                           using a
Rasmussen,      Adherence      Hazard ratio    No increase in
  JAMA.           to statins     from Cox        cancer-related
  2007            and beta-      model           admissions by
                  blockers                       statin or beta-

                  Nonequivalent          Proposed           Causal
Citation         Exposure Effect      Source of Bias       Criteria

Zaadstra,       Patients with MS     Recall bias        Unstated
  Mult Scler.     had higher rates
  2008            of concussion

Dusetzina,      Greater decline      Unobserved         No
  Breast          in strong            confounding,       biological
  Cancer          inhibitor            temporal           plausibility
  Res Treat.      antidepressant       changes in
  2013            use among            antidepressant
                  tamoxifen users      use
                  than aromatase
                  inhibitor users
Rasmussen,      Adherence to         Healthy user       No
  JAMA.           calcium              bias               biological
  2007            channel                                 plausibility
                  blockers was
                  not associated
                  with mortality
                  endpoints, as

Table 3: Bradford-Hill Criteria to Consider When Identifying Control
Outcomes or Exposures

Criteria                             Description

Strength        The larger the association, the more likely that the
                  association is causal. However, a small association
                  does not mean that there is not a causal effect as
                  expected treatment effects in medicine are often
Consistency     Findings have been replicated by other researchers
                  and/or in different samples.
Specificity     The more specific an association between a factor
                  and an effect is, the greater the probability of a
                  causal relationship. Causation is likely if the
                  association is identified under specific
                  circumstances and that there is no other likely
Temporality     Cause precedes effect; if there is an expected delay
                  between the cause and expected effect, then the
                  effect must occur after that delay.
Biological      For exposures that follow a dose-response curve,
  Gradient        greater exposure should generally lead to greater
                  incidence of the effect. In some cases, the mere
                  presence of the factor can trigger the effect. In
                  other cases, an inverse proportion is observed:
                  greater exposure leads to lower incidence.
Plausibility    A plausible biological mechanism between cause and
                  effect is helpful.
Coherence       Coherence between epidemiological and laboratory
                  findings increases the likelihood of an effect.
                  Results need to be interpreted in light of existing
                  data and known facts of the natural history and
                  disease biology.
Experiment      Reducing exposure to the risk factor reduces the
                  likelihood of the outcome.
Analogy         Exposures with similar mechanisms of action may
                  result in similar outcomes.
COPYRIGHT 2015 Health Research and Educational Trust
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2015 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:METHODS ARTICLE
Author:Dusetzina, Stacie B.; Brookhart, M. Alan; Maciejewski, Matthew L.
Publication:Health Services Research
Date:Oct 1, 2015
Previous Article:Modeling area-level health rankings.
Next Article:Quality of disease management and risk of mortality in English primary care practices.

Terms of use | Privacy policy | Copyright © 2020 Farlex, Inc. | Feedback | For webmasters