Uncover bias in clinical trials: drug research should be objective, but when it's funded by manufacturers, study results are often predictably rosy. Look for biases that taint clinical trials.

When physicians make treatment decisions and prescribe medications for patients, they rely on information from clinical trials. The medical community universally believes that data on the clinical effectiveness (1) and toxicity of new drugs, compared with available alternatives, must be available to prescribing physicians--and it must be accurate.

[ILLUSTRATION OMITTED]

There is no shortage of medical literature evaluating safety and efficacy claims based on clinical trials. Not as well known is the extensive collection of journal articles and other publications written by members of the medical community specifically attacking clinical trials that are funded by the pharmaceutical industry.

Well-respected physicians have noted that science often is not objective. (2) One medical journal article states:

   Emotional investment in particular ideas and personal interest in
   academic success may lead investigators to overemphasize the
   importance of their findings and the quality of their work. Even
   more serious conflicts arise when for-profit organizations,
   including pharmaceutical companies, provide funds for research and
   consulting, conduct data management and analyses, and write reports
   on behalf of the investigators. (3)

A clinical trial typically is designed to evaluate safety or efficacy, but not both. "Efficacy" is not the same as "clinical effectiveness." Efficacy simply refers to any arbitrarily chosen effect, which may or may not be clinically relevant. The World Health Organization defines clinical effectiveness as "the likelihood and extent of desired clinically relevant effects in patients with the specified indication." (4)

Once an article is written about a clinical trial, the results can be published or reported in many ways: peer-reviewed journals, symposium proceedings that are adjunct to journals, and "controlled circulation" publications (so-called throwaway medical journals that pharmaceutical companies give to doctors and libraries for free). Study results can also be presented at medical educational seminars.

A clinical trial published in a medical journal usually contains several parts, each with a specific function. Think of the mnemonic IMRC: introduction, methods, results, and conclusion. (5) Understanding how these studies are organized can help you locate the type of information you need to expose a study's flaws.

The introduction contains a basic clinical background of the study, a review of previous studies, and a primary and secondary hypothesis or scientific theory that the study either proved or disproved.

Methods define the study's patient population, the criteria used to select participants, the study design (such as randomization methods, blinding, and controls), and treatment. The methods section defines the primary and secondary outcomes, subgroup analysis, and surrogate markers. It also provides the statistical reference (type of statistical analysis), confidence intervals, outcome comparisons, and power. [For an explanation of some of these terms, see "Epidemiology in Plain English" on page 34.]

The results section contains a summary of the data collected. It might contain tables, figures, or a written description of the results. It also contains any differences in outcomes based on intention to treat, any treatment effects, and an evaluation of confounding factors and interactions (both of which may introduce bias). Confounding factors can introduce bias when a study trait is associated with a particular outcome but is inappropriately described as causing the outcome.

The conclusion (or discussion) interprets the results, including precision and bias. It should contain the study's limitations and strengths. It should also make suggestions for future research and propose specific hypotheses based on the results.

Flawed methodology

It is well known that a poorly designed study can artificially inflate benefits of a test drug. A double-blind, randomized, placebo-controlled clinical trial (RCT) is generally considered the gold standard of valid methodology in the medical community, although that idea is sometimes challenged. (6) An RCT allocates medications to be tested on subjects at random, so that neither the researcher nor the subject knows who receives a particular test medication.

The basic idea behind random allocation is that it makes different treatment groups within a trial statistically equivalent and eliminates "cognitive bias"--a distortion in the way a person perceives reality. But RCTs are not immune to bias. Medical journal authors often cite key problems with the validity of randomly controlled double-blind studies that are funded by the pharmaceutical industry.

Inadequate treatment concealment. When patient selection or treatment allocation is not random or adequately concealed, a study can result in false favorable results for the drug company. Inadequate treatment concealment--when the subject and investigators can guess which drug is being given to study subjects--is the primary reason cited in medical journals for bias in industry-funded RCTs.

A classic example is a study in which random assignments were kept in a sealed but semitransparent envelope, and investigators were able to see into the envelope and learn the type of medication each trial subject received. (7)

Another example: When a drug is administered in a specific form for example, a liquid and only one group receives medication in that form or intravenously, clinical subjects and investigators can determine which drug each group receives. This exposes the clinical trial to distorted reporting bias.

If investigators who enroll patients can direct the type of medication given to each subject, the study is inadequately randomized. For example, in one study, researchers who knew which group of patients would receive which treatment enrolled them based on their prognosis. (8) Patients who would have been assigned to an ineffective treatment were deemed "inappropriate" and rejected from the study, and some were directed to the "appropriate" category and enrolled.

Such flawed methodology significantly distorts the results of clinical trials. For example, researchers noted that inadequate or unclear concealment of treatment allocation was associated with an exaggerated treatment effect in four studies. (9) Conversely, odds ratios for adverse events from trials with inadequate concealment were on average 30 percent lower (more beneficial to the industry) than those from trials with adequate methodology. (10)

Failures in randomization are probably more frequent than suspected. Thirty-two percent of reports published in some specialist journals (11) and 41 percent of those published in some premier general journals (12) specified an inadequate method for generating random assignments. In both groups, only about a quarter of the trials reported adequate allocation concealment.

Always check to see whether the study you are evaluating has reported appropriate concealment allocation. The failure to prevent foreknowledge of treatment allocation has been found to be associated with exaggerated treatment effects of, on average, 30 percent to 40 percent of studies reported in medical journals. (13)

Attrition and compliance. Not all patients who begin a clinical trial participate for its duration. A clinical trial patient has the right to withdraw from a study at any time, for any reason. Sometimes the patient chooses to quit the trial, sometimes the patient's physician directs him or her to withdraw for safety reasons, and sometimes patients are forced to withdraw due to serious adverse events--including death. Even when a patient continues to "participate" in a clinical trial, he or she might stop taking the study medication. The percentage of participants who fail to complete a trial is called the "attrition" or "dropout" rate. (14)

Epidemiologist Austin Bradford Hill explained that excluding patients from clinical studies after "admission to the treated or control group" might affect the validity of clinical trials. (15) Hill described patients admitted in a clinical trial as patients intended to be treated with the study medication (or control) and patients who completed a clinical trial as those who actually received the treatment. He wrote that "unless the losses are very few and therefore unimportant, we may inevitably have to keep such patients in the comparison and thus measure the 'intention to treat' in a given way, rather than the actual treatment." (16)

In 1970, a group of researchers concluded that a particular method evaluated to treat arterial occlusion reduced the risk of recurrent unfavorable outcomes by 26 percent when compared with conventional treatments. (17) Their analysis included this statement: "Excluding the 5 patients who died and the 10 who had permanent postoperative deficits, there remained 79 patients available for follow-up and at risk of subsequent persistent stroke and death." (18) Almost 10 years after the original study, new authors incorporated the 15 patients excluded in the original analysis and came to a different conclusion. They identified the reduction in risk to be only 17 percent. (19)

Researchers evaluating the impact of attrition bias concluded that it generally leads to exaggerated treatment and underreported adverse effects. (20) Always evaluate the data in a clinical trial to determine whether patients were excluded after enrollment. Look for evaluations that include the entire population of enrolled individuals, using the term "intention to treat." Studies that evaluated only "treated" individuals probably excluded patients improperly, making the studies' conclusions suspect.

A patient may stop taking a medication in a clinical trial even though the investigator believes the patient is complying. Reports of patients "checking" medication are common. Trials should always include mechanisms to verify patient compliance. The most accurate method is a blood test to measure the presence of the chemical in the blood. The least accurate--but most used--method is verification by phone interviews and pill count.

Sometimes the study does not include verification of compliance; failure to do so is a significant flaw that you should expose. Patient noncompliance affects the validity of the study and can result in underreporting adverse events or overreporting benefits (if the patient is checking the comparator drug).

Misleading claims

A well-conducted study is not necessarily free from misrepresentations. In fact, many authors have expressed the belief that industry-funded studies are not necessarily more poorly designed than others (21) and that a well-designed study is no guarantee that its results are not misleading.

There is little disagreement in the medical community that there exists a systemwide bias that favors products made by a company funding the research. Often-cited results estimate that an industry-funded clinical trial is four times more likely to find results favoring its product. (22)

A few key reasons have emerged to explain this bias. Because they are simple and generally accepted in the medical literature, they are useful tools in your arsenal to attack the conclusions of a seemingly well-conducted, industry-funded trial. The problems most commonly reported in the medical literature that affect the validity of conclusions reported in industry-funded clinical trials include the following:

Inappropriate comparator drugs. In any clinical study, a drug--either by itself or in combination with another drug--can be compared to a placebo, another drug, or other groups of drugs. The choice of a comparator can determine the outcome of a trial and can be used to establish bias.

For example, in a company-sponsored comparison trial involving a non-steroidal anti-inflammatory drug, the dose of the company's drug was higher than that of the comparison drug. For the class of drug studied, the higher the dose, the more effective the drug. The researchers selected an inappropriate comparator drug dose, which resulted in positive findings reported on the company's drug. The medical community challenged the validity of this trial and concluded that the selection of the comparator drug's dosage was a biased design. (23)

An often-cited example describes a meta-analysis that compares a Pfizer product, fluconazole, to a competitor, amphotericin B. (24) Pfizer paid for 92 percent of the trials. In 79 percent of the trials, both drugs were taken orally, even though amphotericin is poorly absorbed and typically given intravenously. In this case, the comparator drug was carefully selected to fail. This selection process practically guaranteed that Pfizer's drug would produce better results. (25)

Composite end points and surrogate response. Investigators often use composite end points--a group of different but probably related outcomes in a particular disease--to enhance the statistical efficiency of clinical trials. Composite end points are most often used with heart patients. For example, a composite end point in a clinical trial evaluating a new heart medication would include nonfatal myocardial infarction, death, or angina. Therefore, if a clinical trial reports favorable results on "important cardiovascular end points," it would be difficult to determine whether the outcome reflects mostly on the drug's effectiveness in treating angina, preventing death, or preventing a nonfatal myocardial infarction.

A trial of irbesartan versus placebo for diabetic nephropathy demonstrated a problematic primary outcome. The composite end point included death, end-stage renal disease, and doubling of serum creatinine concentration. (26) It was difficult to determine whether the drug was effective in treating damaged kidneys or preventing death.

A surrogate marker is an outcome used as an alternative to a desired or ideal clinical response. For example, suppression of arrhythmia is used as a surrogate marker for prevention of sudden death, and T4 cell counts are used as a measure of an AIDS treatment's efficacy. Surrogate markers are used frequently in clinical trials because they are easier to measure and can be detected earlier than some clinical responses, so surrogate marker trials are cheaper and shorter. Surrogate markers should be viewed suspiciously when used to confirm that a drug is effective.

The FDA must approve a surrogate marker. Examples of approved surrogate markers include lower cholesterol level without evidence of survival benefits in heart patients; lower blood glucose levels without evidence of diabetic complications or survival benefits; and lower blood pressure without evidence of benefit for strokes, myocardial infarction, congestive heart failure, or survival. These FDA-approved responses don't guarantee that a drug is effective.

Concerns about surrogates arise because these results may not be caused by the drug being tested; rather, they may be a coincidence resulting from a third factor. For example, diet might affect cholesterol levels in a clinical trial subject. Effects on surrogates may correlate with one clinical end point but not others. Because surrogates do not reliably predict clinical outcomes, you can point out that a study relied too much on surrogates.

Subgroup analysis. Clinicians often wonder if a subgroup of patients will achieve greater benefits when exposed to a particular treatment. Because chance can lead to apparent but spurious differences in effects between subgroups, you should be suspicious of studies using subgroup analysis.

For example, a study compared two treatments for preventing cardiac events and concluded that

"initiation of antihypertensive treatment involving ACE inhibitors in older subjects, particularly men, appears to lead to better outcomes than treatment with diuretic agents." The possibility that effect differed by sex was not one of a small number of prior hypotheses, and the size of the difference in effect was small (relative risk reductions of 17% (95 % confidence interval 3 to 29%) in men and 0% (-20% to 17%) in women). (27)

The difference between the two medications was significant in men but not women. Investigators might use this fact to argue for differences between the drug's effect on men and women. The real issue is whether chance is a sufficient explanation for the differences in effect, and it was in this case. Generally, the inference that a drug affects men differently "satisfies only one of seven criteria for a valid subgroup analysis." (28)

Analysis of a trial by subgroup results in a separate statistical test for each subgroup. As a result, the probability of false-positive conclusions increases. The greater the number of subgroups analyzed separately, the larger the probability of making false-positive conclusions. Guidelines for determining whether results from subgroup analysis are valid include: Is the magnitude of the effect large? Is the effect consistent across studies? Is there indirect evidence that supports the hypothesized interaction? (29)

Control of the data. Usually, by contract, the data collected in a clinical trial belongs to the sponsoring company, which decides who sees how much of it. Investigators in trials that are conducted in multiple sites may not see all the data, and this can lead to bias.

For example, a group of researchers investigated a new drug for treatment of gastric ulcers, comparing it with ranitidine. At one trial site, they found that the new drug was inferior to ranitidine; at another, that the new drug was superior; and at most sites, that healing rates between the two drugs were similar. The site with results favoring the manufacturer's drug submitted them for publication separately, which made the sponsor's drug look effective. Had all the results of the different sites been merged, the sponsor's drug would not have looked as effective, The company got the results it wanted by controlling how the data was disseminated. (30)

Publication bias, redundant publications, and promotional publications. In several high-profile cases, manufacturers have attempted to prevent studies that are unfavorable to their products from being published. (31) And some authors have shown that research funded by industry appears more often in symposia, which are known to lack peer review and to favor the sponsor's product. (32)

Although the methods of industry-funded trials may not be any different from those in studies funded by other sources, the absence of peer review may result in an overly favorable interpretation of the trial's results. One study noted that when researchers presented their findings at symposia proceedings, their claims that the sponsor's product was superior were often unsupported by the data. (33)

Several investigators have also reported multiple publications of the same trial with different authors--and various versions of the same trial. (34) In 1996, for example, investigators were attempting to perform a meta-analysis of the effects of the antipsychotic agent risperidone. They identified 20 articles and several unpublished reports described as randomized, double-blind trials, but a careful analysis of the reports' substance revealed there were only seven small trials and two large trials, one of which had been reported "in part, transparently, and not so transparently in six different publications ... the authorship was different for each." (35)

Drummond Rennie, a JAMA deputy editor, examined several reported instances of multiple publications and concluded that the "evidence shows that covert reporting of the same data in clinical trials artificially skews the balance of opinion in favor of the new drug." (36) The Uniform Requirements for Manuscripts Submitted to Biomedical Journals prohibits redundant publications with different authors. (37)

Promotional publications such as controlled circulation journals (38) and symposia proceedings (39) are characterized by poorly described studies, selective presentation of data, conclusions that are unsupported by results, misleading statements, and inadequate reviews. Industry sponsorship of publication allows the manufacturer to avoid the peer-review process. Some researchers have found an association between single pharmaceutical company sponsorship and a lack of peer review in symposia proceedings and throwaway journals. (40)

Contract research organizations (CROs). Increasingly, pharmaceutical companies have used the services of contract research organizations--private, nonacademic research groups--to perform their clinical drug testing. In 2000, 60 percent of the industry's research grants went to CROs. (41) In 2001, 13 leading medical journals warned in simultaneous editorials that CROs fail to provide sufficient oversight of clinical trials and that sponsoring companies may bury unfavorable results. (42)

To expose intentionally designed biases that are pervasive in industry-funded clinical studies and publication practices, lawyers representing people injured by prescription drugs should begin at the same place where physicians begin. The medical community has given us a blueprint to follow in assessing a study's reliability. We must learn, understand, and use the medical community's methodologies to attack biases that taint clinical trials. (43)

We need to establish that these biases lead to poor prescribing decisions by physicians--which in turn may severely harm the people who are treated with these drugs.

Notes

(1.) Lisa A. Bero & Drummond Rennie, Influences on the Quality of Published Drug Studies, 12 Intl. J. Tech. Assessment Health Care 209 (1996).

(2.) See e.g. Richard Horton, The Rhetoric of Research, 310 Brit. Med. J. 985 (1995).

(3.) Victor M. Montori et al., Users' Guide to Detecting Misleading Claims in Clinical Research Reports, 329 Brit. Med. J. 1093 (2004).

(4.) World Health Org., Clinical Pharmacological Evaluation in Drug Control, 1993 EUR/ ICP/DSE 173 (Dec. 10-13, 1992).

(5.) Gary H. Lyman & Nicole M. Kuderer, A Primer for Evaluating Clinical Trials, 4 Cancer Control J. 413, 418 tbl. 3 (1997).

(6.) See e.g. Ted J. Kaptchuk, The Double-Blind, Randomized, Placebo-Controlled Trial: Gold Standard or Golden Calf?, 54 J. Clinical Epidemiology 541 (2001); Kenneth E Schulz, Randomised Trials, Human Nature, and Reporting Guidelines, 348 Lancet 596 (1996); Stephen D. Simon, Is the Randomized Clinical Trial the Gold Standard of Research?, 22 J. Andrology 938 (2001).

(7.) See Kenneth E Schulz & David A. Grimes, Allocation Concealment in Randomised Trials: Defending Against Deciphering, 359 Lancet 614 (2002).

(8.) Marc J. Keirse, Amniotomy or Oxytocin for Induction of Labor: Re-Analysis of a Randomized Controlled Trial, 67 Acta Obstetricia et Gynecologica Scandinavica 731 (1988).

(9.) See Peter Juini et al., Assessing the Quality of Randomised Controlled Trials, in Systematic Reviews in Health Care: Meta-Analysis in Context 87 (Matthias Egger et al. eds., 2d ed., Blackwell Publg. Ltd. 2001).

(10.) Id. at 92.

(11.) Kenneth F. Schulz et al., Assessing the Quality of Randomization from Reports of Controlled Trials Published in Obstetrics and Gynecology Journals, 272 JAMA 125 (1994).

(12.) Douglas G. Altman & Caroline J. Dore, Randomisation and Baseline Comparisons in Clinical Trials, 335 Lancet 149 (1990).

(13.) Kenneth F. Schulz et al., Empirical Evidence of Bias: Dimensions of Methodological Quality Associated with Estimates of Treatment Effects in Controlled Trials, 273 JAMA 408 (1995).

(14.) Attrition is not the same as compliance. Compliance is a measure of subject's actual ingestion of a study medication.

(15.) Peter Juni & Matthias Egger, Commentary: Empirical Evidence of Attrition Bias in Clinical Trials, 34 Intl. J. Epidemiology 87 (2005) (citing Austin Bradford Hill, Principles of Medical Statistics (9th ed., The Lancet Ltd. 1971)).

(16.) Id.

(17.) William S. Fields et al., Joint Study of Extracranial Arterial Occlusion: V. Progress Report of Prognosis Following Surgery or Nonsurgical Treatment for Transient Cerebral Ischemic Attacks and Cervical Carotid Artery Lesions, 211 JAMA 1993 (1970).

(18.) See Juni & Egger, supra n.15 (analyzing Fields et al., id.).

(19.) D.L. Sackett & M. Gent, Controversy in Counting and Attributing Events in Clinical Trials, 301 New Eng. J. Med. 1410 (1979).

(20.) See Jayne F. Tierney & Lesley A. Stewart, Investigating Patient Exclusion Bias in Meta-Analysis, 34 Intl. J. Epidemiology 79 (2005).

(21.) See e.g. Joel Lexchin et al., Pharmaceutical Industry Sponsorship and Research Outcome and Quality: Systematic Review, 326 Brit. Med. J. 1167 (2003).

(22.) Id.; see also Justin E. Bekelman et al., Scope and Impact of Financial Conflicts of Interest in Biomedical Research: A Systematic Review, 289 JAMA 454 (2003).

(23.) See Paula A. Rochon et al., A Study of Manufacturer-Supported Trials of Nonsteroidal Anti-Inflammatory Drugs in the Treatment of Arthritis, 154 Archives Internal Med. 157 (1994).

(24.) Helle Krogh Johansen & Peter C. Gotzsche, Problems in the Design and Reporting of Trials of Antifungal Agents Encountered During Meta-Analysis, 282 JAMA 1752 (1999).

(25.) Thomas Bodenheimer, Conflict of Interest in Clinical Drug Trials: A Risk Factor for Scientific Misconduct (2000), www.hhs.gov/ohrp/coi/ bodenheimer.htm.

(26.) Edmund J. Lewis et al., Renoprotective Effect of the Angiotensin-Receptor Antagonist Irbesartan in Patient with Nephropathy Due to Type 2 Diabetes, 345 New Eng. J. Med. 851 (2001).

(27.) Montori et al., supra n. 3, at 1095 (quoting Lindon M.H. Wing et al., A Comparison of Outcomes with Angiotensin-Converting-Enzyme Inhibitors and Diuretics for Hypertension in the Elderly, 348 New Eng. J. Med. 583 (2003)).

(28.) Id.

(29.) Andrew Oxman et al., When to Believe a Subgroup Analysis, in Users' Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice 553, 556 tbl. 2E-2 (Gordon Guyatt & Drummond Rennie eds., AMA Press 2002).

(30.) Karsten Lauritsen et al., Withholding Unfavourable Results in Drug Company Sponsored Clinical Trials, 1 Lancet 1091 (1987).

(31.) See Michael McCarthy, Company Sought to Block Paper's Publication, 356 Lancet 1659 (2000); D.G. Nathan & D.J. Weatherall, Academia and Industry: Lessons from the Unfortunate Events in Toronto, 353 Lancet 771 (1999); Drummond Rennie, Thyroid Storm, 277 JAMA 1238 (1997).

(32.) See e.g. Lisa A. Bero et al., The Publication of Sponsored Symposiums in Medical Journals, 327 New Eng. J. Med. 1135 (1992); Mildred K. Cho & Lisa A. Bero, The Quality of Drug Studies Published in Symposium Proceedings, 124 Annals Internal Med. 485 (1996).

(33.) See Rochon et al., supra n. 23.

(34.) See e.g. Peter C. Gotzsche, Multiple Publication of Reports of Drug Trials, 36 Eur. J. Clinical Pharmacology 429 (1989).

(35.) Patricia Huston & David Moher, Redundancy, Disaggregation, and the Integrity of Medical Research, 347 Lancet 1024 (1996).

(36.) Drummond Rennie, Editorial, Fair Conduct and Fair Reporting of Clinical Trials, 282 JAMA 1766 (1999). In another example, Martin Tramer and colleagues examined full reports of 84 trials--presumably involving 11,980 patients--investigating the effect of ondansetron on postoperative emesis. They found that data from 9 trials had been published in 14 of the reports, without any clear cross-reference, so in fact there were only 8,645 patients in 70 trials. Seventeen percent of the published reports were duplicates, and data from 3,335 patients (28 percent) was reported twice. The authors concluded that this duplicate data led to a 23 percent overestimation of ondansetron's efficacy. Martin R. Trainer et al., Impact of Covert Duplicate Publication on Meta-Analysis: A Case Study, 315 Brit. Med. J. 635 (1997).

(37.) Intl. Comm. of Med. J. Eds., Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication III.D.2 [section] (2006), www.icmje.org/icmje.pdf.

(38.) See Drummond Rennie & Lisa A. Bero, Throw It Away, Sam: The Controlled Circulation Journals, 155 Am. J. Roentgenology 889 (1990).

(39.) Bero et al., supra n. 32.

(40.) Id.

(41.) Sarah Boseley, Drug Firms Accused of Distorting Research, The Guardian (Sept. 10, 2001).

(42.) Frank Davidoff et al., Editorial, Sponsorship, Authorship, and Accountability, 135 Annals Internal Med. 463 (2001).

(43.) For a more detailed presentation of this paper, see Gale D. Pearson, Driving a Truck through Defendants' Claim of Safety and Efficacy in Clinical Studies, 2006 AAJ Annual Convention Reference Materials 1167.

GALE PEARSON is a partner in the law firm of Pearson, Randall & Schumacher in Minneapolis.