Evidence-based practice of research ethics review?
In a recent editorial in the Journal of the American Medical Association, Jeremy Sugarman argues that, "The time is ripe for evidence-based ethics. Similar to evidence-based medicine, an evidence-based ethics would emphasize the importance of data in informing discussions and decision-making about the ethical issues inherent to clinical medicine and research." (1) The Consortium to Examine Clinical Research Ethics headed by Ezekiel Emanuel puts the gathering and use of such evidence into a larger governance perspective by observing that the current US system of oversight for research involving humans "does not systemically assess performance or outcomes." (2) They note particularly the lack of validated measures for evaluating such outcomes and cite the Institute of Medicine's call for the development of an "independent body to develop measures and collect performance data." (3) These recommendations for developing an evidence-based ethics for human subjects protection are quite similar to those made in the first systematic examination of Canadian governance of health research involving human subjects--a study in which both authors of this paper were involved. Based on the research findings in that study, we noted particularly the need for experimentation and research to fill gaps in knowledge, such as appropriate standards for performance-focussed review. (4)
In this paper we explore the need for and barriers to evidence-based ethics for human subjects protection. We present a new analysis of interviews gathered for our earlier study with key informants involved in ethics review or its governance in Canada. While our informants believed that their own review committees were effective, they could offer very little support for this belief. When queried further about the desirability of collecting evidence of effectiveness (or ineffectiveness), contrasting opinions were expressed about needing to trust the integrity of researchers and lack of trust in ethics reviews conducted by other committees. We believe that these contrasting themes of trust and distrust in the context of scant empirical evidence help illuminate the need for and barriers to evidence-based approaches.
Evidence-based practice (EBP), "the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients," (5) has become standard in virtually all health professions. (6) EBP rests on an assumption that while clinical expertise is essential, it is not sufficient for good patient care. Systematic observation and critical appraisal of research literature following specific rules of evidence are thought to reduce the biases that may influence practice grounded solely in experience and unsystematic clinical observation. (7) The criteria for ranking levels of evidence vary from one source to another, (8) but the overall pattern is consistent, with systematic reviews, meta-analyses, and randomised controlled trials positioned as the best level of evidence for or against an intervention. While expert opinion and committee consensus are considered valid contributions to evidence-based practice in health care, they are not seen as particularly rigorous or persuasive forms of evidence.
Important critiques of EBP in health care raise fundamental questions about what counts as evidence, the role of patient preferences and values, the role of clinical experience and expertise, the implications of practice contexts, and even the epistemological grounding of approaches that systematically privilege some forms of knowing over others. (9) Although qualitative research has historically fared poorly in EBP, recently there has been a proliferation of guidelines for critical appraisal of qualitative research studies. (10) While on the one hand some have argued that such systematic approaches to critical appraisal violate core assumptions of naturalistic research paradigms, on the other hand it is clear that within scholarly communities there are--and must be--ways to assess the quality of research. Thus, however we may choose to categorise types of evidence, even those who dispute the conceptual underpinnings of EBP generally agree with the idea that available evidence should somehow be evaluated and incorporated into practice.
To conceptualise ethical review of human subjects research grounded in critical evaluation of existing evidence, it is essential to focus on the fundamental ends or goals of research ethics. Like Brody, (11) we see a relatively broad international consensus on these basic ends, emerging out of a shared concern about cases of abuse of subjects in research. (12) The consensus is expressed in numerous professional, national and international statements, guidelines and regulations of primarily clinical research. In our earlier work on Canadian governance, (13) we expressed this consensus in terms of three primary goals: (1) promoting socially beneficial research; (2) protecting human subjects from unnecessary harms, including affronts to dignity; and (3) maintaining trust between the research community and society as a whole. Public trust in research depends in part on the successful fulfilment of the first two goals but adds important elements of representation, accountability and legitimation to them. An evidence-based research ethics would focus on how well particular research projects are likely to and actually do fulfill these three goals. As a process, it would involve the collection and assessment of information about the achievement of these goals as well as the effective dissemination of the information to the research community (researchers, their institutions and research sponsors) and the general public (in particular populations from which research subjects are drawn).
The Goals of Ethical Review
The empirical research methods used for our study have been reported on in detail elsewhere. (14) The results reported here are based on qualitative interviews with 43 participants, all members or Chairs of Research Ethics Boards (REBs), or members of policy-setting national organizations in Canada. The consensus reflected in research ethics literature concerning the goals or objectives of research ethics review was widely supported among participants. Almost all mentioned protecting research subjects from unnecessary harm, promoting sound research, protecting researchers, and enhancing the credibility of research institutions and the research endeavor as a whole. Though there were some differences in emphasis or priority, most participants named protection of research subjects as the central objective: "If you're looking after that person I can't think of anything that would look bad in the headlines of the newspaper the next day."
Effectiveness of Research Ethics Review
Almost all participants thought current ethics review processes were working effectively; only five disagreed or expressed doubts and concerns. Yet most participants were able to offer little evidence for their beliefs. They spoke of REBs doing a "good job" and described protocols that came to them as "well reviewed." However, when asked how they would know if the processes were effective they admitted they really had no way of knowing.
I think it's very effective. Interviewer: How would you know? Respondent: Yeah, how do I measure it ...? Well, one is tempted to say intuition, which is one way of knowing.
As one REB member said, "I would have to think it's effective since I've been there thirteen years, wouldn't I! It'd be awful to think it's not effective."
One form of evidence for effectiveness offered by several REB members was reputation--with the public, other REBs and local researchers. Participants pointed out that local researchers never had serious quarrels with the review process; some had even appreciated it.
I had comments from people [investigators] who said they had appreciated their reviews.... I get phone calls all the time.... If people [investigators] were really unhappy I would hear about it.
Similarly, the absence of complaints from research subjects was offered by many as evidence that the review process works effectively. Less drastically, several people noted that research subjects rarely, if ever, call the research institution to complain.
Another measure of effectiveness offered by some participants was the fact that the vast majority of research protocols are revised. That was taken as an indication that REBs are identifying ethical problems in research: "I think we have certainly sent back a number of protocols that were methodologically flawed and would not provide any useful information.... So I think that we've been effective." In contrast, the fact that REBs are not finding problems in protocols was also offered as evidence that REBs are doing an effective job. Several participants suggested the fact that protocols submitted for review are of better quality than they were previously indicates that REBs are creating a more ethically-informed research community: "The protocols we are seeing coming in to us are much better written than they've ever been before. When I first came the consent forms were disastrous."
Along with lack of complaints, revising protocols and improved quality of protocols, another form of evidence that was suggested to indicate the effectiveness of review processes was administrative efficiency. Participants mentioned fast turnaround times, a self-administering process, and low likelihood of protocols not being reviewed. As one REB chair said, "I'm told we have the reputation of being effectively efficient."
On the other hand, the evidence offered by the small number of participants who raised doubts about effectiveness was similarly weak, tending toward intuitive and anecdotal claims. As one participant stated, "To be blunt, a lot of REBs have no idea what they are doing ... I don't think nearly enough ethical issues are flagged in your average review. I don't think people are called on things." The doubts raised about the effectiveness of ethics review concerned issues of variability, lack of consistency between committees and the impact of individual committee members on the quality of review.
I would like to say I think it's really effective, but ... I think there's quite a variation in how the protocols are reviewed.... I do worry about the quality of review. I'm not sure I should be saying this, but I really am not convinced.
Participants worried that REBs often "lose track of the broader perspective," concentrating on bureaucratic concerns rather than substantive ethical concerns. Several people suggested REBs are overly concerned with consent forms. In raising these concerns, however, participants did not offer evidence to support their claims--they simply expressed expert opinions. Only one participant cited published research on the experiences of research subjects.
One REB Chair identified the kind of evidence he would find convincing regarding the effectiveness or lack of effectiveness of research ethics review:
A standard protocol should be sent anonymously to Boards to see how they dealt with it.... How do I know that an x-ray specialist is effective? I do what? I give them x-rays more than once and compare the reports. I give a standard x-ray which shows a tumour or pneumonia to six radiologists and see what they say. So, yeah, I think this is what one does.
In contrast virtually the only evidence offered by most participants was that some sites require annual reports, which enable review of the study, at least on the basis of forms submitted by investigators. However, as one participant noted, "realistically you only get what people are telling you."
Lastly, one hospital REB had been conducting annual random audits of research. Each year the REB selected eight to ten studies, for each examining "some of the charts and patient records from their study, to make sure the documentation is complete and that the processes are being followed correctly." As a member of that REB suggested, auditing the paper trail may provide evidence about project management, but it hardly constitutes evidence about research subject protection.
We do audits every year, but it never extends beyond the paper trail. It doesn't extend to the subjects to see what their experience was in a research project. We really leave it up to the investigators. We see the consent, and we see it's clearly written, and has all the features in it that describe the study, and other basic things we look for.
Trust and Distrust
Given the absence of satisfactory evidence for or against the effectiveness of research ethics review, most participants expressed at least a degree of support for the idea of generating such evidence to evaluate how well their work in fact meets the goals of ethical research. Most participants, including many who were adamant that research ethics review is indeed effective, agreed that current awareness of what happens once a protocol is approved is inadequate. One participant argued that it is unethical not to gather ongoing evidence concerning what happens after ethical approval is granted. Another said: "Right now the ethics committee approves stuff and it hasn't got a clue what goes on!... It's reliant on the sense of integrity of the researcher."
This participant articulates a fundamental concern that arose every time participants talked about the absence of evidence concerning the effectiveness of ethics review: participants perceived ethics review of research involving human subjects as resting on trust in the integrity of researchers, and on cultural norms that insist such trust must be essentially unquestioning.
Overwhelmingly, participants argued that trust in researchers is the foundation for the research endeavour.
The system has to be based on trust ... We trust that people aren't going to do things in the classroom that they shouldn't be doing. ... And I want an ethics review process that works on trust ... You tell the researcher what is acceptable and isn't acceptable as an institution. You have to trust people after that.
According to participants, reliance on trust pervades all levels. Funding agencies trust research institutions to implement ethical review; research institutions trust in the conscientiousness and goodwill of REB members; REBs trust in the integrity and professional behaviour of researchers.
In the context of this emphasis on trust as the basis for an ethical system of research, the idea of systematically collecting evidence about what actually happens in terms of protection of research subjects may appear offensive. Participant responses suggest that the gathering
of evidence may be seen as inherently distrustful.
If you start having surprise visits on people to see if they're being ethical, it's going to create a tremendous environment of distrust and resentment. I couldn't stand it myself as a scientist ... You would have to get people basically to surprise their colleagues and spy on them.
Participants talked about monitoring of ongoing research as "snooping," "spying," "policing," and "Big Brother watching."
Intriguingly, the one place where trust seems to be dramatically lacking is when it comes to the work of other REBs. When asked if there were possibilities for streamlining review processes, such as reducing duplication, the major concern raised was that one REB cannot trust another. One participant stated bluntly, "We don't trust the ethics process at other institutions." Another noted, "Everybody says, 'Can't you do something about multi-centre clinical trial review?' And everybody says, 'But we don't trust those other guys!'" There is significant agreement across sites about the fundamental principles and standards of research ethics, and about substantive ethical issues. It appears, however, that when members of one REB do not know how another REB arrives at its decisions and whether a protocol is subjected to the same level of scrutiny, they have no evidence upon which to base trust in other REBs.
These qualitative data suggest that while there is considerable consensus regarding the goals of research ethics review, there is strikingly little evidence about whether in fact these goals are accomplished. Almost all of the participants believed, often passionately, that ethics review is effective. Yet the empirical evidence offered for or against such claims was scant. Intuition, reputation, absence of complaints, quality of protocols, administrative efficiency, and personal perceptions about lack of consistency and narrow focus on consent forms were offered as the best available evidence. This level of evidence might be described as expert opinion based on experience. While expert opinion and practice experience are valid components of clinical decision making, they are rarely the only forms of evidence available.
It may well be that these interviewees are right, that research ethics review is highly effective in meeting the goals of protecting research subjects, ensuring good research and assuring the public. The point is that most of those who claimed their REB was doing an effective job offered little or no empirical basis for that claim. They simply put forward a passionate belief that they must be doing their job effectively, expressing a gut-level sense or "intuition" that all is as it should be, unable to point to any evidence for that claim. Those aspects of research ethics review for which they were able to suggest evidence concerned administration, reputation and perceptions, not protection of subjects. One participant even insisted that the process at her institution was "very tight, very stringent" despite noting that research at the same institution had found research subjects "didn't have a clue" that they were even in research studies let alone what the study was about or who the investigator was. This sounds like belief in the effectiveness of ethics review in spite of empirical evidence. To be clear, the point is not that the interviewees were disingenuous in their claims that they work effectively. The point is that evidence for effectiveness--particularly effectiveness concerning the agreed-upon goals of ethical review--is weak.
While some forms of evidence offered by participants address some ethical objectives, they generally fail to show us the extent to which research participants are being protected from undue harm, which most participants named as the central goal of research ethics review. For example, a high degree of administrative efficiency may advance the progress of research, but it may or may not help to protect research subjects. On the other hand, doubts that ethics review is effective, because it focuses too much on consent forms, for example, offer equally unsatisfactory evidence concerning effectiveness. It is, after all, possible that research ethics review may be imperfectly carried out, yet still manage to adequately protect human subjects. We simply don't know one way or the other. At most what we heard from participants was anecdotal evidence that REB processes often fail to fully examine complex ethical issues.
A major barrier to evidence-based ethics review seems to be a pervasive belief that systematically gathering evidence would undermine a trust relationship that is foundational to current research ethics practices. Systematic outcome evaluation appears to be interpreted as an accusation of pervasive dishonesty. However, we believe that working with human subjects is not a right; it is a privilege with attendant responsibilities. Hence, the burden of proof in research involving humans lies with the research community to demonstrate insofar as possible that subjects are adequately protected.
In this area we see intriguing and perhaps ironic connections to the work of authors on trust in research. For example in the major study of informed consent that Kass led for the human radiation experiments inquiry, (15) trust of research subjects in the beneficence of researchers and ultimately in the benefits of research for the subject emerges as a major theme--despite the awareness of many subjects that no therapeutic benefits were offered in the consent process. But if it turns out that those involved in the governance of research ethics are fearful that an evidence-based approach to research ethics would evince too much distrust of researchers, then it hardly seems that the obvious trust that subjects place in researchers is warranted.
Similarly, Moreno (16) has expressed worries about "strong protectionism" in current modes of governance for research involving humans. He sees this as potentially counterproductive, indeed as possibly leading researchers to take a moral holiday leaving responsibility for research protection in the hands of distant administrators and bureaucrats. While Moreno may well be right in his concerns, the evidence we have found is disquieting on two fronts. First, the reticence of REBs and others to gather systematic evidence concerning the effectiveness of REB processes in meeting ethical objectives may well leave research subjects and the general public in a situation of trusting blindly. Second, the failure to evaluate outcomes through evidence makes the system of protection unwieldy and unreliable in that ethics committees lack the evidence to establish trust in each other's reviews. So to speak, each REB appears to fly by the seat of its pants, but does not trust other REBs' flying abilities.
It is paradoxical that those in the health professions base their practice on evidence-based standards except in the case of ethical review. Health professionals are becoming accustomed to seeking, collecting, evaluating and using the best available evidence in their clinical work. In research ethics, however, the data presented here suggest that outcome evidence is not required, perhaps even contra-indicated. There is no question that systematically gathering evidence in this arena would be challenging. Nonetheless, in the meanwhile, the absence of evidence undermines the goal of promoting good research. For example, when widely publicised instances of research abuse are not countered by solid evidence of the effectiveness of ethical review processes, public trust in the research endeavour is likely to be eroded. Moreover, the reluctance of REBs to accept each others' findings results in delays, expense and ethically-meaningful inconsistencies for multi-site clinical trials. REBs need evidence in order to trust one another.
At the same time, lack of evidence may undermine the respect researchers have for ethical review. REBs are perceived as bureaucratic gate keepers, at best a necessary evil. (17) But this is unsurprising if ethics review lacks the credibility of a solid evidential base. It is all too easy to then dismiss it as a bureaucratic process done only for appearances. Health researchers are trained to be sceptical of anecdotal claims; the lack of evidence-based ethical review fosters scepticism about the value of such review.
An evidence-based approach to ethics review would be grounded in empirical research of various types. There is no reason to import unquestioningly existing evidence hierarchies from EBP. Nonetheless, it is clear that a wide range of research methodologies might be fruitfully employed to contribute valuable research evidence concerning varying dimensions of research ethics. The experiences of research subjects, the meaning of consent, the impact of differing approaches to consent, the reasons research subjects enroll in clinical trials, consistency within and among REBs, experiences of researchers concerning the ethical aspects of their work--these are just some of the aspects of research ethics for which we need empirical evidence. Whatever the methods, it is possible to gather quality research evidence on the impact of ethics review and the effects of research on the ethically defined interests of research subjects.
One urgent step toward an evidence-based process for research ethics would be establishing and validating measures to assess the effectiveness of research ethics processes for achieving "adequate" human subject protection. This obviously would require judicious experimentation and a serious investment of time and resources. At the same time, other less measurable aspects of ethical research need exploration, and exploration in naturalistic research designs, both as a precursor to establishing outcome measures, and as a way of avoiding the pitfalls of an overly-positivist approach to understanding the effects of research ethics processes. The resource investment required could be well worth the effort in terms of the fundamental goals of promoting socially beneficial research, providing appropriate human subjects protection and building warranted trust on the part of the public.
The original data analysed in this study was gathered as part of a larger project funded by the Law Commission of Canada, Contract #98-09-1, on the ethical governance of health-related research involving human subjects. The re-analysis and re-interpretation of data was conducted as part of a current study sponsored by the Canadian Institutes of Health Research. Our thanks are to those who participated in the original study, to our co-researchers on both the Law Commission and CIHR projects, and to our peer reviewers for the Health Law Review.
1. Jeremy Sugarman, "Determining the Appropriateness of Including Children in Clinical Research: How Thick is the Ice?" (2004) 291 Journal of the American Medical Association 494 at 496.
2. Ezekiel J. Emanuel et al., "Oversight of Human Participants Research: Identifying Problems to Evaluate Reform Proposals" (2004) 141 Annals of Internal Medicine 282 at 286.
3. Ibid. at 288.
4. Michael McDonald et al., The Governance of Health Research Involving Human Subjects (HRIHS) (Ottawa: Law Commission of Canada, 2000) at 315 [McDonald, "Governance"].
5. David L. Sackett et al., "Evidence-Based Medicine: What it is and What it isn't." (1996) 312 British Medical Journal 71.
6. See e.g. Evidence-Based Medicine Working Group, "Evidence-Based Medicine: a New Approach in Teaching the Practice of Medicine" (1992) 268 Journal of the American Medical Association 2420 [EBM Working Group]; Jay Siwek et al., "How to Write an Evidence-Based Clinical Review Article" (2002) 65 American Family Physician 251 [Siwek]; Evidence-Based Medicine Working Group, "Evidence-based Health Care: a New Approach to Teaching the Practice of Health Care" (1994) 58 Journal of Dental Educication 648; Canadian Nurses Association, Evidence-Based Decision-making and Nursing Practice (Policy Statement, November 1998); Canadian Association of Occupational Therapists et al., "Joint Position Statement on Evidence-based Occupational Therapy" (1999) 66 Canadian Journal of Occupational Therapy 267; D.L. MacIntyre, C.A. McAuley & D. Parke-Taillon, "Canadian Physiotherapy Research and Evidence-based Practice Initiatives in the 1990s." (1999) 4 Physical Therapy Review 127; Research Committee of the Australian Physiotherapy Association, "Evidence-Based Practice" (1999) 45 Australian Journal of Physiotherapy. 167.
7. EBM Working Group, ibid.
8. Siwek, supra note 6; Bob Phillips et al., "Levels of Evidence and Grades of Recommendation." Centre for Evidence Based Medicine, online:
<http://www.cebm.net/levels_of_evidence.asp>; University Library, University of Illinois at Chicago, "Evidence Based Medicine: Is All Evidence Created Equal?" online: <http://www.uic.edu/depts/lib/lhsp/resources/levels.shtlm>; The Canadian Task Force on the Periodic Health Examination, The Canadian Guide to Clinical Preventive Health Care (Ottawa: Health Canada, 1994).
9. See e.g. Ross E.G. Upshur, Elizabeth G. Von Denkerhof & Vivek Goel, "Meaning and Measurement: an Inclusive Model of Evidence in Health Care." (2001) 7 Journal of Evaluative Clinical Practice 91; Ross E.G. Upshur, "If Not Evidence, Then What? Or Does Medicine Really Need a Base?" (2002) 8 Journal of Evaluative Clinical Practice 113.
10. Joan M. Eakin & Eric Mykhalovskiy, "Reframing the Evaluation of Qualitative Health Research: Reflections on a Review of Appraisal Guidelines in the Health Sciences." (2003) 9 Journal of Evaluative Clinical Practice 187.
11. Barach A. Brody, The Ethics of Biomedical Research: An International Perspective (New York: Oxford University Press, 1998).
12. See generally David J. Rothman, Strangers at the Bedside: A History of How Law and Bioethics Transformed Medical Decision Making (New York: Basic Books, 1991).
13. McDonald, "Governance," supra note 4.
14. Ibid. at 174.
15. United States Advisory Commission on Human Radiation Experiments, Final Report of the Advisory Committee on Human Radiation Experiments (New York: Oxford University Press, 1996).
16. J.D. Moreno, "Protectionism in research involving human subjects" in National Bioethics Advisory Commission, eds., Ethical and Policy Issues in Research Involving Humans, Vol. 2 (Bethesda, MD: National Bioethics Advisory Commission, 2001) I1.
17. McDonald, "Governance," supra note 4 at 179.
Dr. Brenda Beagan, Associate Professor, School of Occupational Therapy, Dalhousie University, Halifax, Nova Scotia. Dr. Michael McDonald, Chair of the W. Maurice Young Centre for Applied Ethics*, University of British Columbia, Vancouver. British Columbia. * Institution to which the work should be attributed. Corresponding author: firstname.lastname@example.org or 902 494 6555
|Printer friendly Cite/link Email Feedback|
|Author:||Beagan, Brenda; McDonald, Michael|
|Publication:||Health Law Review|
|Date:||Mar 22, 2005|
|Previous Article:||Legal and ethical issues associated with patient recruitment in clinical trials: the case of competitive enrolment.|
|Next Article:||A cultural understanding of research ethics governance.|