Forensic science evidence and the conditions for rational (jury) evaluation.
CONTENTS I Introduction II What Jurors Need: The Basic Thesis (in Its Strong Form) A Some Caveats III Enlightenment 'Experts': Contrasting Conventional Legal Practices A Inadequate Insights: Legal Criteria and Their Limitations 1 Conceding Methodological Limitations 2 Speculation about the Validity and Reliability of Opinions Derived from Untested Techniques 3 Speculation about Error and Uncertainty 4 Most Legal Criteria and Heuristics 5 Moderating the Expressions and Terminology Available to the Forensic Analyst 6 The Apparent Independence or Impartiality of the Forensic Analyst 7 Allowing the Jury to Decide Based on What Transpires at Trial 8 Deference to the Forensic Analyst B The Limits of Trial Safeguards in Response to Forensic Science and Medicine Evidence C The Apparent Strength of the Case and the Unrecognised Problem of 'Masking' IV Jury Competence: Jury Research and Legal Attitudes A Research on Expert Evidence and the Common Law Jury B Judicial Experience and the Experience of the Law V Irrationality, Agnosticism and Post-enlightenment Judging
Historically, in response to proffers of forensic science and medicine evidence, concerns about the competence of juries have been assuaged by expressions of confidence in jury abilities, the celebration of trial safeguards and most recently through initiatives intended to improve the presentation of expert evidence, notably training, the use of primers, refined terminologies, new display technologies, encouraging questions, and revised directions and warnings. (1) Whereas trial and appellate judges have maintained strong public confidence in the effectiveness of the trial and the competence of juries based largely on their (individual or institutional) experience, experimental psychologists have based their confidence on empirical studies of jury performance, particularly jury verdicts, exit surveys and responses to complex evidence. Where empirical studies identify limitations--often via experimental recreations of adjudicative processes--jury researchers (and others) tend to suggest that these might be substantially mitigated or overcome through more careful presentation of evidence and relatively minor modifications to conventional trial procedures. (2)
In response, this article explains why pervasive legal confidence, as well as some of the dominant readings of relevant empirical research, seem to be misplaced. It suggests that the reluctance to require techniques (or processes) relied upon by forensic analysts to be formally assessed means that a considerable volume of forensic science and medicine evidence is not conducive to, and cannot be presented in ways that are susceptible to, rational evaluation. (3) Liberal approaches to admissibility (specifically current interpretations of uniform Evidence Acts ss 55, 56, 79, 135 and 137) (4) result in the routine admission of expert opinions without attending to 'specialised knowledge' or the kinds of information and interpretive tools necessary to unpack, understand and evaluate them. (5) This article explains why the failure to formally test techniques means that in most cases limitations and risks cannot be gauged or credibly explored via traditional trial mechanisms such as cross-examination (and testimonial concessions), the use of rebuttal witnesses, or careful judicial instruction. (6) None of these legal safeguards can replace formal scientific evaluation. They do not provide the kinds of information that would enable a decision-maker to assess a technique or ability and rationally assign a probative value to a derivative opinion. This article explains the importance of validating techniques in regular use. Simultaneously, it reinforces the need for more rigorous admissibility gatekeeping by prosecutors and judges. For, unless they are presented in a manner that is susceptible to rational evaluation, the admission of expert opinions threatens the overarching objectives of factual rectitude and fairness.
II WHAT JURORS NEED: THE BASIC THESIS (IN ITS STRONG FORM)
There are certain (normative) assumptions, which do not seem to be controversial, governing obligations and the way evidence is adduced and presented in accusatorial trials. (7) It is, for example, the responsibility of the state, usually through a prosecutor, to persuade the tribunal of fact (or fact-finder) of the accused's guilt. (8) Requiring the state to prove the case against the accused beyond reasonable doubt encapsulates the burden and standard of proof flowing from the presumption of innocence and the premium placed on not convicting the innocent. Apart from a handful of exceptions, such as where mental incapacity is in issue, the accused has few formal obligations and is not expected to prove his or her innocence. (9) Because of the way the burden of proof is allocated the accused does not need to testify or even adduce evidence.
In terms of the evaluation of evidence, there are few rules or prescriptions and even fewer means of imposing them upon the trier of fact. Australian juries decide in camera and do not provide reasons. Their understanding and evaluation of the evidence, along with their reasoning, is largely inscrutable. (10) Nevertheless, the orthodox and longstanding legal commitment underpinning accusatorial prosecutions is to the fact-finder being able to understand all of the evidence presented during the trial. There is no obligation upon the fact-finder to accept it, but the fact-finder should consider all of the admissible evidence. (11) In accordance with this commitment, fact-finders must be placed in a position that enables them to understand and evaluate admissible evidence. (12)
This article is concerned with expert opinion evidence adduced by the state and the ability of the jury (and simultaneously lawyers and judges) to understand and evaluate it. (13) At trial, the fact-finder should be placed in a position to understand and evaluate any opinion presented by a forensic analyst. (14) They should not have to defer to the authority or standing of the expert or the plausibility of the claim. Inexorably, this article draws attention to the performance of lawyers and judges, as well as the effectiveness of criminal proceedings, procedures, rules and safeguards in relation to forensic science and medicine evidence. In relation to the focus of this article, the overwhelming majority of forensic science and medicine evidence is prepared and adduced by the state, and the state maintains a special interest in the conduct of criminal proceedings. (15)
My thesis builds on these orthodox commitments but introduces a largely unexplored set of issues with radical implications for how we understand the operation of our criminal justice institutions. The basic thesis runs as follows: specific types of information are required to evaluate (ie 'weigh') most types of forensic science and medicine evidence. In the absence of this information many types of forensic science and medicine evidence are not susceptible to rational evaluation. To put this in more legally sensitive terms: the attempt to assign a probative value to incriminating opinion evidence is, in the absence of specific kinds of information, unavoidably speculative or impressionistic. (16)
Difficulties arise because criminal trials do not usually supply the requisite information or an environment conducive to the rational evaluation of expert opinion evidence. (17) That is, the kinds of information ordinarily required to make sense of opinions--including insights into the conceptual or methodological reasons for specific kinds of testing, the meaning of test results, as well as the way opinions are expressed--are not routinely provided and are rarely presented in a balanced way. In consequence, the fact-finder is expected (and effectively required) to evaluate incriminating opinions and assign a probative value without the kinds of information and assistance that would enable them to do so rationally.
The thesis, and concerns about conspicuous informational voids, emerged from my work on comparison and pattern recognition techniques, sometimes described as the identification sciences. (18) These are the staples of the forensic sciences, and include: DNA profiling, drug and chemical assays, comparisons of latent fingerprints, ballistics and tool marks, documents and handwriting, gait, shoe, foot and tyre prints, glass, paint, hair, fibre and bite marks, as well as use of image or voice recordings to assist with identification, some forms of digital evidence, and so forth. Following from the thesis it is my contention that in order to understand and evaluate the probative value of expert opinions in these (and many other) domains, quite specific information is required. (19) What follows is a hierarchical list of factors (or criteria) providing the kinds of information that facilitate rational fact-finding. (20) The first three are the most important and, to varying degrees, the remaining criteria flow from them. (21)
First, the most fundamental issue in any attempt to evaluate an opinion or result is to know whether the underlying technique or process used to produce it actually works. This applies whether the technique is a complex technical process (such as DNA profiling) or a beguilingly straightforward visual comparison of two images on a screen (for latent fingerprints, ballistics and facial mapping). It also applies to new techniques and those in long use. Determining whether a technique works--ie does what it is supposed to--requires some kind of experimental test or evaluation, usually an independent validation study. Validation studies should be conducted in conditions where the 'ground truth' is known so that the performance of the technique can be assessed against an answer that is known to be correct. (22) Validation studies tell us whether the technique (or process) actually works, how well, and in what conditions.
A recent report on the forensic sciences prepared by the United States research committee of the National Academies of Sciences explained validation (and reliability) in the following terms:
One particular task of science is the validation of new methods to determine their reliability under different conditions and their limitations. Such studies begin with a clear hypothesis (e.g., 'new method X can reliably associate biological evidence with its source'). An unbiased experiment is designed to provide useful data about the hypothesis. Those data--measurements collected through methodical prescribed observations under well-specified and controlled conditions--are then analyzed to support or refute the hypothesis. The thresholds for supporting or refuting the hypothesis are clearly articulated before the experiment is run. The most important outcomes from such a validation study are (1) information about whether or not the method can discriminate the hypothesis from an alternative, and (2) assessments of the sources of errors and their consequences on the decisions returned by the method. (23)
Almost all techniques (or processes), and most expert claims, can and should be subjected to validation or some kind of analogous experimental testing. (24) In most cases the results of experimental testing should be published or made available through publicly accessible repositories. (25) All expert opinion evidence in uniform Evidence Acts jurisdictions should be based on identifiable 'specialised knowledge'. Validation studies provide the kind of knowledge that can support the opinions of those with relevant 'training, study or experience'. (26)
Second, as the extract above implies, there is also a need to direct attention to limitations and uncertainties and the error rate associated with the technique. (27) These provide insight into limitations, the number of mistakes, and some of the risks associated with the application of the technique (or process). (28) Indicative error rates are usually generated through validation studies, and occasionally through system-wide tests. (29) Though, uncertainties, limitations and risks might, in addition, be extrapolated from other domains such as biomedicine. (30)
Third, the last of the major factors is evidence of the analyst's proficiency using the validated technique (or process). This is to be contrasted with long experience, including experience doing similar things. For, rigorous proficiency testing provides information about the abilities of the forensic analyst. It confirms that the analyst is competent using a particular technique and is an expert in the specific domain. (31) That is, regardless of qualifications and experience, rigorous proficiency testing tells us whether the forensic analyst performs a task or set of tasks better than non-experts or chance. A significantly enhanced level of performance is precisely what it means to be an expert. (32) Evidence that techniques work, and that the forensic analyst performs markedly better than an ordinary person, is what courts should require before opinions are presented to fact-finders. (33)
Beyond these fundamental criteria there is a need to attend to standards. Are there any? Were they developed through formal testing and were they used in the specific application of the technique in the case? Techniques should be standardised and standards should be based on the results of experimental, usually validation, studies. Standards developed or imposed without formal evaluation may have limited value. (34) Similarly, proficiency with a technique that has not been validated, even if the witness purports to have applied 'standards', produces opinions of unknown value.
In many cases there should also be an explanation of contextual bias and other threats to cognition and interpretation, along with a description of the efforts to avoid or eliminate them. Notwithstanding their historical interest in partisanship and bias, especially 'adversarial bias' in civil proceedings, Australian judges are yet to respond to the serious threat to forensic science and medicine evidence posed by a range of undesirable practices that remain remarkably common within investigative communities. (35) Without wanting to trivialise the dangers created by forensic analysts working intimately with investigators as a team or being aligned ideologically with the police or prosecutors, these may not be the major threats to cognition and interpretation. (36)
Several peak scientific organisations have recently expressed grave concerns about the way forensic analysts are routinely provided with information about the case and the accused that is often prejudicial or suggestive but not required for the analysis. (37) Preliminary studies, along with experiences in other domains of scientific and biomedical endeavour, have demonstrated that even subtle suggestions and environmental influences can change the way analysts select and test samples, interpret data and report experimental results. Specific studies have shown that interpretations of latent fingerprints and DNA electropherograms can be influenced--ie substantially altered and even reversed--by exposing the analysts to gratuitous information or suggestive processes. (38) These studies indicate that opinions derived from techniques that are otherwise valid and reliable may be vulnerable to insidious cognitive biases and, moreover, that long experience and even awareness of the dangers posed do not guarantee resilience to such biases. (39)
The threat posed by contextual bias is particularly invidious because the influence can be subtle. Analysts are not always conscious of their exposure to biasing information or its influence. (40) The upshot is that exposure and influences might not be documented and may be difficult to explore during the trial. How, for example, do you cross-examine a forensic analyst about unconscious influences? (41) The dangers created by suggestion and other cues and biases are the main reason why most clinical trials are double-blind--neither the doctor nor the patient knows who received the active ingredient and who received the placebo (or comparator). Significantly, there is no evidence that training and experience enable forensic analysts to circumvent the effects of cognitive biases, especially where evidence is ambiguous or interpretation difficult. Experienced medical doctors have proved incapable of avoiding such influences. (42) The most effective response to threats to cognition seems to be to restrict access to information rather than try to document exposure or somehow tease out subtle psychological influences in their aftermath, sometimes years later. Contextual bias is a real threat to forensic science and medicine evidence but is yet to receive serious consideration by appellate courts. (43)
Information about the frequency of feature(s) in relevant populations and whether they are independent of one another is vital where analysis (or extrapolation from analysis) involves some kind of matching or comparison in an attempt to link a trace to a person or an object. (44) Where techniques are based on comparisons, it is essential to have an idea, drawn from empirical data, of how frequent features of interest are in specific populations. This applies to population genetics and DNA databases (for DNA profiling), fingerprints, the soles of shoes, accents and intonations (for voice comparison), face and body features and even tattoos in image comparisons, as well as to posture and movement for forensic gait analysis. In the absence of empirically-based information on the distribution of features, attaching significance to (apparent) similarities or 'matches' can be dangerous. (45) Where there appears to be more than one similar feature we also need to know the degree of in(ter)dependence between them. For, independence of features will often strengthen the significance of actual similarities. This information, along with an awareness of the value of a technique and the level of error (and other uncertainties and limitations), enables opinions to be assessed. In its absence, the appearance of one or more similar features may or may not be particularly probative.
Finally, those asked to evaluate the opinions of forensic analysts need to be confident that the particular expressions--ie the form of words--selected to capture results and convey opinions are derived from and supported by independent research. (46) The way forensic analysts express their opinions (ie conclusion or interpretation) should be based on what validation and proficiency studies can support. The results of validation studies (and information about frequency), as well as error rates, other limitations and uncertainties and proficiency, should structure and constrain the way conclusions are expressed in reports and testimony. Moreover, opinions should be expressed in ways that are both scientifically defensible and simultaneously conducive to comprehension by laypersons. (47) Managing these needs may be difficult, but however we moderate expressions to ease comprehension at trial, the underlying analytical process must be valid and reliable. (48)
These kinds of information and insights, especially the first three (namely, independent validation studies, error rate and limitations, and proficiency), are vitally important in any attempt to assess the probative value of opinions associated with pattern recognition and comparison techniques and most other kinds of scientific and medical processes. Together, validation studies, error rates and insight into the proficiency (or competence as opposed to bare experience) of the analyst, provide a yardstick against which an opinion can be evaluated. (49)
In principle, given the professional responsibilities and formal obligations of prosecutors (eg bar rules and prosecutorial guidelines) and forensic analysts (eg professional guidelines, codes of conduct and practice directions), information about validity, error rates and limitations, and proficiency should be proactively disclosed and explained during the presentation of the prosecution case. (50) It is the responsibility of the prosecutor to present expert opinion evidence in a manner that reflects its known value. (51) Where techniques have not been formally evaluated or are speculative, the prosecutor should not adduce the opinion. For, it is not an opinion based on 'specialised knowledge'. (52)
Many years ago, writing for an appeals court in Scotland, Lord President Cooper somewhat presciently drafted the following passage:
Expert witnesses however skilled or eminent can give no more than evidence. They cannot usurp the functions of the jury or Judge sitting as a jury ... Their duty is to furnish the Judge or jury with the necessary scientific criteria for testing the accuracy of their conclusions so as to enable the Judge or jury to form their own independent judgment by the application of these criteria to the facts proved in evidence. The scientific opinion evidence, if intelligible, convincing and tested, becomes a factor (and often an important factor) for consideration along with the whole other evidence in the case, but the decision is for the Judge or jury. In particular the bare ipse dixit of a scientist, however eminent, upon the issue in controversy, will normally carry little weight, for it cannot be tested by cross-examination nor independently appraised, and the parties have invoked the decision of a judicial tribunal and not an oracular pronouncement by an expert. (53)
This extract was reproduced in the High Court of Australia in Dasreef Pty Ltd v Hawchar and has been approved by courts of appeal in New South Wales, Victoria and Western Australia. (54)
This article is a modern elaboration of this basic commitment: explaining the need to provide information that enables the fact-finder to evaluate expert opinion evidence and to exclude the evidence where this information is not provided. (55) Unfortunately, many techniques used by forensic analysts and relied upon by prosecutors--including some techniques in regular use--are yet to be evaluated and so cannot be expressed in terms that reflect known values or enable rational evaluation. (56) In the absence of formal evaluation we do not know if techniques and derivative opinions are probative. (57) Even if we assume that opinions are probative (ie that they could be 'accepted' by a jury) as part of our test for relevance we have no way of evaluating them. (58) Admission in these circumstances invariably introduces a real risk of unfair prejudice to the defendant through misunderstanding, over-reading or unwarranted deference. (59)
A Some Caveats
Before proceeding, a few qualifications and clarifications are useful. In advancing the thesis, it is not my intention to suggest that there is only one way to interpret expert opinion evidence, nor do I intend to suggest that there is only one proper value. Rather, it is my contention that in most circumstances specific types of information need to be provided to, and understood by, a decision-maker before they can rationally evaluate--ie attribute probative value or 'weight' to--an opinion said to be expert. In most cases, formal evaluation (through independent testing prior to the investigation and trial) will provide a range within which the expert opinion will fall. Of course, where within this range the evidence is to be placed is a matter for the tribunal of fact. (60) My basic contention is that, absent requisite information, the allocation of a value tends to be capricious (even though the decisionmaker may think otherwise). For, there is always a non-trivial danger that the fact-finder will assign a probative value that is higher than the technique (or the proficiency of the analyst) can sustain. In the absence of validation and rigorous proficiency testing we cannot discount the gravity of this risk.
The previous paragraph recognises that fact-finders might assign probative value to the opinions of forensic analysts regardless of how the evidence is presented, explained or challenged. The point is not that our fact-finders have been unable to make decisions, rather the problem is that they have been deciding in ways that tend to be inattentive to highly pertinent, and arguably essential, information. The failure to produce and provide the kinds of information that are conducive to rational decision-making not only makes legal proceedings inefficient, cumbersome and inconsistent but it means that they fall short of espoused commitments to fairness and are more likely to result in mistaken verdicts that are difficult to identify and correct.
Some might contend that provided the fact-finder understands the 'bottom line', gist or direction of an expert's opinion that is sufficient. Such arguments tend to undervalue the historical commitment to decision-makers actually understanding the evidence (and any limitations or criticisms). (61) Simultaneously, it might encourage the jury to place excessive reliance upon non-scientific evidence and epiphenomena (see below Part III) to mediate the reception and interpretation of opinions presented as scientific or technical.
Finally, in advancing the thesis, it is useful to note that forensic science and forensic medicine evidence can be distinguished from most other types of evidence routinely admitted in criminal proceedings. The state, after all, has far more control over what happens 'backstage'. Significantly, the state is in a position to evaluate techniques, develop empirically-based standards, shield analysts from gratuitous information, enforce rigorous proficiency testing regimes, as well as select the terminology used in reports and testimony. (62) The fact that these things have not been routinely addressed does not mean that they do not matter or should not be undertaken. (63) Rather, these oversights implicate lawyers and judges in the pre-scientific and speculative condition of many techniques and practices used by our forensic analysts and relied upon by prosecutors. Liberal admissibility standards, along with the actual weakness of trial safeguards and prosecutorial obligations (see below Part III(B)), have contributed to the questionable epistemic status of a great deal of forensic science and medicine evidence.
III ENLIGHTENMENT 'EXPERTS': CONTRASTING CONVENTIONAL LEGAL PRACTICES
Information conducive to the rational evaluation of 'opinion based on specialised knowledge' is sometimes presented in criminal proceedings in Australia. Adequate disclosure and detailed explanation is, however, the exception. (64) Ordinarily, to the extent that validation, error rates, limitations, and proficiency are raised, they tend to be traversed in rather superficial ways, where it is not obvious that lawyers, judges and fact-finders appreciate their significance or the significance of their absence.
Prosecutors and forensic analysts do not routinely present evidence in ways designed to enhance understanding, let alone facilitate, rational evaluation. Defence lawyers, whether through resourcing constraints, incompetence or for 'tactical reasons', irregularly develop oversights, omissions and limitations in ways that are comprehensible to laypersons or would persuade them of their epistemic significance. Judicial instructions and directions rarely capture or convey the significance of omissions and knowledge gaps. Judicial instructions might touch upon them but they seldom endorse, or lend legal imprimatur, to the significance of methodological limitations and oversights. (65) Rather, they tend to leave the significance of these fundamental deficiencies as jury questions--issues of weight for the fact-finder to somehow 'factor in'. Similarly, courts of appeal have not taken the failure to validate sufficiently seriously. Trusting in the trial and trial personnel, and in many cases the strength of the overall case (on review), they tend to uphold the admission of opinions based on techniques that are testable but are yet to be independently tested. (66) Appellate courts display a tendency to defer to the imagined capabilities of laypersons informed by Panglossian impressions of the effectiveness of trial safeguards.
A Inadequate Insights: Legal Criteria and Their Limitations
Lacking the kinds of insights and information that would facilitate rational evaluation of forensic science and medicine evidence, our courts have developed a range of alternative criteria. These include: requiring the forensic analyst to concede that validation testing has not been performed; allowing forensic analysts to offer 'educated' guesses about validity, error rates and limitations; imposing constraints on the form of expression the analyst might use; relying on legal (admissibility) criteria such as the existence of a 'field, formal qualifications, experience, previous admission, and apparent impartiality; as well as expecting the defence to expose limitations in conjunction with a tendency to treat defence failures, oversights and omissions as tactical and therefore deliberate. (67) In sharp contrast to the kinds of indicia outlined above in Part II, these legal criteria (or heuristics) direct attention toward secondary qualities or epiphenomena. The remainder of this Part endeavours to explain some of the problems and limitations with the kinds of criteria and mechanisms ordinarily relied upon to evaluate the opinions of forensic analysts in trials and on appeal.
1 Conceding Methodological Limitations
Disclosing or conceding that formal evaluation has not been undertaken should not provide an admissibility pathway for opinions based on untested techniques. (68) Significantly, disclosing that appropriate testing has yet to be performed does not enable a fact-finder to evaluate the technique. All such concessions do is reveal that scientifically appropriate forms of evaluation have not been conducted. There seems to be a belief that such concessions will enable those lacking methodological training and technical sophistication to appreciate the significance of methodological shortcomings and omissions. However, 'known unknowns' do not place the fact-finder in a position to evaluate opinion evidence. Rather, they provide insight into what needs to be done to produce knowledge. The absence of relevant information obliges the fact-finder to rely on criteria and considerations that may not be particularly informative. In the context of the trial, for example, admission might be used by jurors to draw the inference that the technique is basically sound or that other criteria are of more utility than formal evaluation.
2 Speculation about the Validity and Reliability of Opinions Derived from Untested Techniques
In the absence of formal evaluation, undertaken in conditions where the correct answer is known, the analyst should not be invited to opine on the probative value of the opinion. In these circumstances claims about validity and reliability, whether explicit or implicit, are unavoidably speculative. The analyst does not know whether the technique works, nor how often, nor how well. This not only applies to untested techniques, but also to novel applications of proven techniques. Independent testing provides insights into validity and reliability. We should not allow forensic analysts to speculate about the value of a technique or express their confidence in a particular opinion in the absence of testing.
Where the analyst is highly trained and experienced, the significance of validation, error rates and limitations, standards, proficiency and contextual bias, may be marginalised or obscured. (69) Indeed, dangers are acute where the results of testing are not available to mediate the opinions of those presented as experienced. An analyst's experience, confidence and the strength of his or her opinion are not necessarily correlated with ability or accuracy. (70) Inattention to appropriate criteria introduces the risk that a forensic analyst will (inadvertently) exaggerate the probative value of his or her evidence, and unreasonably reject or trivialise (defence) criticisms and mainstream scientific insights. (71) A significant proportion of our forensic analysts, particularly those in areas such as crash investigation, ballistics and tool marks, document examination, and latent fingerprint, shoe, foot and tyre print comparisons, do not possess formal scientific qualifications. Many more are not conversant with relevant methodological precepts, probabilistic approaches to expression and a range of risks that inform practice in mainstream biomedicine, engineering and science. (72)
3 Speculation about Error and Uncertainty
In the absence of formal evaluation, claims about error rates, accuracy and uncertainty are speculative, and likely to be wrong or misleading. As with speculation about the validity of techniques, forensic analysts should not be allowed to express incriminating opinions at trial on the basis that the defence is entitled to try to obtain concessions about the analyst's impression of errors, uncertainties and other limitations. Making the defence responsible for exposing limitations and oversights will, in many cases, require them to ask a forensic analyst--who has not proactively disclosed or acknowledged limitations in their report and testimony, and might not even understand or accept them--whether such limitations exist and ought to be taken into consideration. (73)
4 Most Legal Criteria and Heuristics
Conventional admissibility criteria and heuristics--such as formal qualifications, a 'field', 'training, study or experience, prior legal recognition and admission--do not provide direct insight into validity, error rates and limitations, or proficiency. (74) This observation is both confronting and revealing. Our admissibility criteria do not focus attention on the validity and reliability of expert evidence. They do not require those adducing and relying upon the opinions of forensic analysts--in the vast majority of proceedings, the prosecutor--to demonstrate that the underlying technique is demonstrably reliable and that the analyst is proficient in its use. Rather, admissibility is predicated upon superficial or indirect indicia such as formal qualifications in a relevant (or apparently relevant) 'field', whether a court has recognised the 'field', and whether the forensic analyst has done this sort of thing before or has been previously admitted (in Australia or elsewhere). (75) Legal criteria are largely focused on the 'trappings' of expertise. They reference and privilege past legal practice, antiquated images of expertise and trial safeguards above formal evaluation and evidence of validity and proficiency. (76) They are relatively easy to satisfy and therefore facilitate the reception of weak, speculative and unreliable opinions. (77) Legal criteria facilitate the admission of opinions as expert without requiring proof of relevant expertise.
Unfortunately, these kinds of criteria cast an unfortunate pall over the trial as they often loom large, indeed far too large, in the imaginations and understandings of lawyers and judges. Legal criteria are likely to inform and sometimes dominate examination-in-chief and cross-examination, challenges to the opinions of forensic analysts, as well as efforts to repair credibility and probative value in re-examination. They also tend to dominate the representation, and presumably understanding, of forensic science and medicine evidence on appeal. This might not be completely unexpected where the necessary testing has not been performed and the significance of this and other oversights is not always recognised. Attention to legal factors is likely to create confusion and confound good decision-making. Legal factors also tend to waste time and resources as relatively insignificant, secondary considerations are explored at the cost of attention to more fundamental epistemological issues. (78)
5 Moderating the Expressions and Terminology Available to the Forensic Analyst
Several courts have imposed restrictions on the kinds of expressions that particular types of forensic analyst might use as a kind of admissibility compromise. (79) Though, there is remarkably little coherence across the expressions used for ostensibly similar comparison techniques. (80) In most cases the terms used to express a forensic analyst's opinion are not based on the results of formal testing, do not draw upon empirical evidence of frequencies, and do not incorporate known limitations, uncertainties and the inescapable risk of error. (81) While judicial attempts to moderate the strength of opinions might reduce some of the excesses, they do not address fundamental questions about whether techniques work, how well they work and whether the analyst is proficient. They do not provide information that would enable the fact-finder to evaluate the qualified opinion and may, somewhat perversely, make a forensic analyst, relying upon a technique that is yet to be validated, appear more reasonable than if left to his or her own devices.
6 The Apparent Independence or Impartiality of the Forensic Analyst
It is highly desirable to have access to independent expert opinions. (82) Independence is generally desirable but it cannot overcome the failure to have formally evaluated a technique. Independence cannot make an impression or interpretation, derived from an untested technique, valid.
In thinking about the (mis)use of independence, we should not overlook the structural disadvantages confronting the accused and their defence lawyers at trial. (83) Whereas forensic analysts called by the state tend to enter proceedings (portrayed) as disinterested and highly-experienced experts, having few apparent interests in the instant proceedings, expert witnesses called by the defence are relatively easy to portray as interested and partisan. Defence experts are usually paid for the specific appearance and do not obtain the 'insulation' available to the state's forensic analysts from routine employment and experience. (84) In many cases defence witnesses are not in forensic practice or part of a forensic institution. Most operate primarily as rebuttal witnesses or 'consultants'. These consultants undertake independent analysis of the specific trace or sample relatively rarely. (85) Rather than positive evidence of non-guilt, they usually present methodological insights and criticisms directed at the assumptions, practices and the conclusions of the state's highly-experienced and apparently disinterested forensic analysts. Such rebuttal experts are relatively easy to stereotype as 'hired guns' or ivory tower academics whose testimony can be portrayed as motivated or a counsel of perfection. (86) Compounding their apparent limitations, rebuttal witnesses called by the defence usually only respond to one part of the prosecution case (or story)--the forensic science or medicine evidence--and may be the only witness called by the defendant. (87) Where the rebuttal expert is espousing mainstream, and highly pertinent, methodological criticisms of the state's inadequate forensic science evidence, in the midst of adversarial proceedings this may be very difficult for judges and jurors to appreciate. The alignments imposed by our system of investigation and prosecution, along with continuing reliance on misguided legal heuristics, have a tendency to privilege forensic analysts called by the state and in the process obscure the significance of methodological precepts from the mainstream sciences. (88)
7 Allowing the Jury to Decide Based on What Transpires at Trial
The jury may be asked to determine guilt without the kinds of information that would enable them to credibly evaluate the expert evidence. Where the jury is not provided with information about validity, limitations, errors and proficiency, assessment of a forensic analyst's impression is unavoidably speculative. In consequence, the jury will be obliged to guess and in the process may rely upon available, though potentially misleading, heuristics. (89) In many, perhaps most, cases the resources required to rationally evaluate the expert opinion evidence are not provided to them. (90) In these circumstances there is always a real (ie non-trivial) risk that the forensic analyst cannot do what he or she claims, that he or she will perform at a level significantly below that suggested, or that his or her opinion will have been contaminated by other evidence or the circumstances in which it was obtained. There is always a real risk that the jury will assign a higher probative value, and possibly a much higher probative value, than the technique can support--ie even taken 'at its highest'. (91) Courts should not reward the state's failure to use orthodox scientific methods and testing protocols by allowing the jury to speculate about validity and weight or to assign any weight they deem appropriate. To leave such opinions to the fact-finder always threatens to be unfairly prejudicial to the defendant because validity, reliability, error rate, the value of experience, the issue of contamination and so on, cannot be positively resolved through cross-examination or rebuttal evidence.
Conventional legal approaches to the opinions of forensic analysts oblige fact-finders to rely on their impressions of techniques in conjunction with other considerations such as qualifications and experience, confidence, demeanour, credibility and the analyst's resilience in response to any cross-examination or criticism. Impressions of the analyst's demeanour, credibility and performance, even in conjunction with other considerations such as apparent plausibility, do not provide adequate grounds for determining whether a technique works or whether the analyst is proficient. They do not provide 'good grounds' for ascribing a probative value to the opinion evidence. (92) Rather, they require fact-finders, to the extent that they engage with the incriminating opinions of forensic analysts, to trust or defer to the expert, to rely on factors that do not directly address the question of expertise (and validity, reliability or proficiency), or to use other strands of prosecution evidence as a 'makeweight'--this last issue is developed in Part III(C).
8 Deference to the Forensic Analyst
The risk of deference to the state's forensic analysts is heightened where limitations and uncertainties are not presented to the fact-finder. Earlier, we saw that deference is conventionally seen as undesirable and at odds with our ostensibly rational approach to fact-finding. (93) It seems to follow that we should not allow deference--or attribute comprehension to fact-finders--where requisite testing has yet to be performed and limitations with techniques and derivative opinions are not known and so cannot be clearly explained.
In sum, the main point is not that conventional legal criteria are without significance. Rather, it is that in general they do not provide direct insight into probative value, the abilities of analysts, or features of the investigative environment that might have compromised the reliability of the opinion evidence. Legal criteria do not provide consistent guidance and are likely to mislead. They might be used to negate or reject specific opinions, such as where an analyst appears excessively partisan, does not possess qualifications or relevant experience, or implodes under skilled cross-examination. Legal criteria are, however, of less utility when it comes to assigning a positive value to an opinion. (94)
Individually and collectively conventional legal insights do not enable us to determine whether a technique works, how well and in what conditions, whether the conclusion is expressed in appropriate terms, and whether the analyst is genuinely expert. These, it might be thought, are serious constraints on decision-making.
|Printer friendly Cite/link Email Feedback|
|Title Annotation:||I. Introduction through III. Enlightenment 'Experts': Contrasting Legal Practices A. Inadequate Insights: Legal Criteria and their Limitations, p. 77-104|
|Publication:||Melbourne University Law Review|
|Date:||Aug 1, 2015|
|Previous Article:||Lessons lost in sentencing: welding individualised justice to indigenous justice.|
|Next Article:||Forensic science evidence and the conditions for rational (jury) evaluation.|