Printer Friendly

What's crucial in evaluation research: a reply to Bennett.

Beware researchers proclaiming an innocence in respect of methodological principles. Every nut, bolt and washer in laboratory instrument is the carrier of epistemological assumptions. Every aspect of design, data collection and analysis in social research bears a commitment to a particular model of social explanation. This paper examines Bennett's reconstruction of his own research, detects some wishful thinking and seeks to extract from the debate further clarification on what is crucial in evaluation research.

Our `What Works, (1994) article is part of a larger attempt (1995, 1997) to bring scientific In@j to bring scientific realist thinking to bear on the practice of evaluation research. In What's New' (1995), Bennet begins his reply by reporting what we take to be crucial in realist investigation, namely that, `outcomes unearthed in empirical investigation are intelligible only if we understand the underlying mechanisms which give rise to them and the contexts which sustain them'. We are somewhat gratified to have this little proposition about generative causation brought to the centre of attention because we believe it has powerful ramifications for the way we manage all social research. Bennett, however, is rather less impressed. Having looked over some of our earlier writing on the realist conception of causality, he remains unconvinced that such thinking is in any way contrary to experimental evaluation. He continues in this vein in the main body of his critique, seeking to show that the notions of mechanisms and contexts are indeed part of the argot of the pioneers of the quasi-experimental method, and, moreover, that subject to a budgetary constraint or two, his own evaluation (1989, 1991) of `contact patrols' was indeed attentive to these underlying features. We remain in blunt disagreement on all counts.

Bennett seems not to grasp the basic difference between the principles of successionist and generative causation. He caricatures our depiction of the former as one of `merely seeking associations between treatments and outcomes' and goes on to floor us with the old chestnut that `no one believes that association is itself sufficient proof to determine causality (this issue, p. 569). In truth, we had assumed that this debate had gone some way beyond the correlation-does-not-equal-causation level. The real dispute, of course, is located after the moment in which the interesting empirical association has been turned up and concerns what additional inferential strategies are brought to bear in order to establish it as causal. Successionists (and thus, by our lights, experimentalists) prefer a control-based strategy. They aim to show that input actually led to output by seeking to hold constant (by randomization, matching, partial regression, etc.) all other potential confounding influences. Realists use a theory-driven strategy in which they assume that the regularity is constituted by the action of some mechanism, and knowledge of the context in which such mechanism is likely to be fired is used to organize empirical work.

These are the proper terms of our argument, which we can now use to assess Bennett's case. In describing the successionist strategy, he claims that we invent a philosophy which bears no relation to that espoused by the architects of quasi-experimentation whose real concerns, he insists, were those of method. This is a strange description of the interests of Donald Campbell who spent a lifetime trying to dovetail research principle and practice. What is more, if we turn to the very first page of the key text cited by Bennett, we find Campbell (and Cook) discussing the relationship between different philosophies of causation and the language of experimentation. Read on and we find the surprising admission that their blueprint for quasi-experimentation, `deliberately adopt(s) an outmoded position derived from Mill's inductivist canons' (Cook and Campbell 1979:, our italics). What is more, and unlike Bennett, the founding fathers of field experimentation are aware of a range of alternative philosophies of causality. They thus take time to consider what we call generative causation (which they refer to as micromediation), which is rejected as aspiring to unachievable levels of explicitness and confused on the matter of bi-directional causality (1979: 14-18). Needless to say, we do not share this assessment, which was made in advance of the most substantial body of realist writing and which can be thought of as `early-Campbell' on `essentialist-realism' (see Shadish et al. 1994: ch. 4, on Campbell's changing philosophy, and Pawson 1989: ch. 6, on varieties of `realism').

We do not suppose, of course, that these brief citations begin to settle the argument. The crucial point for present purposes, however, and contra Bennett, is that from thc outset, philosophical preferences were there and were avowedly successionist. Cook and Campbell (1979: 33) even use the expression `molar causation of the traditional push-pull variety' to describe their working model of causality. Alas, much of their legacy has consisted in blind, control-based applications of their methods which show no understanding of alternative explanatory strategies, and which appreciate none of the dilemmas which led the founders of a method to describe their philosophical preferences as outmoded.

Bennett's understanding of the realist interpretation of causation is also flawed, missing repeatedly the key point that it is an attempt to codify the nature of explanatory propositions. Social programmes are, at heart, attempts to make people change. What brings the change about for a realist is a `mechanism'. The realist thus assumes that any observed change is actually constituted by the action of some real underlying process. So while we can speak, in a kind of shorthand, of neighbourhood policing causing reduction in fear of crime, we know that it actually takes interaction, persuasion and reasoning to change people's minds. Thus the realist begins investigation by asking `what it is about a programme which might cause it to work?'. Programmes are complex, prolonged encounters. Evaluation thus has the task of discovering precisely what new ideas and resources are presented to subjects and whether these have the capacity to change future thinking and action. Ideas, of course, are unequal in their power to transform and so inquiry faces the further daunting task of considering ways in which the `programme theory' might be welcomed, resisted or overlooked.

The nature of this reception depends in turn on the resources of the subjects, which turns the explanatory circle on to context. Context refers to the culture, resources, and opportunity structures which enable certain actions and constrain others. Context is thus not something we can control out in programme evaluation because programmes do not work in contextless situations. For the realist, context refers to the community culture which provides the seedbed for the development of the problem; context refers to those community features which prompt it to be targeted by programme managers; context is the existing normative structure which renders plausible or implausible the success of entire social programmes. In short, context is no `unusual event', no `threat', no `confounding variable' which one should eliminate from enquiries. For the realist, the task is to incorporate.

The two previous paragraphs introduce the language of generative causation. They form the beginnings of a realist ontology of the nature of change within social programmes. They set challenging priorities for investigation which we happen to believe are poorly served by the existing repertoire of quasi-experimental techniques. This brings us again to Bennett's research. He seeks to steal our realist clothes in claiming that all along his inquiry was attentive to the contexts and mechanisms of contact patrols. We still beg to differ. Instead of understanding mechanisms, he deals with mechanics. Rather than considering the salience of context, he provides a touch of local colour.

Mechanisms, for us, refer to the set of ideas and opportunities embedded in a programme. Programmes are, quite literally, suggestions for future action and they work (or not) according to whether these `suggestions' enter the reasoning of subjects, so changing that reasoning and subsequent action. The basic test of the realist proposition, therefore, is to discover which mechanisms within the programme lead to which outcomes. Of contact patrols, we would therefore want to ask which of their many different features leads to a change in levels of fear of crime. Bennett's report deals extensively with the workings of the contact patrol programme, but not in this way. To be sure, Bennett has written papers elsewhere (1990) on theories of fear of crime. To be sure, he introduces his `contact patrol' report (1989: 8) by outlining the same broad `ideas', namely the `social control' perspective, and the victimization perspective' which were built into the original US fear-reduction programmes. The point is, however, these theories disappear and remain untested in the course of the research because Bennett has quite different purposes in mind when inspecting the actual functioning of the patrols (1989: 27-58).

Let us look closely at what is reported. We are given an extensive, descriptive account of the operation of the police team, the nature, apparatus and definition of the contact, and a tale or two about subsidiary events and leafleting which accompanied the programme. This touches on some crucial differences in the nature of the police response but the detail is there basically to serve a `familiarization function' to enable the reader to get a `feel' for what went on. Then there is the major `audit function' in assessing `implementation effectiveness'. This details the admirably high rates of `contact' and `beat integrity'. It is the pivot of the analysis of the programme but, as we pointed out in our initial paper, the purpose is not to investigate how the programme actually worked but rather to show that that it was there and there in good measure, thus allowing Bennett to conclude, the main programme elements were implemented effectively and constitute a programme capable of being evaluated in terms of its outcome effectiveness'. Finally, there is a brief exploration of police perceptions, the message of which is that the programme creates job satisfaction and optimism about the community. This allows Bennett to praise the organizational skills and motivation of the police team and thus serves an `accreditation function'.

All of these features (scene setting, thoroughness, audit, mindfulness of sponsors) go to make Bennett a master of the commissioned report (as we seem to keep saying) BUT NOT YET A REALIST. What is missing in all of this reportage is any test of which of these different ideas, ingredients and processes of patrolling actually goes on to produce change.

We also see as a trifle optimistic Bennett's claim (this issue, p. 571) that he does indeed deal with context, and that we deliberately overlook the information he had provided on the two study sites chosen. To be sure, in the original report there is slightly more than met the eyes of BJC readers, namely a four-paragraph effort along the lines that, `The estate gave the impression of being slightly run down but not without hope ... there were a worrying number of packs of dogs who roamed the area without supervision ... some residents took pride in their homes and had attempted to improve the appearance of their dwellings by maintaining their gardens and decorating their windows.' (1989:28). This is mere anecdote, near caricature, a taxi-traveller's tale, and not at all the kind of thing we have in mind when we recommend attention be paid to context.

What is crucial is the salience of such community conditions for mechanism triggering' and this is not addressed. Once again, it is propositions rather than reportage which the scientific realist seeks to develop and test. What we want to know, and fail to know via Bennett, is just how these communities assisted or impeded police aspirations. Remember that programmes care `suggestions' and suggestions go down much better in some localities than others. Hope and Foster (1992), for instance, have demonstrated that the `internal' culture of an estate provides for a background level of social control and physical vulnerability which sets limits on what an initiative may achieve. The `social mix', carefully observed and monitored within their programme area, acted so that the initiative had different effects on `criminality' and `fear' from street to street, tower block to tower block. In the same way, the potential for change generated by police officers knocking on doors, introducing themselves and so forth is obviously going to be shaped by the local cultural conditions which await such actions, which will vary from apathy, hostility, guardedness, welcome, wonder, to plainjust-not-being-there.

Investigating conditioning by culture, is not a research strategy ready and waiting in the experimentalist toolbox. Communities, within that tradition, are there to be matched and scanned for unusual events. The purpose of Bennett's snapshot description of `context' in terms of crime rates and dog packs is once again the `familiarization function', there to help along the reader's intuitive notion of the `kind of place this research is about'. We are staggered at the thought of the gazetteer of travellers' tales which might be accumulated as background to Bennett's `ideal' of 50 experimental and 50 control areas (this issue, p. 571).

What we are saying here is that evaluators need to begin research with some theory of the differences within and between communities in their ability to sustain the programme theory. Cases should be systematically chosen to chip away at such hypotheses. Our suggested remedy is thus not simply the injection of a bit of qualitative evidence in the form of an ethnography of the experimental Estate, which Bennett seems to see as the critical thrust of our paper and which, save for lack of cash, he would have been quite happy to oblige. We do not simply want to swap Bennett's brief traveller's tales for richer, empathetic ones. Our key point is the one which evokes disdain, being regarded by Bennett as, `not the stuff that methodological critiques are made of' (this issue, p. 571). But yes indeed, it is `theory' which Bennett lacks. By this we do not mean background currents of thought which get dropped as soon as the real experimental work gets underway. Rather, evaluation needs to be driven by propositions of a quite specific character which guide our search for possible causal mechanisms which will be triggered in given contexts and which will produce specified outcome patterns. Evaluation research should always be informed by a range of these. Some will derive from practitioners, some from social science theory, and some from the results of previous evaluation studies. Bennett's own evaluation of contact patrols could have produced transferable lessons had it started with such explanatory objectives:

(1) What is it about contact patrols which might lead to a reduction in fear of crime in the housing estates in which they were introduced? That is, what causal mechanisms might be triggered by such patrols in the context in question?

(2) What estate conditions are significant for contact patrols potentially to trigger those causal mechanisms which generate changes in fear of crime among residents?

(3) What outcome patterns (changes in fear of crime within and between the different estates studied) are generated by which causal mechanisms acting in which contextual conditions?

In the last analysis, of course, it is for the reader to decide which of two reconstructions of Bennett's research is nearer the mark. The same audience can also be left to decide just who is being `exasperated and disdainful' (this issue, p. 572). Our interest in selecting his work for criticism was elaborated fully in our original paper (1994: 294). We have attempted to argue in the spirit of a form of realism, namely `critical realism' which Campbell was happy to have associated with his name (Shadish et al. 1991: ch. 4). This stresses that methodological progress comes from celebrating critical commentary and open learning, and avoiding the ad hominen and ad institutionem.


BENNETT, T. (1989), `Contact Patrols in Birmingham and London: An Evaluation of a Fear Reduction Strategy'. Report to the Home Office Research and Planning Unit. ----- (1990), Evaluating Neighbourhood Watch. Aldershot: Gower. ----- (1991), `The Effectiveness of a Police-Initiated Fear-Reducing Strategy', British Journal of Criminology, 31/1: 1-14. ----- (1996), `What's New in Evaluation Research? A Note on the Pawson and Tilley Article', British Journal of Criminology, this issue. CooK, T.D., and Campbell, D. T. (1979), Quasi-experimentation. Chicago: Rand McNally. Hope, T., and Foster, J. (1992), `Conflicting Forces: Changing the Dynamics of Crime and Community on a "Problem" Estate', British Journal of Criminology, 32/4: 488-504. Pawson, R., (1989), A Measure For Measures. London: Routledge. Pawson, R., and Tilley, N. (1994), `What Works in Evaluation Research?', British Journal of criminology, 34/3: 291-306. ----- (1995), `Whither (European) Evaluation Research?', Knowledge and Policy, 8/3: 20-33. ----- (1995) forthcoming), Realistic Evaluation. London: Sage. Shadish, W. R. et al.(1991), Foundations of Program Evaluation. Newbury
COPYRIGHT 1996 Oxford University Press
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 1996 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Title Annotation:response to article in this journal, p.567; Trevor Bennett's response to the authors' criticism of quasi-experimental methodology
Author:Pawson, Ray; Tilley, Nick
Publication:British Journal of Criminology
Date:Sep 22, 1996
Previous Article:What's new in evaluation research? a note on the Pawson and Tilley article.
Next Article:Crime seasonality and serious scholarship: a comment on Farrell and Pease.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters