Comment on Sparrow and Curtice and Kellner.
I remember back in the early 1980s, market researchers got very concerned about the adequacy of telephone sampling as a general population research tool. In particular, the performance of the opinion polls was the catalyst for major debate based on the fear that telephone polling ran the risk of bringing the industry into disrepute by producing poor estimates of voting intention. A substantial body of potential Labour supporters could not be reached by phone at the time, though the size of that group was diminishing year by year as telephone penetration rose.
In 1987 the Market Research Development Fund (MRDF) conducted a controlled experiment during the election campaign to establish the facts once and for all (Husbands 1987). I was Chairman of MRDF at the time and played an active part in directing the research. The findings were fascinating. The telephone sample was indeed somewhat biased as expected when analysed unweighted. However, the application of increasing levels of weighting reduced the bias until, at the final step, the use of 'superweighting', developed by Professor Bill Miller at Strathclyde University, provided a perfect correction. The unpublished MRDF telephone sample, interviewed in the last few days of the campaign and superweighted, proved to be a more accurate predictor of the election outcome than any of the published polls!
Telephone interviewing passed the test and came of age, but not without clever massaging of the data to counteract the inevitable biases in the raw data.
When I was asked to read the Sparrow and Curtice and the Kellner papers, I anticipated that the debate would be repeated 20 years on, but this time about the dangers of internet polling. To some extent my expectations were met. Sparrow and Curtice clearly represent the camp that fears the potential damage that internet polls could do. However, from the very outset, they concede that the record of the internet-based YouGov polls at various recent elections has been excellent. Kellner, the founder of YouGov and defender of internet polling, is even more bullish in his exposition of 'the proof of the pudding' argument. Neither paper has anything but praise for the way YouGov has performed at elections. When judged against the yardstick of how close their final estimates of share of vote were to the actual distribution of votes cast in the election, YouGov has done very well.
As a precursor to setting out his own methodology, Kellner presents a strong critique of the recent performance of traditional polls and the shortcomings of their methods. Samples achieved using the internet may be unrepresentative, he admits, but counters that this is also true of fast turnaround quota sample face-to-face polls, or low response/high refusal telephone polls. He recognises that ICM, managed by Sparrow, has performed better than most pollsters, but points out that ICM uses new ways of weighting and adjusting raw data that have been controversial.
Kellner almost predicts the nature of the likely attack on his approach in the second paragraph of his paper: 'Were the record of traditional companies perfect ... the accuracy of online surveys could be judged by how close their figures came to those obtained from face to face or telephone interviews.'
He also establishes in this same sentence one of the main thrusts of his defence. From Kellner's position, internet polls and other polls all face the same problem: how to achieve a reliable estimate of voting behaviour based on substantially flawed sampling strategies. The technical debate, it turns out, is not about interviewing methodology, nor sampling reliability. The technical debate is really about weighting strategies, or modelling the data to correct for the inevitable shortcomings that any of the methods bring to the results. It is the search for the modern equivalent of Miller's 'superweighting' to be applied in the early twenty-first century to internet polls and, if Kellner's case is accepted, also to traditional polls.
The case presented by Sparrow and Curtice is somewhat different. They do not set out to completely rubbish e-methods. They accept that internet polls have done well at election prediction. However, their aim is to establish reasonable doubt. Using a well constructed traditional poll they are able to explore the make-up of the sub-group who have internet access, and a further sub-group who claim they would be prepared to join an internet panel--a surrogate for the universe open to YouGov panel. They show that these sub-groups are not representative and that normal weighting strategies do not allow the answers of the total representative sample to be replicated by adjusting the e-panel potentials.
They then proceed to carry out e-interviews with a sample of those willing to join an internet panel. On a wide range of political issues they demonstrate that sometimes the surrogate e-panel produces results in line with the traditional poll and sometimes it doesn't. This is a fair step on the road to establishing reasonable doubt.
Kellner counters on two fronts. First, Sparrow and Curtice do not replicate his special methodology so he is not surprised. Second, why hold up the traditional poll as the yardstick? Why should it be taken as the 'right answer' when judging the e-measures?
To make sure that the special expertise attack remains the most powerful, Kellner repeats the attitude measures from Sparrow and Curtice on his YouGov panel and produces results which, in nearly every case, are similar to the traditional poll measures of the Sparrow and Curtice full sample. This is a good attempt to minimise the reasonable doubt.
Sparrow and Curtice close their paper with a final attempt to establish reasonable doubt. They compare the results published by YouGov and some of the traditional polls on a specific issue over a period of time. For example, they show MORI and ICM very much in line with their measures of public opinion on the Euro from early 2002 to mid-2003. Over the same period the YouGov poll moves from being much less anti the Euro to a position in the middle of the MORI--ICM measures. 'Quite why the discrepancy suddenly disappeared is not clear,' they conclude, implying that YouGov had done something to adjust its measures and, by implication, that its earlier measures were inaccurate.
The case for reasonable doubt is reopened. The YouGov poll methodology might work brilliantly for election forecasting but what guarantee is there that it will work for other issues, or work equally well for all political issues? The sampling/weighting/data modelling strategy that is appropriate for election prediction may not be optimum for many other issues, nor is it necessarily constant over time.
So I ask you, the Research Jury, to read the evidence and ask yourself if you think there is reasonable doubt established. In my heart I think there is (or at least there should be). However, from my rational side I am left with a quote attributed to Ronald Reagan in the first draft of Kellner's paper, a quote edited out for the final version: 'We know how it works in practice. The challenge is to make it work in theory.' My research to verify this Reagan quote led me to discover that 'In theory, theory and practice are the same thing, but in practice they're not' and 'The difference between theory and practice is always greater in practice than in theory.'
The jury is out. Over to you.
[C] 2004 The Market Research Society
Husbands, C. (1987) The telephone study of voting intentions in the June 1987 General Election, Journal of the Market Research Society, 29, 4, pp. 405-418.
|Printer friendly Cite/link Email Feedback|
|Publication:||International Journal of Market Research|
|Date:||Mar 22, 2004|
|Previous Article:||Measuring the attitudes of the general public via internet polls: an evaluation.|
|Next Article:||Public attitudes to dependency and the welfare state.|