Race and cell phones challenge pollsters.
Polling in the 2008 presidential elections faced two distinct challenges: the substantial growth in the share of the electorate that relies primarily or exclusively on cell phones, and having an African American as one of the major candidates. Both had the potential to skew the results.
Since the 1970s, national polls have used telephone interviewing with all residential land line numbers, listed and unlisted, as the source from which samples were drawn. But by 2008, almost twenty percent of the voting age population relied solely on cell phones. If cell phone numbers were not included in the sample, how might that affect polling accuracy about candidate preference?
In the past, polls estimating support for a black candidate had sometimes erred on the high side. It became known as the "Bradley effect" since it first appeared in Tom Bradley's race for governor in California. The suspected culprits were white respondents who gave socially desirable responses (I'll vote for my party's candidate even if he or she is black) but, in the voting booth, went the other way. It's called "covert racism."
For candidate preferences, that standard definition for polling accuracy is the difference between the actual vote margin between the two candidates (Barack Obama and John McCain in this case) and the margin for the last poll conducted prior to the election. The National Council on Public Polls tallied the results for the 17 most prominent polls in 2008 and the polling industry stalwarts were much relieved. On average, the polls were off only 0.8 percent from the final result, about the same as 2004 when the cell phone share was minimal and both candidates were white.
Did it matter whether the polls sampled only from land lines or if they also included cell phones? Superficially, no. Both the 11 land-line-only surveys and the six with cell phone supplements had the same 0.8 percent gap. Since the cell-phone-only segment tilts considerably towards younger voters, the land line polls adjusted through weighting. They assumed that, for the same demographic, respondents would have the same preference independent of whether they use a land line or a cell phone.
But an analysis by Scott Keeter and his colleagues at the Pew Center for People and the Press reveals a more nuanced pattern. In a paper delivered at the American Association for Public Opinion Research Annual Meeting in May, they reanalyzed the results from six sets of Pew surveys done during the two months prior to the 2008 election. For each set, Pew conducted two parallel surveys: one land line only and one land line with a cell phone supplement. In four of the six sets, Obama's margin over McCain was higher (typically by about one percentage point) on the survey that included cell phones than it was on the survey that only used land lines. That is not enough to affect significantly the overall poll accuracy but it does suggest, for the first time in presidential polling, that cell-phone-only respondents, even when adjusting for demographic factors, might have slightly different political preferences. Using the same set of polls, the Pew researchers probed for two possible sources of covert racism: "reluctance by racially conservative poll respondents to say that they intended to vote against the black candidate" and "greater resistance among racially conservative voters to be interviewed."
To check out the first possibility, they divided the white non-Hispanic sample into two groups: those interviewed by whites and those interviewed by blacks. Controlling for other factors such as political party identification, there was no significant difference in the response pattern. The Obama-McCain split was essentially the same in both segments.
To probe the second source of bias, they interviewed a "hard-to-reach" sample of 1,000 respondents who had either refused to participate in an earlier Pew survey or had not been answered after five attempts. Simultaneously, they asked the same questions on a normal Pew survey. Both surveys were done in August 2008 and both had similar McCain, Obama shares.
Political polling continues to adjust to a changing set of circumstances but, for now, all appears relatively well.
Terry Jones is professor of political science at UM-St. Louis