Erring on the margin of error.

1. Introduction

As most teachers of probability and statistics know, one of the most difficult concepts to convey to students is that of sampling error. Yet with the proliferation of the reporting of the results of public-opinion polls in the news media, students and the general public alike are exposed to this concept almost on a daily basis. In fact, when the authors recently accessed the Dow-Jones Interactive News Library (1) and typed in the words "public opinion poll," we registered nearly 60,000 "hits" in major newspapers and newswires for the period 1990 to the present. Often accompanying the discussion of the poll results is a statement describing the accuracy of the poll's estimates, which ordinarily reads something like, "The margin of error is 3 percentage points with a 95% level of confidence." (2) Because to many readers the meaning of this statement is fuzzy, the article sometimes attempts to clarify what the margin of error indicates about the poll's accuracy. However, from our experience, the attempted explanation is often completely in error--sometimes outrageously so.

In this article, we first briefly explain the correct way of interpreting the margin of error, which currently seems to be the fashionable way to explain in the media what statisticians routinely call sampling error. The margin of error concept is also gradually making its way into the professional economics literature (see Borenstein and Rose 1994; Brezis 1995) and into business/economics statistics textbooks as well (e.g., Anderson, Sweeney, and Williams 2002; Bowerman and O'Connell 2003). Next, we present some typical misinterpretations of the margin of error drawn from the news media that will prove interesting to both teachers and students and useful as classroom examples. Finally, we suggest some ways of using the media misinterpretations to teach and reinforce the correct interpretation of the margin of error (as well as of confidence intervals).

2. Interpretations and Misinterpretations

Suppose that an opinion poll taken of 1000 people (assume the sample has been selected randomly) finds that 60% of those sampled believe that economists are extremely boring. Suppose also that the margin of error (with 95% confidence) is reported to be 3 percentage points. The correct interpretation of this margin of error is that if repeated samples of size 1000 were to be taken, approximately (3) 95% of the time the sample proportions (p) would lie within 3 percentage points (0.03) of the true population proportion (p)--the proportion of all people who believe that economists are extremely boring. In other words,

Prob([absolute value of p - p] [less than or equal to] 0.03) = 0.95.

Conversely, only 5% of the time would the sample proportions be more than 0.03 away from the population proportion. But what kinds of interpretations are often given for the margin of error by the news media? Here are several examples:

* From a survey on the incidence of AIDS among clerics, an article described the poll's 3.5 percentage-point margin of error to mean that "if the same poll were conducted 100 times, 95 percent of those times the results would be no more than 3.5 percentage points higher or lower than the results of this poll" (Thomas 2000, p. A18) (italics added).

* In a telephone survey of Ohio adults on how trustworthy they believed real estate agents to be, the Columbus Dispatch reported that the survey had a "2 to 3 point margin of error and a 95 percent confidence level, meaning the results have a 95 percent probability of being true no matter the size of the polling group" (Columbus Dispatch 1998, p. 8J).

* A telephone poll of 1014 Texas adults on the death penalty reported a margin of error of 3 percentage points, which was interpreted as "each response can vary that much in either direction" (Hoppe 1998, p. 39A).

* In a Newsday report attempting to explain what a "1.8 rating-point plus-or-minus margin of error" meant for a Nielsen rating of news shows, the following explanation was given: "That means, among the many possible permutations of those figures, ... that last week's ratings could have been what they were said to be the week before, or that the week before's [sic] could have been what last week's ratings were said to be." (Kubasik 1990, p. 9). (We've read this explanation a number of times and are still not sure what it means.)

* In a Jakarta Post article (misinterpretations of the margin of error are not restricted to U.S. newspapers) concerning the results of a survey of 1000 people's opinions about human rights violations, the survey's claim of a 3.1% margin of error with a "reliability level of 95 percent" was interpreted in the following way: "This means that if a similar survey is conducted again ... there is a 95 percent chance that it will be of the same result--with only a 3.1 chance of error" (Jakarta Post 1998, n.p.).

* In a poll taken during the 2000 presidential race showing that Al Gore had a 6-point lead over George Bush, a margin of error of plus or minus 4 percentage points was said to mean that "there is a 95 percent probability the results of the poll are within 4 percentage points higher or lower than the finding" (Associated Press Newswires 2000).

* In a consumer survey regarding customer service issues that were the subject of labor negotiations, a telephone survey of 2400 customers of US WEST reported a margin of error of "plus or minus 2.0 percentage points." The interpretation given was that "if the survey had been conducted 20 times using the same approach, the results would be the same 19 out of 20 times" (PR Newswire 1998).

* In a poll of 729 voters in Arlington, Texas, regarding their positions concerning a proposed sales tax increase, the 3.7 percentage point margin of error was explained in this way: "If the poll were conducted in the same manner 100 times, in 95 of those instances the reported results would be within 3.7 percentage points of the results that would be obtained by interviewing every registered voter in Arlington" (Phillips 1991). (This interpretation errs by implying that in exactly 95 cases out of 100, the reported results would be within 3.7 percentage points of the population results.)

3. Some "Authoritative" Source Explanations

Interestingly, further search also revealed a number of authoritative sources that attempt to explain the margin of error but that also do so incorrectly.

* On a Public Agenda Online Web site, a posting entitled "A Guide to Sample Size and Margin of Error" has the following explanation for a "3 percent margin of error": "That means that if you asked a question from this poll 100 times, 95 of those times the results would be within 3 percentage points of the original answer" (italics added). The explanation continues: "For example, if 50 percent of a sample of 1,000 randomly selected Americans said they favor recycling laws, in 95 cases out of 100, 50 percent of the entire population in the U.S. would also have given the same response had they been asked, give or take 3 percentage points" (Public Agenda Online).

* In a Washington Post article (which ironically criticizes Marilyn vos Savant for a slip-up regarding the margin of error in her "Ask Marilyn" column), Richard Morin (1998, p. C5) quotes a former standards chair of the American Association for Public Opinion Research as the source for his explanation of a 3 percentage point margin of error: "What that plus or minus 3 percentage points means is that if the same survey were conducted under the same conditions 100 times, about 95 of the resulting proportions should be within 3 percentage points of the one that you now have" (italics added).

* The most surprising discovery came when we checked the home page of the Gallup Organization, which contains a link to a posting entitled "How Polls Are Conducted: Your Frequently Asked Questions Answered." (4) The posting provided the following explanation for a margin of error of plus or minus 3 percentage points:

   Thus, if we find in a given poll that President Clinton's rating is
   50%, the margin of error indicates that the true rating is very
   likely to be between 53% and 47%.... To be more specific, the laws
   of probability say that if we were to conduct the same survey 100
   times, asking people in each survey to rate the job Bill Clinton is
   doing as president, in 95 out of those 100 polls, we would find his
   rating to be between 47% and 53% (Gallup Organization).

In short, the Gallup explanation--indeed, all three authoritative-source explanations above--commit the same error. (5) Instead of saying that in 95 out of 100 polls we would find the president's ratings to be within 3 percentage points of the population rating, the interpretation states that in 95 out of 100 polls we would find his ratings to be within 3 percentage points of the rating found in this sample. (6)

4. Some Classroom Applications

We strongly believe that examples drawn from the news media can greatly enhance and facilitate the effective teaching of statistics (see Becker 1998). We suggest below several ways that we have done this with margin-of-error misinterpretations.

(i) Instructors can present several examples of misinterpretations of the margin of error drawn from various newspaper articles along with at least one correct interpretation. In basic statistics classes, one of the authors projects four or five margin-of-error interpretations on a screen. (The ones we have presented here work particularly well, especially the humorous ones.) Students are then called upon to see if they can recognize which of the interpretations are in error, explain why they are in error, and then correct them accordingly.

(ii) Students can also be challenged to find new examples of margin-of-error misinterpretations while surfing the Interact or reading the newspaper. In fact, several of the examples presented here were provided to the authors by students. Students can also be asked whether they can discover other authoritative Web sources, such as the Gallup Web site, that provide incorrect explanations of the margin of error.

(iii) It is also instructive to ask students to classify common margin-of-error misinterpretations into various categories. From our experience, the most frequent errors (both in the popular press and in class) are those that explain how far away similar sample results would be from those reported in the sample actually taken, rather than how far away they would be from the population value. But as our earlier examples show, there are several other categories of commonly encountered errors, such as:

(a) Interpreting the margin of error to mean that, if 100 samples were to be taken, the results would be within the stated margin of error in exactly 95 of those 100 samples;

(b) Interpreting the margin of error to mean that, if the survey were to be repeated, in 95 percent of the cases the results would be the same as that found in the present sample.

And of course, there are those interpretations that are merely badly "botched."

(iv) Finally, from our experience it is instructive to ask students to convert margin-of-error estimates into confidence intervals. It is rare when articles in the popular press present confidence interval estimates in reporting poll results. However, just as with the margin of error, students often have difficulty interpreting confidence intervals correctly. The most common error that we have encountered in class is the explanation that "95% of the time the confidence interval will include the sample value." But other types of misinterpretation also occur. For example, in a study of a school district's student attendance record, a 90% confidence interval for the mean number of school days missed per year was reported as 3-9 days. The interval was misinterpreted as indicating that if repeated random samples were to be taken, 90% of them would produce sample means of from 3 to 9 days. (7)

Nor are authoritative sources free from making errors interpreting confidence intervals. For example, according to a manual published by the American College of Physicians explaining how to report statistics in medicine, a 95% confidence interval for estimating the mean drug-induced reduction in internal eye pressure was calculated to be 5.5 to 14.5 mm Hg. What this means, according to the manual, is that "if the study were to be repeated 100 times with a sample of the same size and characteristics, 95 of these trials would probably show a mean decrease of pressure between 5.5 and 14.5 mm Hg" (Lang and Secic 1997, p. 28).

In short, we have found that involving students in the detection and correction of margin-of-error misinterpretations is an effective way of teaching this often misunderstood concept. In addition, using real-world examples drawn from the popular press and the Internet can be both engaging and enjoyable. (8)

The authors wish to thank Jon Innes for suggestions and for several of the examples presented here and Gauri Karve for research assistance. The views expressed in this material are those of the authors and do not reflect the official policy or position of the U.S. Government, the U.S. Department of Defense, or the U.S. Department of the Air Force.

(1) The Dow-Jones Interactive News Library (now called Factiva) is an electronic database containing about 6000 newspapers, newswires, and magazines.

(2) The origin of the term "margin of error" is unclear but may have been first coined by journalists to describe the degree of sampling error. In any case, the term is now becoming increasingly common.

(3) There are two reasons why the qualification "approximately" must be used. First, the magnitude of the estimated margin of error varies from sample to sample because the true standard error of the sampling distribution of the sample proportion is unknown. Second, the normal distribution is only an approximation to the true sampling distribution of the sample proportion.

(4) The explanation given in the Gallup posting is taken from Newport, Saad, and Moore (1997).

(5) In response to an e-mail query by the authors, the Gallup Organization replied that it routinely directs inquiries concerning the interpretation of polling results to the "Frequently Asked Questions" link.

(6) In section 1, we noted that the margin-of-error concept has recently begun to appear in a number of basic business/economics statistics textbooks. For example, in addition to those already mentioned, see Becker (1995); Shiffler and Adams (1995); Siegel (1997); Harnett and Horrell (1998); Ovedovitz (2001); and Moore and McCabe (2003). In some cases, very little explanation is provided for the meaning of the margin of error, aside from noting that its magnitude is one-half that of the corresponding confidence interval.

(7) This example of a misinterpretation is presented in Pfaffenberger and Patterson (1987, p. 401).

(8) One of the anonymous referees has suggested that the confusion in correctly interpreting confidence intervals could be avoided by adopting a Bayesian approach. According to this approach, a margin of error of 3 percentage points would signify that there is a "probability" of 95% that the population value lies within 3 percentage points of the sample estimate. As the referee states, "This is how students want to interpret this result, and indeed most instructors would probably confess that they think about it this way also, except when they are teaching it!" We thank the referee for this observation, although the wide variety of misinterpretations that we have observed in press examples and encountered in the classroom make us somewhat less optimistic.

References

Anderson, David, Dennis Sweeney, and Thomas Williams. 2002. Statistics for business and economics. 8th edition. Cincinnati: South-Western/Thomson Learning.

Associated Press Newswires. 2000. Gore keeps edge over Bush in poll this month. 27 March.

Becket, William. 1995. Statistics for business and economics. Cincinnati: South-Western/Thomson Learning.

Becker, William. 1998. Engaging students in quantitative analysis with short case examples from the academic and popular press. American Economic Review 88:480-91.

Borenstein, Severin, and Nancy L. Rose. 1994. Competition and price dispersion in the U.S. airline industry. Journal of Political Economy 102:653-83.

Bowerman, Bruce, and Richard O'Connell. 2003. Business statistics in practice. 3rd edition. New York: McGraw-Hill, Irwin.

Brezis, Elise S. 1995. Foreign capital flows in the century of Britain's industrial revolution: New estimates, controlled conjectures. Economic History Review, New Series 48:46-67.

Columbus Dispatch. 1998. Real estate agents get high marks. 9 August, p. 8J.

Gallup Organization. Frequently asked questions: How polls are conducted. Accessed 2 March 2003. Available http://www. gallup.com/help/FAQs/poll1.sp.

Harnett, Donald, and James Horrell. 1998. Data, statistics, and decision models with Excel. New York: John Wiley & Sons, Inc.

Hoppe, Christy. 1998. Poll shows Texans favor adult justice for violent youth. Dallas Morning News, 21 June, p. 39A. Jakarta Post. 1998. Survey reveals ABRI's poor public image. 29 September, n.p.

Kubasik, Ben. 1990. TV spots. Newsday, 23 August, p. 9.

Lang, Thomas, and Michelle Secic. 1997. How to report statistics in medicine. Philadelphia: American College of Physicians.

Moore, David, and George McCabe. 2003. Introduction to the practice of statistics. 4th edition. New York: W.H. Freeman and Company.

Morin, Richard, 1998. Unconventional wisdom. Washington Post. 14 June, p. C5.

Newport, Frank, Lydia Saad, and David Moore. 1997. Where America stands. New York: John Wiley & Sons.

Ovedovitz, Albert. 2001. Business statistics in brief. Cincinnati: South-Western/Thomson Learning.

Pfaffenberger, Roger, and James Patterson. 1987 Statistical methods. Homewood, IL: Richard D. Irwin.

Phillips, Cheryl. 1991, Analysts say poll indicates tight tax vote. Fort Worth Star-Telegram, 13 January.

PR Newswire. 1998. Customer survey shows support across region for key US WEST service initiatives in labor contract negotiations. 25 August.

Public Agenda Online. Best estimates: A guide to sample size and margin of error. Accessed 14 November 2001. Available http:www.publicagenda.org/aboutpubopinion/aboutpubop4.htm.

Shiffler, Ronald, and Arthur Adams. 1995. Introductory business statistics with computer applications. 2nd edition. Wadsworth, CA: Duxbury.

Siegel, Andrew. 1997. Practical business statistics. 3rd edition, Homewood, IL: Irwin.

Thomas, Judy L. 2000. Priests speak out in national survey. Kansas City Star. 30 January, p. A18.

Robert J. Thornton * and Jennifer A. Thornton ([dagger])

* Department of Economics, Lehigh University, 621 Taylor Street, Bethlehem, PA 18015. USA; E-mail [e-mail]; corresponding author.

([dagger]) Clinical Investigation Facility, David Grant USAF Medical Center, 101 Bodin Circle, Travis AFB. CA 94535, USA; E-mail [e-mail].

Received May 2003; accepted August 2003.