Droppin' the gloves on junk stats: amateur analysts shame professional sports reporters by eviscerating a flawed study on hockey fighting.
Mortifyingly typical was the handling of a study released in October by three U.S. sports economists, John Heyne, Aju Fenn and Stacey Brook. The highlight, or so we were told by reporters, was a regression-analysis finding that fighting helps win NHL games. CTV and the National Post gave it respectful attention, and other outlets batted it around, quarrelling with or accepting the sexy result without discussing how it was reached. None made any serious effort to critique the paper, and most might be forced to admit that they were essentially incapable of making heads or tails of its tables. But there are dozens of amateur hockey bloggers who do have relevant training in statistics and scientific method, and they set upon the study like wolves.
It turns out the material on fighting takes up no more than about five per cent of the study. The authors' stated goal was to find out how well a multitude of variables, like faceoff wins and save percentage, correlate to winning. When you actually read the study, you find a surprise: fights aren't even one of the included variables! The actual statistical finding was that the total number of major penalties a team takes is correlated strongly to its performance, but in the article the authors make an outrageous unsupported leap from "number of majors" to "overall quantity of fighting" and hypothesize that "even though fighting results in a penalty, it is shown to be able to jumpstart a team into action and elicit better play."
Many second-hand reports in print and broadcast missed this verbal swindle completely, and all failed to ask a simple question: don't all fights involve players from both teams on the ice? Even granting the utility of major penalties as a proxy for fighting, Heyne, Fenn and Brook present nothing to warrant the causal assertion that fighting leads to wins. An equally sound interpretation of the same correlation would be that good teams pile up major penalties, not because fighting makes them better, but because their games with other good teams are more urgent, passionate and inherently violent. A member of the ghastly Coyotes really has no reason to take a swing at anybody, with the possible exception of his general manager.
The media also failed to notice the study's most glaring, suspicious asymmetry: the variables are correlated with goals against to measure defensive performance, but nowhere are they connected with goals for as a measure of offence. This alone would kill it dead in a half-decent peer review. The Post, to its credit, tried to include some specific numbers from the study's tables--but it fumbled slightly, mentioning the positive correlation between majors and overall team points (about 0.08) without subtracting the separate effect of the attendant five penalty minutes (-0.05).
In short, the whole thing was an epic mess. The "economists" behind the study must be left to their consciences. As for the sportswriters and talking heads, some will say it is absurd to expect them to be scientists or critics of science. But if that's true, it means that their nonscientific bull detectors must be all the more highly polished. And they must learn to resist the editor's universal instinct for zany, rabble-rousing news stories, ultimately based on clashing nerd-jock stereotypes. Leave number crunching to the amateurs.
|Printer friendly Cite/link Email Feedback|
|Date:||Dec 18, 2006|
|Previous Article:||Rediscovering the sacred and the taboo: when bioethicist Margaret Somerville received an honorary degree at Ryerson University last June, more than a...|
|Next Article:||The cutest little demon: raising a toddler provides some insight into the need for God's chastisements.|
|Sex and the Kinsey guy: while sexual fluidity may attract talented actors like Kinsey's Peter Sarsgaard, it seems to frighten away audiences--and...|