No Section

Danger in cherry-picking the polls

The margin of error on a subsample can be enormous in a subtle bias.

The New York Times

October 17, 2012 at 2:39AM

Even a fairly calm day in the polling, like the past couple of days, can give people opportunities to see what they want to see in the data.

The most egregious form of this is cherry-picking the three or four polling results that you like best for your candidate. The vast majority of the time, you can find a couple that are favorable for your side.

If you looked at only the three best national polls for President Obama on Monday, you would conclude that he was 3 points ahead in the national race. If you looked at only Mitt Romney's three best polls, you would say that he was ahead by 2 points instead.

Most people avoid this sort of mistake. It's just too flagrant a case of cherry-picking when 20 polls are published in a day and somebody discusses only two or three of them.

There is a more subtle form of bias, however, that a lot more of us are prone to. That bias is to look at all the data -- except for the two or three data points that you like least, which you dismiss as being "outliers."

If you're a Democrat, for example, and throw out Romney's three most favorable polls from the 10 national surveys published Monday, you can claim that Obama is ahead by 1.3 percentage points. If you're a Republican and do the same thing, dropping Obama's three best polls, you will have Romney ahead by 1 point instead.

That is not quite as biased as cherry-picking the best results -- but it gets you halfway there, and it is easier to rationalize. There is something that can be critiqued about almost every poll: the methodology, or the demographics, or the sample size, or the pollster's history, or something else. Often, these critiques have some truth in them. Not all polls are as methodologically sound as others. But frequently people come up with reasons to avoid looking at the polls they don't like -- while giving a pass to those they do.

Likewise, people sometimes make too much of demographic or geographic subsamples within a poll that make their candidate look good. The most recent Washington Post/ABC poll had Obama performing better in what it termed swing states than in the country as a whole; a recent Gallup poll showed just the opposite.

These subsamples of swing-state voters from national polls are largely useless. A typical national poll may interview 1,000 people, of which perhaps 250 or 300 will live in swing states.

The margin of error on a 250- or 300-person subsample is enormous: about plus or minus 6 percentage points. In contrast, the state polls that are released on a given day include, combined, thousands of interviews. There is just no reason to focus on what 250 or 300 people say when you can look at what 2,500 or 3,000 do instead.

Historically, the second presidential debate has moved the numbers by about 2.5 percentage points in one direction or another. If that gain were in Obama's favor, he would re-establish a clear enough lead that there would be little doubt about who was ahead. Another shift toward Romney, however, and he would probably lead in enough polls to show him on a path to 270 electoral votes.