MyDD :: polling matters

polling matters

by Jerome Armstrong, Sat Sep 11, 2004 at 03:23:44 PM EST

Mark Schulman, from Schulman, Ronca & Bucucvalas, Inc.(SRBI), which has done polling for Newsweek for many years, responds to Alan Reifman's discussion of poll weighting.

Alan, your web site presents a very insightful discussion of the weighting issue, which often accounts for major differences among poll findings. I'm on my way out of the country at the moment, so the following are just some quick observations. As you know, we do the Time Magazine Poll, and I've received a number of queries over the past few weeks about this issue. It not only needs to be addressed, but weighting by party ID can result in serious distortion of the horserace numbers. Here's why:

1. As an observer of party identification tallies day after day on our election surveys, it's clear that we're not measuring a constant factor. It varies day by day, week by week.

2. Why does it vary? Most polls place the party ID question near the end of the questionnaire, so that it does not interact or contaminate the horserace measure and any other head-to-head candidate comparisons. The horserace always takes priority, since that's the topline number we report. As a result, respondents may tend to bring their party ID in line with their partisan choice, particularly after having gone through an extensive battery of election items. It's simply "cognitive consistency." Hence, a Bush surge, for example, might elevate the number of voters later in the survey identifying themselves as Republicans.

3. Since party ID is a "variable" and not an enduring constant, as is age or gender, it varies!

4. Voting behavior literature (Campbell, Converse, Miller and Stokes, The American Voter, for example), used to posit party ID as anchoring partisan choice, as if it were a constant. It's likely that party ID was a more enduring "constant" in the 1950's, but, that was then, and this is now! Voters are just not as tied to party as in the past. Let's get over this likely out of date notion that party ID is a constant that anchors the vote. The causal arrows here are unclear, that is, what influences what.

5. Hence, weighting by party ID, and its party ID, not party registration, can seriously distort the horserace data. Weighting by party ID would damp down the Bush surge over the past few weeks. Yes, there may be some "at home" selection bias when we interview during party convention periods. However, not all that many folks watch the conventions and the networks provide little convention coverage.

Finally, my choice, and the choice of most the major media polls, is to weight by factors that we know are real, such as age, gender, region, education, number of adults in household, number of voice phone lines, etc. While you can argue about the reliability of Census data, I'll place my bets with the Census rather than party ID.

David Moore has a good discussion of this issue as well on the Gallup web site.

That's the short version of my views. I really do believe that we need to put this issue to rest and stop pretending that there's legitimacy to party ID weighting. I look forward to further comment!

It can't be ignored that media vehicles such as Newsweek and Time live for things like "the Bush surge", as Mark puts it. They want a horserace that makes news & sells magazines. Regardless of the argument, SRBI's treatment of party ID as being somewhat fluid (and therefore not a factor to weight for) satisfys that need. Rasmussen 2004, Zogby, and SUSA might not be as flashy in their ups and downs, but their accuracy is just fine, or better, and arguably due to their Party ID weighting. I'll put the responses in the extended entry.

Tags: Demographics (all tags)

Permalink :: 6 Comments :: Post a Comment

Alan Abramowitz responded: (none / 0)

Mark Schulman writes that, "voting behavior literature (Campbell, Converse, Miller and Stokes, The American Voter, for example), used to posit party ID as anchoring partisan choice, as if it were a constant. It's likely that party ID was a more enduring "constant" in the 1950's, but, that was then, and this is now! Voters are just not as tied to party as in the past. Let's get over this likely out of date notion that party ID is a constant that anchors the vote. The causal arrows here are unclear, that is, what influences what."

This is actually a complete misreading of the current political science view of party identification. See the recent work of Larry Bartels and Green, Palmquist, and Schickler's Partisan Hearts and Minds (Yale University Press). The idea that party identification fluctuates widely during the course of a campaign is influenced by short term factors such as current vote choice is inconsistent with this research. I can understand the problems with using a party id weight because it is not a "constant" in the same way that gender, race, and age are, but neither is it plausible that it will vary by as much as 10 percentage points in the course of a few weeks. Look at the national exit polls over the past several presidential elections--there is relatively little variation in the distribution of party identification and there is a consistent Democratic advantage of a few percentage points. This Democratic advantage has diminished considerably over time, but that's over a long time, not over a few weeks.

Aside from the issue of weighting, another reason for the fluctuations over time in both partisanship and candidate preference is the use of likely voter screens, especially when applied several weeks or more before an election. Bob Erikson of Columbia University has written an excellent analysis of the consequences of this for the 2000 Gallup tracking poll which produced huge variations in candidate preference over the course of just a few days during the campaign. His conclusions is that most of the variation in the tracking poll results was due to the likely voter screen rather than change in the preferences of registered voters. My own examination of the most recent Gallup Poll indicates that, based on the candidate preferences of Democrats, Republicans, and independents that they report in their "dissecting the vote" analysis, their LV sample must have had a Republican advantage in party id of about 10 percentage points. In my view, this is quite unrealistic as a projection of what the 2004 electorate will look like and it has a dramatic impact on the trial heat results since the correlation between party id and candidate preference is extremely strong. The resulting 7 point discrepancy between RVs and LVs is not plausible. Moreover, the LV result is what totally dominates the news coverage and shapes perceptions of the state of the presidential race in a way that I believe is quite misleading. Psychologically, there is a huge difference between reporting a 1 percent Bush lead and a 7 percent Bush lead.

by Jerome Armstrong on Sat Sep 11, 2004 at 03:28:35 PM EST

My response: (none / 0)

I would propose that if SRBI and others insist on reporting the horserace results with an unweighted party ID (the fluidity of which is not backed up in actual results), that they also report the "likely results", by factoring in the respondants party ID.

Party affiliation just doesn't, in the end, jump around as much as the unweighted for party ID horserace numbers protend. The election returns have a predictable party ID, as Zogby's exit polling numbers from the previous 3 elections show, and as the multi-year Harris Interactive party affiliation poll shows:
http://www.harrisinteractive.com/harris_poll/index.asp?PID=444

This is not to argue that party affiliation remains as strong as it once was. As the Harris polling above shows, it's becoming less so each year. But as a factor to rely on in predicting results, it's still predictable enough to not be ignored. Maybe Scott Rasmussen (given his unremarkable 2000 polling experience) can speak to the one-sided argument of pretending that there's legitimacy to not weighting party ID.

by Jerome Armstrong on Sat Sep 11, 2004 at 03:40:45 PM EST

Mark Schulman responds: (none / 0)

Again, over time, there will be some consistency in the aggregate party id levels. But, that's over time. A candidate surge will very likely produce a short-term increase, even if small, in the percentage of respondents "identifying" with the surging candidate's party. Again, we ask the party id question at the conclusion of the survey, after the respondent has been funneled though 12-15 minutes of candidate questions. Do you really think that there's no tendency toward cognitive consistency?

Finally, while aggregate party ID levels remian the same, that does not mean that individual voters themselves at the margins are consistent over time.

One way to get some insight into this issue is to conduct an experimental design, where you insert the party id item upfront, before candidate choice. I don't have this luxury because party id would likely contaminate the horserace question.

I'm not thinking of myself as a "combatant" here. My believe is that public research should be an open book, not a black box. Therefore, I find this exchange quite valuable. Thank you all.

by Jerome Armstrong on Sat Sep 11, 2004 at 03:42:09 PM EST

Which causes which? (none / 0)

Mark Schulman says "A candidate surge will very likely produce a short-term increase, even if small, in the percentage of respondents "identifying" with the surging candidate's party."

Isn't it just as plausible to believe that the opposite is true? An increase in the percentage of respondents "identifying" with one party in a particular poll will very likely produce a short-term "surge" in support for that party's candidate? This seems especially likely to happen during party convention weeks.

I believe that party ID is not as solid as age, gender or other demographics, but it's also by far the best predictor of candidate support. It's too important not to use as a weighting variable.

by EvanstonDem on Sat Sep 11, 2004 at 11:27:49 PM EST
[ Parent ]

This seems a bit odd. (none / 0)

It almost seems that he's come up with a term "cognitive consistency" to rationalize his conclusion that party id numbers aren't more fixed than has been pretty much assumed for many decades. This is very strange, especially when considering the drastic difference between the RV and LV outcomes.
It seems odd that he would claim that fixed party ID is essentially quaint, when at the same time it seems the voting public are more polarized than in a generation. And registered voters who aren't independents seem extremely partisan.

This is very weird.

phat

by phatass on Sat Sep 11, 2004 at 05:23:59 PM EST

Re: polling matters (none / 0)

Ok, we may be able to rationalize why correcting for party ID is not done. But please tell me why the news weeklies over sample the "red states" in their polls. From looking at Newsweek and Time data it appears to me that their party ID skewing is a direct result of their geographical skewing. Is it that people in "red" states answer the phone and talk to pollsters more readily than in "blue" or "purple" ones? Geography is invariant and they should take only a set number of respondents per area code.

by ineedalife on Sun Sep 12, 2004 at 08:53:10 PM EST

You are not logged in.

In order to post a comment, you must be logged in. If you have a member account, please log in to comment.

If not, you can make an account right here. It's quick and free.