As Twitchy reported Wednesday, the Public Policy Polling Company, a firm that conducts polls for Democrats, has come under fire from FiveThirtyEight editor Nate Silver and others for withholding the results of some of its polls.
Yesterday PPP came in for a second round of thrashing after Nate Cohn of The New Republic raised serious questions about PPP’s fly-by-the-pants approach to polling:
After examining PPP’s polls from 2012 and conducting a lengthy exchange with PPP’s director, I’ve found that PPP withheld controversial elements of its methodology, to the extent it even has one, and treated its data inconsistently. The racial composition of PPP’s surveys was informed by whether respondents voted for Obama or John McCain in 2008, even though it wasn’t stated in its methodology. PPP then deleted the question from detailed releases to avoid criticism. Throughout its seemingly successful run, PPP used amateurish weighting techniques that distorted its samples—embracing a unique, ad hoc philosophy that, time and time again, seemed to save PPP from producing outlying results. The end result is unscientific and unsettling.
…
PPP’s opacity and flexibility goes too far. In employing amateurish weighting techniques, withholding controversial methodological details, or deleting questions to avoid scrutiny, the firm does not inspire trust. We need pollsters taking representative samples with a rigorous and dependable methodology. Unfortunately, that’s not PPP.
Once again, Silver took to Twitter to blast PPP:
TNR's @nate_cohn (no relation!) raises some further great questions about @ppppolls methodology: http://t.co/Sw4BNfJock
— Nate Silver (@NateSilver538) September 12, 2013
The main problem with @ppppolls is that their approach to polling is extremely ad hoc. (1/5)
— Nate Silver (@NateSilver538) September 12, 2013
Ultimately, that ad-hockery stems from a lack of appreciation/understanding for the statistical fundamentals behind polling. (2/5)
— Nate Silver (@NateSilver538) September 12, 2013
.@ppppolls proudly endorse the idea of using of "gut feeling" in conducting their polls. http://t.co/eAwh2sglcc (3/5)
— Nate Silver (@NateSilver538) September 12, 2013
But "gut feeling" is often used to excuse all sorts of conscious and unconscious biases. (4/5)
— Nate Silver (@NateSilver538) September 12, 2013
Statistical methods certainly do require *judgment*. But good judgment is based on evaluating PROCESS… not RESULTS. (5/5)
— Nate Silver (@NateSilver538) September 12, 2013
.@ppppolls: How well do your methods work when there aren't other polls in the field to "gut check" against? https://t.co/mN5mJW1MTD
— Nate Silver (@NateSilver538) September 12, 2013
.@ppppolls posted their full email exchange with @Nate_Cohn — raises even more questions than Nate's post itself http://t.co/58I8StYeCY
— Nate Silver (@NateSilver538) September 12, 2013
Most astounding is how @ppppolls admits that fear of being trolled by Republicans is a big factor in how they do their demographic weighting
— Nate Silver (@NateSilver538) September 12, 2013
Silver will continue to incorporate PPP’s questionable polls in its much-heralded “poll of polls.” But don’t worry because he is going to come up with a weighting method that is “punative” to PPP and other pollsters using sketchy methodologies.
@DrewLinzer @ppppolls: No but I will try to come up with a weighting method that is more punative to pollsters who calibrate/herd off others
— Nate Silver (@NateSilver538) September 12, 2013
In other words: Garbage in, garbage out.
@fivethirtyeight @ppppolls That's good. I trust it will be based on statistical fundamentals rather than results, and not extremely ad hoc.
— Drew Linzer (@DrewLinzer) September 12, 2013
Ouch.
Looking forward to @Nate_Cohn's equally thorough takedown of the entire polling industry.
— Drew Linzer (@DrewLinzer) September 12, 2013
Poll aggregators like me and @fivethirtyeight sure didn't object to @ppppolls when they were supplying 18% of our state-level data last fall
— Drew Linzer (@DrewLinzer) September 12, 2013
Editor’s note: The headline of this post has been changed because we thought of a better one.
