With #skewed polling in the news this campaign season, I stand as a lonely voice for noisy, biased, self-reported, and yes, skewed data in the Presidential race.
This piece is not about the purported bias in public polling, though I could go on and on about the shoddy reporting and analysis about polls. It’s about all the people who are getting into the polling game (Engage included) by using social media and Internet data to try and get a fix on what’s going on in in real time. This post is a field guide to these types of efforts, explaining where they’re useful, and how they do (and don’t) beat the polls that captivate the political class.
Why Scientific Polls Aren’t Enough
Let’s ask the first-order question here: Why?
Public opinion polls seem to be pretty good at forecasting the winners of elections, so why reinvent the wheel with newfangled metrics like tweets-per-minute or Facebook’s “people talking about this” number that aren’t scientific and whose subjects tend to be overly partisan and biased? Why study the Internet to figure out how public opinion is changing minute-by-minute?
On this, I still think Sir Edmund Hillary’s answer when asked why he would climb Mount Everest serves as a good guide: Because it’s there.
After more than a decade doing online activation, I can testify to the fact that there’s just about nothing users like better than answering polls. Millions of people answer online surveys every day, but the public polls released at the height of the election season reflect interviews with only a few thousand respondents per night.
In this age of abundant data, why is it getting harder and harder for pollsters to collect useful data on the electorate? Response rates to telephone surveys continue to plummet, and pollsters must recalibrate their methodologies to include cell phone-only households. You would think the Internet could pick up the slack here, but curators like Talking Points Memo won’t include online-only polls like YouGov in their averages. Technology hasn’t translated into a quantum leap forward in the volume of responses and quality of polling data.
As puzzling as I find anti-Internet bias, I have to concede there are some valid concerns here. To get a perfectly unbiased sample, you have to harass people over the phone because virtually all methodologies on the Internet are opt-in, and people who opt in to things are different than those who don’t. By definition, polls are about finding people who don’t already want to take them. Finding the rare person willing to sit through an interview and then balancing their responses is an expensive proposition. According to Chuck Todd, to do it right, NBC and the Wall Street Journal shell out between $40,000 and $60,000 per poll:
@patrickruffini so you know, a national poll, properly done w/live callers costs anywhere from 40-60K per survey.
The end product of these polls (which come in more and less-expensive varieties) is between 500 and 3,000 interviews that, on average, reflect public opinion as of a few days ago. After Mitt Romney’s crushing win in the first debate, we did not know that he had moved slightly ahead in the race until more than a week later. This is partly due to how polling shifts play out over several news cycles, but also because of the delay in reporting poll samples. Gallup, for instance, uses a 7-day rolling sample, which means that the median interview took place four days before the poll was released. So, polls can be accurate as of 2 to 4 days ago, but not accurate as of now.
It’s also difficult (nigh impossible if the group is small enough) to reliably measure polling movement among different subgroups in the electorate systematically over time. The smaller the group gets, the more it’s anyone’s guess as to what the real numbers are.
Yes, groups often pay more to poll specific demographics, but only a few times per election cycle. We are nowhere close, for instance, to having a RealClearPolitics average of, say, married women in Ohio, a measure that would be relevant to how campaigns actually spend money. And unless something drastically changes with how traditional polling is done, we will never have this. Ever.
The reason is because providing a balanced, unbiased sample is expensive. But what if you didn’t have to balance the sample?
This is where mining relevant streams of Internet data can help.
Social Media is the World’s Biggest Data Platform
We think of social media as the world’s biggest conversational platform. But it’s no slouch in the data department either.
Facebook users generate around 684,478 pieces of content per minute. Twitter users tweet 200 million times daily. This doesn’t even count the countless petabytes (exabytes? yottabytes?) of user account data on millions of websites, tied to demographics.
The sum total of these interactions speak volumes about each of us as a person. Not every variable will be public about every person, but our tendency to interact socially online, the language we use, and what we post can speak volumes about our personalities, our values, and our political beliefs. And this is just from what we post publicly.
Pollsters and most journalists have shied away from analyzing this data for a few reasons. First, obviously, is privacy. Second, we still lack the processing power and analytical capability to usefully make sense of these large data sets. And third, the easy, topline queries are often misleading, reflecting certain skews in online phenomena, in the sites themselves, and in things that are fundamentally hard to control for, like media attention or virality. Noting that Obama leads by 3-to-1 on Facebook is not terribly interesting, because it could reflect his incumbent status, his global popularity, his 4-year-headstart, or his cult-like status in 2008.
Nonetheless, if you ask the right questions — you can get at certain answers faster and with more granular data than a traditional poll.
To Get the Data, Embrace the Skew
During the Vice Presidential debates, Xbox Live polled viewers live during the debate as to who thought they won. The answers may have been disheartening for the Romney-Ryan ticket: undecided voters on the platform thought Joe Biden won the debate by a 44 to 23 percent margin.
But the sample was skewed: Xbox viewers as a whole were voting for Obama over Romney by a 52-36 percent margin — while public polls are tied. As gamers, the Xbox voter is typically younger, and so even the undecided might be left-leaning. Data from our Trendsetter app, which measures the political affinities of page likers on Facebook, is consistent with these results, showing a roughly 60-40 pro-Obama Xbox skew.
Before we use this skew to summarily discard the results, consider this: each question got 30,000 responses, presumably tied to rich demographic information. This means that, within the Xbox community, you have large samples of hundreds of voters for one of dozens of different slices of the electorate.
These large sample sizes mean you can get an extremely granular view of opinion changing over time, especially when data is tied to real user accounts with demographic info. Even if we don’t re-weight the demographics from the Xbox poll back to the overall population, because of the sheer volume of data, there is intrinsic value in studying the data shifts and the patterns evidenced in the polls internals.
The overall skew doesn’t matter, because we aren’t interested in the toplines (the Presidential horserace number). Traditional polls do a good enough job of measuring those. What we’re interested in is measuring change among niche demographics and doing it in real time, without the 2-to-4 day delay. When it comes to measuring what happened in the last 24 hours, campaign polls give us no data or extremely rough data. Sheer volume means Internet data can do a better job of this, particularly if it can be confirmed across multiple data sets.
In the recent debates, I polled my Twitter audience as to who they thought won. Most polls received between 200 and 1,000 responses, measured as retweets. Some tried to poke fun at this, given that my Twitter followers appear to skew 10-to-1 towards Romney based on the results. But my goal wasn’t to suggest that Romney was winning public opinion by 10-to-1. It was to collect as much data as fast as possible, extracting insight where appropriate. Last night, I asked people to indicate whether they thought each candidate was winning by a little or a lot. The data could suggest that Obama voters were a bit more enthusiastic about their guy’s performance, even though there were fewer of them (irrelevant for the purposes of this analysis).
RT if you think Romney is winning decisively. Favorite if you think he’s winning narrowly.
Asking the broader question of how well Twitter performed as a barometer during the debates, Twitter searches for “Romney winning” or “Obama winning” all accurately predicted the results of snap polling done after each debate. They showed Romney dominating the first debate from 20 minutes in, while a more muddled back-and-forth picture emerged from the remaining two debates — also consistent with the polls. After the conventions, we outlined the case for how Twitter reactions to major speakers forecasted the nightly movement in the polls, and found (with one or two exceptions) a clear correlation.
Twitter is full of biased and self-interested political actors, but it mirrors and reinforces the media narrative and thus public opinion. You can’t really measure undecided voters on Twitter, but you can tell which side’s partisans felt great, and which felt “Meh.” And you can quantify this in real time, getting ahead of the polls. Even with an unrepresentative sample, we’ve found it to be a good guide of broader opinion, but you have to drill down on specific search queries and eschew broad metrics like tweets-per-minute and treat sentiment analysis with caution. For instance, we found that use of a candidate’s name in conjunction with “awesome” could be a better indicator of positive reaction to a candidate than positive sentiment scores.
Towards the Hourly Tracking Poll
The 2012 elections won’t resolve the question of whether Big Data can predict election outcomes, but it holds great promise if we can embrace the heretical idea that balance isn’t the be-all, end-all, while we mine insights from the deep of Internet data.
Your next project can be fast, cheap, and good — pick two. Opinion data can be fast, balanced, and big — pick two. In looking at absurdly large data sets, and embracing the inherent skew represented by the bias in Xbox or Facebook users, asking the right questions, you can get at things no poll can — subtle changes in the samples and among specific demographics, measured day by day, or even hour by hour.
Why should this be important, beyond feeding the media-political beast with near real-time analytics?
The political world has embraced real-time data everywhere else — in everything from voter ID calls, to fundraising emails, to online advertising. Why wouldn’t public opinion research work the same way? People like giving their opinion. Is there a way to better harness these willing participants into actionable data?
After all three debates, the political discussion quickly descended into meme graphics about Big Bird, binders, and bayonets. This was fed in part by a data-driven feedback loop of hardcore partisans on social media — combined with a complete absence of data about how these attacks worked in real time with undecideds. Interviews conducted after the fact showed these attacks fell flat with those voters, yet the memes went on for days. Real-time polling might mean less Big Bird — and more messaging that’s actually relevant in Ohio.
Understanding influence is a huge topic in social media. A number of players, like Klout and PeerIndex, have built hugely successful platforms around rewarding highly influential social media users.
These platforms are great at measuring celebrity. If you’re Lady Gaga, you have a Klout score of 92. If you’re Barack Obama, your score is 91. Beyond that, microcelebrities with large Twitter followings and a healthy degree of interaction on the platform will earn high Klout scores, but what we’re talking about is a relatively small sliver of the social media universe.
This left us wondering: what would a good influence score look like for the rest of us who aren’t Twitter celebrities? And specifically, what does it look like on Facebook, the world’s biggest social stage?
Today, we’re launching Trendsetter, a platform which lets you discover who’s influential and what they care about.
Connect with the app and you’ll get your Trendsetter score — and see where you stack up compared to your friends. Trendsetter measures interactions with pages on Facebook and generates an individualized Trendsetter score for you and your friends. A high Trendsetter score means you’re very likely to tell your friends about things on Facebook, have niche tastes, and tend to be early to the party when it comes to liking brands and content. A lower Trendsetter score means you’re quieter in interacting on Facebook and tend to have more mainstream tastes — but when you do share, it’s because it really matters.
For years, through measures like the Net Promoter Score, marketers have been trying to understand the voters and consumers most likely to share things. We have an inkling that just a cursory glance at someone’s social media profile can tell you more about people’s propensity to share, and Trendsetter aims to show you what moves them.
A Trendsetter report gives you a wealth of data about your network — who the biggest early adopters are among your friends, what Facebook pages these early adopters like, what types of things they’re interested in, and how they’re distributed throughout the country. Here’s what my Trendsetter report looks like:
I knew we were onto something when our algorithm ranked Jesse Thomas of the DC-based digital agency JESS3 as the #1 Trendsetter in my network. Jesse is the consummate early adopter, and this makes him the biggest Trendsetter amongst my friends.
Trendsetter is a joint project of Engage and the Winston Group, a strategic communications and polling firm. With the Winston Group, we’ll be developing quick, one-question surveys for Trendsetter users, and breaking down the answers in interesting ways based on user interests and social influence — a level of detail it would be very hard to get at in a traditional opinion survey.
At Engage, we have long preached the gospel of social data as the new polling. A Facebook app called Wisdom is demonstrating what that actually means, building detailed demographic profiles on 3.8 million Facebook users and breaking down their page likes. This data is particularly interesting in light of the current Presidential election, where the rivaling tastes of supporters of President Barack Obama and the Republican candidates can tell you a lot about the nation’s cultural divide.
Earlier this evening on Twitter, I broke down the top 10 non-political page likes for supporters of each of the Presidential candidates as reported by Wisdom. To say the least, the results are revealing.
1. Michael Jackson
4. Family Guy
5. Bob Marley
6. Lady Gaga
9. Will Smith
2. The Bible
4. The Beatles
5. Small Business Saturday
7. Johnny Cash
9. Adam Sandler
1. The Beatles
2. Family Guy
3. Johnny Cash
4. Pink Floyd
5. South Park
6. Bob Marley
7. The Office
9. The Daily Show
1. The Daily Show
3. The Office
4. The Beatles
7. The Colbert Report
8. Johnny Cash
9. Family Guy
10. The Onion
1. The Bible
4. Jesus Daily
6. Dave Ramsey
7. Small Business Saturday
9. Jon Voight
1. The Bible
5. The Beatles
6. Dave Ramsey
7. Small Business Saturday
8. Jesus Daily
10. Jesus Christ
1. The Bible
4. Johnny Cash
7. Jesus Daily
8. Dallas Cowboys
9. George Strait
10. Dave Ramsey
If the list for Barack Obama and the rest of the GOP field doesn’t scream “cultural divide!” I don’t know what does. Regardless of the outcome of the Republican primary, 2012 is shaping up as a choice between Eminem and Johnny Cash, or Lady Gaga and The Beatles.
Ron Paul supporters show nearly as much overlap with the cultural preferences of Obama supporters as they do with supporters of Mitt Romney and the other Republican candidates. Ron Paul is the candidate of Pink Floyd, which makes its only appearance on Ron Paul supporters’ top 10. Alone with Jon Huntsman fans, Ron Paul supporters also like watching The Office.
Fans of Rick Santorum, Newt Gingrich, and Rick Perry all have very similar tastes — with Jesus Daily, Chick-fil-A, the Bible, Dave Ramsey, History, and Facebook overlapping all three lists. These also appeared well down their lists, with conservative politicians, organizations, and media figures dominating likes among fans of these candidates. This probably speaks to the fact that, at least among Facebook users, these candidates remain popular primarily among political early adopters.
As the Republican candidate with the most Facebook likes, Romney supporters’ tastes are more likely to veer into the realm of popular culture, albeit an older version of it where Johnny Cash and The Beatles rule.
Huntsman supporter tastes spell eclectic. From fans of NPR to the Daily Show to The Economist, Huntsman appeals to a very unique brand of Republican relative to the field.
Whenever a pundit rushes to proclaim the “death of” something, that’s the surest sign it’ll probably outlive the person making that bold prediction.
Nonetheless, as a general rule, I tend to bet on the future and the old incumbent industries and ways of doing things (eventually) being dislodged, even if progress in that direction is all too slow (Exhibit A: TV vs. online advertising in 2010). At a minimum, the trendlines become clear, even if the actual moment of transition isn’t yet.
With that in mind, I think we should be paying closer attention to what Facebook (and to some degree, Foursquare) was able to do on Election Day as an alternative to traditional polling.
On Election Day, Facebook placed an “I Voted” button on its home page. Over 12.4 million clicked it. That’s roughly one in seven people who voted on November 2nd. It’s also more than double the 5.4 million who clicked the same button in 2008, when overall turnout was roughly 50% higher.
The coolest thing about the button, speaking as a political data geek, wasn’t the fact of its very presence. It’s the analysis Facebook was later able to do on turnout patterns by age and political affiliation and even degrees of connection with other voters.
The chart of turnout levels by age and political party are exactly what you would expect. A steep rise from the low 20′s among young voters to nearly 50% in the 60-65 age bracket. And the enthusiasm gap was evident in these numbers. At almost every age level, Republicans were more likely to vote.*
The breakdown of political party affiliation by state also strikes me as perfectly valid:
This is also the first year I really didn’t look at the exit polls much if at all. Since 2004, it’s become abundantly clear with the rise of early voting and in their well-documented issues in predicting the Presidency of John F. Kerry that they are no more valid than a regular opinion poll conducted over the phone, and in some ways, have tended to miss the mark dramatically in ways no regular pollster would tolerate (I have a hard time believing that a phone pollster would have come up with Kerry by 20 in Pennsylvania or within the margin of error in South Carolina). And, they still have to be adjusted to match actual results a week after the vote! Shouldn’t a poll of 17,000 people, weighted properly, be able to produce results within 1% of the actual results without the benefit of such “adjustments?” Analysts routinely raise questions when the exit polls show voting preferences among groups like Hispanics off from all other polling. If the accuracy of the underlying data can’t be trusted, why would we take the “adjusted” figures at face value as the political community seemingly does?
This isn’t to say that I distrust all polling. As discussed on the podcast the other week, I love polling and consume it religiously in the run up to every election. High profile failures like the 2008 New Hampshire Democratic primary and the 2010 Nevada Senate race aren’t reflective of the overall accuracy of polls in predicting most races. For the most part, they give us a pretty good read on who is likely to win and by how much, and I don’t find them as problematic as the exit polls.
Nonetheless, even with the vastly increased volume of polls, they miss important things, like:
Individual House races didn’t get polled as much as they should have to get a true and accurate read on the state of play in the House. We instead rely on the pseudo-science of Cook and Rothenberg to fill in the blanks, and they always seem to be playing catch up.
Polling in primaries can be very spotty, with months if not weeks between public polls. Low-budget House campaigns don’t have the budget to do much more than a baseline and then one or two brushfire surveys to augment the corpus of public polling, leaving them mostly in the dark about real conditions on the ground.
Polling can’t give you the kind of granular data down to the county level you really need to optimize your GOTV efforts, only by broad regions like “Southern California” or the “San Francisco Bay Area.”
Trying to build an RCP or Pollster-like average for different demographic groupings or for core questions like Party ID that are actually pretty crucial to gauging overall dynamics is virtually impossible because of the different methodologies pollsters use to weight and even define these groups. Some pollsters hold party ID constant, others don’t. You can hedge against uncertainty by averaging the ballot test between polls but the sample sizes on subgroups are often so small that they are practically worthless in developing overall strategy.
This is why I find what Facebook did with their election data so appealing. They have no sample size issues, as they reflect an overall sample of one seventh of the electorate. Only self-selection issues. And increasingly I’ll trade less scientific data for a more insightful, larger data set that gives me granularity a poll can’t. It’s like the difference between a 100×50 thumbnail and a digital photograph in full 12 megapixel glory. You’re likely to get the basic idea from the thumbnail, but good look reading the text on that sign in the background.
Likewise, the “I Voted” project we were part of via Foursquare gave us data a poll couldn’t, visualizing for the first time I’ve seen anywhere online when people vote during the day. Even with all the timezones, you get a clear picture that most people really do tend to vote during the evening, with the 50% mark of total votes cast being reached at around 3pm.
You can nitpick this for a host of demographic reasons, by saying that seniors are not likely to be accounted for, etc. etc. — but what’s the alternative? No data? Flawed exit poll data? When people vote is actually a pretty crucial fact if you’re a field director and the entire campaign comes down to your turnout operation. And if we’re fully transparent about known problems in how people tend to use these services and thus how data is recorded, we can at least try to hedge against them or conduct longitudinal comparison only amongst those subgroupings most likely to have valid data, which is still pretty darn useful.
Nor is self-selection an unknown problem in the world of polling. With refusal rates being what they are, actually taking an entire survey seems to me to be a form of self-selection — how do you know you’re not biasing the results towards folks who are just plain lonely, or don’t have kids who demand their attention? The problem of polling cell phone-only households has also been much discussed, and the fix most pollsters have settled on is to reweight youth and minority numbers up, assuming that the cell phone-only voters in those groups match up nicely with landline voters. (Nate Silver’s post on this is a must-read.)
As services like Facebook get better about collecting anonymous data on tens of millions of users and cross-referencing it to party affiliation and variables most pollsters haven’t even thought of yet — how do MST3K fans break down? — I can see us moving away from polls as the be-all end-all for demographic research and moving to study large troves of data based on millions of user profiles. Self-selection and self-ID remain valid concerns, but less and less so as Facebook penetrates deeper into every age and ethnic group and region of the country. Three years ago, I was able to use Facebook data to study how fans of popular movies, TV shows, and bands broke down ideologically, and how ideology shifted for individual ages (not just age groups, ages) year to year. I bet the data today would be even more interesting.
At Engage, we’ve started conducting experiments with large datasets we encounter based on actual voter behavior and not surveys. We’ve been able to track the extent of an opponent’s media buy by looking at Google search query data and the likelihood of voters in individual counties to interact with a candidate in a teletownhall setting, based on a sample sizes in the tens of thousands. The former allowed us to get a better sense of the precise day the polling started to move and latter prediction turned out to be eerily prescient in predicting the final results. There are countless other experiments one could do with access to the right data, which is becoming more and more available.
None of this is to say that the discipline of marrying data mining and traditional survey research isn’t messy. Relying on metrics like counting Facebook fans or Google search query volume can be downright misleading because they’re subject to campaigns themselves manipulating the numbers or the digital equivalent of highway onlookers slowing down to gawk at a car wreck. You might be getting a lot of attention, but not for the right reason. Models will need to be built that account for the effect of celebrity candidates, with these less reliable data points occasionally discarded (as Nate Silver has said in predicting the Academy Awards, don’t let the model make you predict something you know is wrong).
Despite the obvious drawbacks, I find the opportunity presented by Big Data — the kind with millions, rather than just hundreds or thousands, of records — intensely exciting. Obama ’08 was a Big Data campaign. Instead of only relying on polls, they used trends collected daily in hundreds of thousands of Voter ID to allocate money in real time. Done right, we can use access to data to route around some of the shortcomings of traditional polls (cost, sample size limits, speed of data collection) in the same way that blogs and social media, albeit messier, have routed around the failures of elite media.
– * The dropoff among very old voters, which manifests some in the real electorate, but not as dramatically as on Facebook can likely be explained by diminished overall online usage among the elderly. If you’re 80 and on Facebook, it’s demographically likely not as many of your peers are on it, so you’re less likely to use it daily and hence click the button, among other factors.