And while most of us don’t have access to a full-time analytics staff, A/B testing is within reach for any organization of any size. In fact, the Obama campaign used Optimizely, a web-based A/B testing platform, that you can leverage for your organization for as little as $19 per month depending on your site traffic.
What is A/B Testing?
In reference to websites, an A/B test compares two variables of an element — color, text, or placement — to determine whether or not a change in that element improves a desired outcome.
Here’s a quick guide to get started with A/B testing on your site.
Once you’ve set up your Optimizely account, you’ll need to provide your web developer or vendor with a snippet of code that will allow you to make edits to your site from Optimizely. That’s the last you’ll need to deal with code for optimizely.
Determine what you want to optimize
The key to worthwhile optimization is having a measurable outcome (like a donation) and a clearly defined goal (more donations). For organizations that are new to A/B testing, increasing email capture and donations are two of the most attainable goals for A/B testing.
Outline your experiment
Now that you know what you want to improve, identify elements on your site that you can tweak to improve the response. For example, if you’re optimizing for email signups, you could test the submit button text. Does changing “submit” on the button to “join” increase your conversion rate? Do more people donate if the button is red or blue? In order to get actionable insights from your test, you should find apples to apples comparisons.
Good A/B Test
Red Button vs Blue Button.
This test will reveal any difference button color has on conversions.
Good A/B Test
“Join” vs “Submit”.
This test will reveal how changing the text on the submit button will affect signups.
Bad A/B Test
Red “Join” button vs Blue “submit button.
After this test, you won’t be able to isolate the source of any change in performance.
Implement and run your experiment
Using Optimizely’s editor, you can easily tweak design and textual elements on your site and get the experiment up and running. The more traffic your site gets, the more data you’ll receive, and the better your A/B test will perform. You want a statistically significant outcome to ensure that the change in outcome you’re seeing is actually the result of the change in design you made.
While Optimizely will let you run multiple variations, for organizations that might not see much web traffic, it’s best to conduct one test at a time to zero in on the best design for your page.
Avoid the Frankenstein Effect
You should always make sure you test every combination before settling on a final design change. Photo A may outperform Photo B and a red background may out perform a blue background, but you cannot assume that Photo A on a red background will outperform Photo A on a blue background or even Photo B on a red background. In short, test EVERYTHING.
Testing shouldn’t stop at your web design. Look for other opportunities to conduct focused, measurable A/B tests. This could include testing email subject lines or online ad creative.
In the immediate aftermath of the 2012 presidential campaign – like any campaign – two things happened: the winners went to bragging and the losers started pointing fingers. One thing became clear. Obama for America’s digital, technology, and analytics teams were indispensable in securing the president’s reelection.
The Cave is what OFA called the windowless room that housed their analytics team. Like digital in 2008, analytics came of age in the 2012 campaign. OFA’s analytics team had 50 staffers. By comparison, the Romney-Ryan campaign had a data team of 4 people.
Veterans of OFA have been surprisingly forthcoming in providing details on how they leveraged the latest in technology and digital strategy to make their campaign as effective and efficient as possible.
In 2016, Republicans can’t afford to fight the battles of 2012. We have to look forward to the future and start preparing now.
Over the past week, social media monitoring software has really come into its own as a serious tool for the media and political analysis. For the first time, lots of people were monitoring instantaneous changes in public reaction from the speeches, looking at metrics like tweets-per-minute through real-time charting tools like this one from Flowics.
The sheer amount of information that comes from these tools is a godsend, but without context they can fall short. A read on social media sentiment associated with Mitt Romney throughout the convention may not tell us very much. And touting the fact that @BarackObama’s “This seat’s taken” tweet was the second most retweeted in Twitter history doesn’t tell us very much about how those not on Twitter reacted to Clint Eastwood’s speech.
So, we decided to do a little bit more digging to see if we could find something more meaningful and actionable.
We wanted to quantify online reaction to each of the major RNC speakers in a way that might reflect the offline audience response. So, we avoided looking at general commentary on Romney or the convention — but rather at which speech content. Chances are if you’ve tweeted a line from a speech, you strongly approve of it or think it was significant. We wanted to see which speech lines “tested” the best based on social mentions, leaving the deepest impressions with audiences online and off.
We’ve collected the results in a treemap visualization that’s broken down along three dimensions: The most-mentioned speech lines grouped by speaker, total mentions of the speaker, and mentions of the speaker with the word “awesome.” All numbers reflect activity on the day of the speech.
The results are interesting, and defy conventional wisdom in places. Despite the political elite’s queasy reaction to Clint Eastwood’s offbeat appearance, he delivered the buzziest line of the Convention, “We own this country,” followed by “Politicians are employees of ours.” Among those expressing an opinion, reaction to this passage was about three-fourths favorable. Eastwood’s other top lines hailed from the parts of his speech that were well received in the hall, and saw virtually no negative reaction online. And Eastwood led all speakers in eliciting a reaction of “Awesome” from social media users.
Here are the ten most memorable lines we found from the 2012 GOP convention, ranked by social media mentions provided by Topsy:
“I would just like to say something, ladies and gentlemen. Something that I think is very important. It is that, you, we — we own this country.” – Clint Eastwood, 6947 mentions
“President Obama promised to begin to slow the rise of the oceans and heal the planet. MY promise…is to help you and your family.” – Mitt Romney, 5890 mentions
“The greatest threat to Medicare is Obamacare, and we’re going to stop it.” – Paul Ryan, 5,363 mentions
“Real leaders don’t follow polls. Real leaders change polls.” – Chris Christie, 5259 mentions
“College graduates should not have to live out their 20s in their childhood bedrooms, staring up at fading Obama posters and wondering when they can move out and get going with life.” – Paul Ryan, 4,395 mentions
“Politicians are employees of ours.” – Clint Eastwood, 3,657 mentions
“When the world needs someone to do the really big stuff, you need an American.” – Mitt Romney, 3,386 mentions
“They believe in teacher’s unions. We believe in teachers.” – Chris Christie, 3,045 mentions
“Let’s get this done.” – Paul Ryan, 2,744 mentions
“I haven’t cried that hard since I found out that there is 23 million unemployed people in this country.” – Clint Eastwood, 2,673 mentions
Watch this space this week for a similar analysis of the Democratic Convention in Charlotte!
Understanding influence is a huge topic in social media. A number of players, like Klout and PeerIndex, have built hugely successful platforms around rewarding highly influential social media users.
These platforms are great at measuring celebrity. If you’re Lady Gaga, you have a Klout score of 92. If you’re Barack Obama, your score is 91. Beyond that, microcelebrities with large Twitter followings and a healthy degree of interaction on the platform will earn high Klout scores, but what we’re talking about is a relatively small sliver of the social media universe.
This left us wondering: what would a good influence score look like for the rest of us who aren’t Twitter celebrities? And specifically, what does it look like on Facebook, the world’s biggest social stage?
Today, we’re launching Trendsetter, a platform which lets you discover who’s influential and what they care about.
Connect with the app and you’ll get your Trendsetter score — and see where you stack up compared to your friends. Trendsetter measures interactions with pages on Facebook and generates an individualized Trendsetter score for you and your friends. A high Trendsetter score means you’re very likely to tell your friends about things on Facebook, have niche tastes, and tend to be early to the party when it comes to liking brands and content. A lower Trendsetter score means you’re quieter in interacting on Facebook and tend to have more mainstream tastes — but when you do share, it’s because it really matters.
For years, through measures like the Net Promoter Score, marketers have been trying to understand the voters and consumers most likely to share things. We have an inkling that just a cursory glance at someone’s social media profile can tell you more about people’s propensity to share, and Trendsetter aims to show you what moves them.
A Trendsetter report gives you a wealth of data about your network — who the biggest early adopters are among your friends, what Facebook pages these early adopters like, what types of things they’re interested in, and how they’re distributed throughout the country. Here’s what my Trendsetter report looks like:
I knew we were onto something when our algorithm ranked Jesse Thomas of the DC-based digital agency JESS3 as the #1 Trendsetter in my network. Jesse is the consummate early adopter, and this makes him the biggest Trendsetter amongst my friends.
Trendsetter is a joint project of Engage and the Winston Group, a strategic communications and polling firm. With the Winston Group, we’ll be developing quick, one-question surveys for Trendsetter users, and breaking down the answers in interesting ways based on user interests and social influence — a level of detail it would be very hard to get at in a traditional opinion survey.
Quick question: when was the last time you saw an ad on Facebook that seemed so hilariously off-base from your interests that you remarked on it to your friends?
Probably recently. Facebook has conditioned us to expect ads that relate to our interests to the point where people are surprised when the ads aren’t relevant.
I, for one, am appreciative of our data-driven online advertising culture. Advertising is just a way for companies to communicate with consumers about products the company thinks they might find useful. Thanks to great innovation in the online advertising space, most ads you see online are individualized — by zip code, estimated income range, likely gender, or, most controversially, by what other websites you’ve visited or if you’ve recently visited a particular advertiser’s website.
The controversy about ad targeting in general has continued to be front and center this week, with an article in The Atlantic bemoaning the prevalence of targeted political ads on partisan websites as keeping “people within the boundaries of the things they once read and thoughts they once had.” But ad targeting isn’t perfect, and it’s quite possible to see non-conservative ads on conservative websites.
In fact, almost everyone’s jobs are reliant on the market (i.e., other people) finding value in the work their employer does, whether it’s Proctor and Gable or a non-profit. If sales go up, the company will need to hire more employees or use more services/supplies to meet that demand. And if those organizations can reach out to people who could be interested in their “products” more efficiently — whether they make Febreze or raise money for children’s cancer research — we all benefit.
But to be sure that relevant ads are beneficial, let’s walk through two examples.
1) I see an ad for Lakers tickets. I don’t follow basketball or live in LA, so I don’t pay attention to this ad. I don’t click on it. I don’t get any value out of this ad.
2) I see an ad from Amazon.com advertising a book that has a high reader correlation with a book I’ve recently bought from Amazon. I click on the link, read the reviews, buy the book, and love it. I have benefited from seeing this ad. On top of that, it’s made an efficiency improvement in my life: rather than spending an hour or two at the bookstore and perhaps stumbling upon this book, I’ve bought it in a matter of maybe ten minutes and had it shipped to my house. (Putting aside the fact that I personally love meandering through bookstores, this ad has still made an improvement in utility over the other ad.)
Targeted advertising is also a major reason why we are able to enjoy a wealth of news, information and entertainment online free of charge. Never before have we all had such an enormous amount of content at our fingertips. But “free” content is never free, and if it comes at the cost of me (gasp!) seeing an ad for something I might like, that’s a trade-off I’m willing to make.
Late last year, we have the privilege of working with The Leadership Institute to visualize the organization’s incredible accomplishments in 2011. The Leadership Institute has a reputation that is second to none in Washington, D.C. and in conservative circles across the country; but few people know the full extent of LI’s influence and impact on training conservative leaders.
We were tasked with telling LI’s complete story of accomplishments in a single infographic – a challenging task from a communications and design stand-point. We are very proud of the finished product and wanted to share it with you below. Please share this visually compelling run-down of LI’s accomplishments with your friends on Facebook and on Twitter.
Earlier this year, the Engage team had a fantastic time at the South by Southwest (SXSW) Interactive Conference in Austin, Texas. In preparation for SXSW 2012, we have submitted a proposal for a panel titled “Big Data: Powering the Race for the White House.” We need your help to get this panel on the schedule for SXSW 2012! The SXSW Panel Picker is now open to the world, meaning that you can vote for our panel to make it on the docket for SXSW 2012. Our panel will include Josh Hendler of Jumo, Kristen Soltis of The Winston Group, Dan Siroker of Optimizely, Alex Lundry of TargetPoint Consulting, and yours truly. Here’s the description of the panel as proposed to the SXSW team:
Despite the advent of new media, campaigns for President still measure the electorate in pretty much the same way they did 40 years ago, through traditional polls to landline phones. That could all change this year. The hottest job in today’s Presidential campaigns is the Data Mining Scientist — whose job it is to sort through terabytes of data and billions of behaviors tracked in voter files, consumer databases, and site logs. They’ll use the numbers to uncover hidden patterns that predict how you’ll vote, if you’ll pony up with a donation, and if you’ll influence your friends to support a candidate. This panel will delve deep into the world of real-time data on Presidential campaigns, showing how it’ll be used to make decisions on everything from the layout of a signup form to where to spend millions of advertising dollars in the closing days of a campaign. Forget about which candidate has the most likes on Facebook or followers on Twitter — and learn why 2012 will be the year of Big Data in American politics.
I want to optimize my site to run just like Obama’s but I don’t have millions of visitors. How do apply these lessons to a smaller operation?
Are conclusions based on new marketing data — much of it subject to possible selection bias — scientifically valid?
The wealth of political and consumer data that’s out there can be overwhelming. How do practitioners in the field avoid “analysis paralysis”?
As a voter, should I be concerned about the privacy implications of Big Data?
What are the resources I need to make this work for my organization?
Sound like something you would want to hear at SXSW or streamed online? Be sure to vote for our panel today and we will look forward to seeing you in Austin in March!
Whenever a pundit rushes to proclaim the “death of” something, that’s the surest sign it’ll probably outlive the person making that bold prediction.
Nonetheless, as a general rule, I tend to bet on the future and the old incumbent industries and ways of doing things (eventually) being dislodged, even if progress in that direction is all too slow (Exhibit A: TV vs. online advertising in 2010). At a minimum, the trendlines become clear, even if the actual moment of transition isn’t yet.
With that in mind, I think we should be paying closer attention to what Facebook (and to some degree, Foursquare) was able to do on Election Day as an alternative to traditional polling.
On Election Day, Facebook placed an “I Voted” button on its home page. Over 12.4 million clicked it. That’s roughly one in seven people who voted on November 2nd. It’s also more than double the 5.4 million who clicked the same button in 2008, when overall turnout was roughly 50% higher.
The coolest thing about the button, speaking as a political data geek, wasn’t the fact of its very presence. It’s the analysis Facebook was later able to do on turnout patterns by age and political affiliation and even degrees of connection with other voters.
The chart of turnout levels by age and political party are exactly what you would expect. A steep rise from the low 20′s among young voters to nearly 50% in the 60-65 age bracket. And the enthusiasm gap was evident in these numbers. At almost every age level, Republicans were more likely to vote.*
The breakdown of political party affiliation by state also strikes me as perfectly valid:
This is also the first year I really didn’t look at the exit polls much if at all. Since 2004, it’s become abundantly clear with the rise of early voting and in their well-documented issues in predicting the Presidency of John F. Kerry that they are no more valid than a regular opinion poll conducted over the phone, and in some ways, have tended to miss the mark dramatically in ways no regular pollster would tolerate (I have a hard time believing that a phone pollster would have come up with Kerry by 20 in Pennsylvania or within the margin of error in South Carolina). And, they still have to be adjusted to match actual results a week after the vote! Shouldn’t a poll of 17,000 people, weighted properly, be able to produce results within 1% of the actual results without the benefit of such “adjustments?” Analysts routinely raise questions when the exit polls show voting preferences among groups like Hispanics off from all other polling. If the accuracy of the underlying data can’t be trusted, why would we take the “adjusted” figures at face value as the political community seemingly does?
This isn’t to say that I distrust all polling. As discussed on the podcast the other week, I love polling and consume it religiously in the run up to every election. High profile failures like the 2008 New Hampshire Democratic primary and the 2010 Nevada Senate race aren’t reflective of the overall accuracy of polls in predicting most races. For the most part, they give us a pretty good read on who is likely to win and by how much, and I don’t find them as problematic as the exit polls.
Nonetheless, even with the vastly increased volume of polls, they miss important things, like:
Individual House races didn’t get polled as much as they should have to get a true and accurate read on the state of play in the House. We instead rely on the pseudo-science of Cook and Rothenberg to fill in the blanks, and they always seem to be playing catch up.
Polling in primaries can be very spotty, with months if not weeks between public polls. Low-budget House campaigns don’t have the budget to do much more than a baseline and then one or two brushfire surveys to augment the corpus of public polling, leaving them mostly in the dark about real conditions on the ground.
Polling can’t give you the kind of granular data down to the county level you really need to optimize your GOTV efforts, only by broad regions like “Southern California” or the “San Francisco Bay Area.”
Trying to build an RCP or Pollster-like average for different demographic groupings or for core questions like Party ID that are actually pretty crucial to gauging overall dynamics is virtually impossible because of the different methodologies pollsters use to weight and even define these groups. Some pollsters hold party ID constant, others don’t. You can hedge against uncertainty by averaging the ballot test between polls but the sample sizes on subgroups are often so small that they are practically worthless in developing overall strategy.
This is why I find what Facebook did with their election data so appealing. They have no sample size issues, as they reflect an overall sample of one seventh of the electorate. Only self-selection issues. And increasingly I’ll trade less scientific data for a more insightful, larger data set that gives me granularity a poll can’t. It’s like the difference between a 100×50 thumbnail and a digital photograph in full 12 megapixel glory. You’re likely to get the basic idea from the thumbnail, but good look reading the text on that sign in the background.
Likewise, the “I Voted” project we were part of via Foursquare gave us data a poll couldn’t, visualizing for the first time I’ve seen anywhere online when people vote during the day. Even with all the timezones, you get a clear picture that most people really do tend to vote during the evening, with the 50% mark of total votes cast being reached at around 3pm.
You can nitpick this for a host of demographic reasons, by saying that seniors are not likely to be accounted for, etc. etc. — but what’s the alternative? No data? Flawed exit poll data? When people vote is actually a pretty crucial fact if you’re a field director and the entire campaign comes down to your turnout operation. And if we’re fully transparent about known problems in how people tend to use these services and thus how data is recorded, we can at least try to hedge against them or conduct longitudinal comparison only amongst those subgroupings most likely to have valid data, which is still pretty darn useful.
Nor is self-selection an unknown problem in the world of polling. With refusal rates being what they are, actually taking an entire survey seems to me to be a form of self-selection — how do you know you’re not biasing the results towards folks who are just plain lonely, or don’t have kids who demand their attention? The problem of polling cell phone-only households has also been much discussed, and the fix most pollsters have settled on is to reweight youth and minority numbers up, assuming that the cell phone-only voters in those groups match up nicely with landline voters. (Nate Silver’s post on this is a must-read.)
As services like Facebook get better about collecting anonymous data on tens of millions of users and cross-referencing it to party affiliation and variables most pollsters haven’t even thought of yet — how do MST3K fans break down? — I can see us moving away from polls as the be-all end-all for demographic research and moving to study large troves of data based on millions of user profiles. Self-selection and self-ID remain valid concerns, but less and less so as Facebook penetrates deeper into every age and ethnic group and region of the country. Three years ago, I was able to use Facebook data to study how fans of popular movies, TV shows, and bands broke down ideologically, and how ideology shifted for individual ages (not just age groups, ages) year to year. I bet the data today would be even more interesting.
At Engage, we’ve started conducting experiments with large datasets we encounter based on actual voter behavior and not surveys. We’ve been able to track the extent of an opponent’s media buy by looking at Google search query data and the likelihood of voters in individual counties to interact with a candidate in a teletownhall setting, based on a sample sizes in the tens of thousands. The former allowed us to get a better sense of the precise day the polling started to move and latter prediction turned out to be eerily prescient in predicting the final results. There are countless other experiments one could do with access to the right data, which is becoming more and more available.
None of this is to say that the discipline of marrying data mining and traditional survey research isn’t messy. Relying on metrics like counting Facebook fans or Google search query volume can be downright misleading because they’re subject to campaigns themselves manipulating the numbers or the digital equivalent of highway onlookers slowing down to gawk at a car wreck. You might be getting a lot of attention, but not for the right reason. Models will need to be built that account for the effect of celebrity candidates, with these less reliable data points occasionally discarded (as Nate Silver has said in predicting the Academy Awards, don’t let the model make you predict something you know is wrong).
Despite the obvious drawbacks, I find the opportunity presented by Big Data — the kind with millions, rather than just hundreds or thousands, of records — intensely exciting. Obama ’08 was a Big Data campaign. Instead of only relying on polls, they used trends collected daily in hundreds of thousands of Voter ID to allocate money in real time. Done right, we can use access to data to route around some of the shortcomings of traditional polls (cost, sample size limits, speed of data collection) in the same way that blogs and social media, albeit messier, have routed around the failures of elite media.
– * The dropoff among very old voters, which manifests some in the real electorate, but not as dramatically as on Facebook can likely be explained by diminished overall online usage among the elderly. If you’re 80 and on Facebook, it’s demographically likely not as many of your peers are on it, so you’re less likely to use it daily and hence click the button, among other factors.