Creating an A/B Testing Culture

In recent months, we’ve rolled out a robust A/B testing program – powered by Optimizely and Google Analytics Content Experiments – on a number of different clients’ projects and campaigns. This has largely led to successful growth of both our business and the level of user engagement in our clients projects, to the point where it is now a standard non-premium phase of every project we do.

A/B tests are normally run on high-value conversion points like donation CTAs, donation form layout, and vote pledges, but they can also be used to test page layouts and designs.

Over the course of this rollout, we’ve identified a recurring issue that has presented a challenge to taking advantage of A/B testing: namely, a failure to obtain full organizational buy-in for this process of testing and for the possibility of unexpected outcomes. For these clients, the challenge of securing organizational consensus for their digital revamp made the prospect of a testing program too much to bear.  

While we had buy in from the staff person managing the project of the client-side, by the time we were rolling out their new site they’d lost their appetite for testing.  

The challenge of taking consensus decisions and risking upsetting the consensus through a testing program seems unpalatable to many organizational staff, and they instead opt for untested user-experience to avoid upsetting their internal team.

This experience is informing an evolution in how we run testing programs in a number of ways. Below are a few reflections and recommendations for helping to guide A/B testing programs.

• Organizations and digital teams need to decide what the key messages and user interaction points are that are pivotal to success, and embrace testing at these points.

• Consensus should be built around the ideas and goals of campaigns and digital properties, and not on the particular language, colours, or content of those properties.

• Have the major stakeholders present. We’re now asking to have senior decision-makers at clients’ organizations attend the initial ‘Mapping meeting’ that we do with them. In the past, we’ve only required the primary points of contact on the client side.

• Explain to everyone in the organization what A/B testing is and why it’s important.

• We’ll soon be rolling out an improved art/design revision round to demonstrate areas we hope to test on the website component of the project is live.

We’ve also blogged some initial reactions when we started testing in NationBuilder using Optimizely & List Splitter a few months back; make sure to take a look at those here.