A primer on A/B Testing

What is A/B Testing?


It is when you are comparing two versions of something to see which one performs better (VMO, 2016). The one that performs better is the one that is chosen. It can be anything from TV ads to websites. It is used often in the world of business.  

The purpose of A/B Testing

The purpose is to optimize the choice of media, simply to get more bang for your buck. The investment into A/B testing has been correlated with positive revenue, sales and leads. This is further supported by the wealth of information and tools available to us (Fung, 2014). The greatest benefit is that it answers the question why? Or what if? (Jenkins, 2014).   

How does it work?

  1.   Study the data to see if there is an issue: For example a high bounce rate on the landing page.
  2.   Construct a hypothesis: Using the data, build a hypothesis to achieve a goal. It can be one element or a complete redesign, but the less the better (Optimizely, 2016).
  3.   Test the hypothesis using A/B testing: Use a control and a variation to be able to better track and understand the true impact of the variation.
  4.   Analyze the data and draw conclusions: There is either a clear winner or no change. Either way that does not mean A/B testing is useless. 80-90 percent of A/B tests are counted as ‘failures’ (Fung, 2014). And are considered not statistically significant. The feedback drawn from the testing will show if the company is on the right track and avoid wasting money and time (Fung, 2014).
  5.   Implement the choice and evaluate: Word of caution, subsequent roll out is to the untreated half of the customer base, there expect aggregate improvement of 2.5% if the test showed an increase of 5% (Fung, 2014).


Some people may see A/B testing as a form of “manipulating emotions” (Constine, 2014). If one is to employ A/B testing, ensure that there is an opt out option for users. Companies should educate employees in ethical research methods (Constine, 2014).