A primer on A/B Testing

What is A/B Testing?

When comparing two versions of something to see which one performs better (VMO, 2016), the one that performs better is the one chosen. It can be anything from TV ads to websites. It is used often in the world of business. 

The purpose of A/B Testing

The purpose is to optimize the choice of media, to get more bang for your buck. The investment into A/B Testing has been correlated with positive revenue, sales, and leads. This is further supported by the wealth of information and tools available to us (Fung, 2014). The most significant benefit is that it answers the question, why? Or what if? (Jenkins, 2014).  

How does it work?

1.   Study the data to see if there is an issue: A high bounce rate on the landing page.
2.   Construct a hypothesis: Using the data, build a theory to achieve a goal. It can be one element or a complete redesign, but the less, the better (Optimizely, 2016).
3.   Test the hypothesis using A/B Testing: Use control and a variation to better track and understand the true impact of the variation.
4.   Analyze the data and draw conclusions: There is either a clear winner or no change. Either way, that does not mean A/B Testing is useless. 80-90 percent of A/B tests are counted as ‘failures’ (Fung, 2014). And are considered not statistically significant. The feedback drawn from the Testing will show if the company is on the right track and avoid wasting money and time (Fung, 2014).
5.   Implement the choice and evaluate: Word of caution, the subsequent rollout is to the untreated half of the customer base, there expect the aggregate improvement of 2.5% if the test showed an increase of 5% (Fung, 2014).

Safeguards

Some people may see A/B Testing as a form of “manipulating emotions” (Constine, 2014). If one is to employ A/B Testing, ensure that there is an opt-out option for users. Companies should educate employees in ethical research methods (Constine, 2014).

References

•         Constine, J. (2014, June 29). The Morality Of A/B Testing. Retrieved from https://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/
•         Fung, K. (2014, December 10). Yes, A/B Testing Is Still Necessary. Retrieved from https://hbr.org/2014/12/yes-ab-testing-is-still-necessary
•         Jenkins, W. (2014, February 5). A/B Testing and the Benefits of an Experimentation Culture. Retrieved from https://hbr.org/2014/02/ab-testing-and-the-benefits-of-an-experimentation-culture
•         Optimizely. (2016). A/B Testing. Retrieved from https://www.optimizely.com/ab-testing/
•         VMO. (2016). The Complete Guide to A/B Testing. Retrieved from https://vwo.com/ab-testing/