A/B Testing: The Past, Today, and the Future

by on July 24, 2018 5:23 pm
Categories: Week4
Tags: , ,

This week’s guest speaker, Suja Viswesan, director of engineering at LinkedIn, talked about A/B testing, a very important component of LinkedIn’s initiative called “People You May Know.” She stated, “A/B testing is the only way to prove causality and quantify feature impact.”1 Building the right product based on big data is challenging in itself. A/B testing is then used to validate if the product is meeting its performance metrics. This harkens one of the characteristics of the LinkedIn culture: to “test everything.”1 While making data-driven decisions is already known to be of paramount importance to businesses, I wanted to dig deeper into how A/B testing came about and works with big data in today’s world.

 

According to Harvard Business Review, “A/B testing, at its most basic, is a way to compare two versions of something to figure out which performs better.”2 This is like the standard experiment where randomized samples are observed either in a controlled environment or a test environment to see if there is a statistically significant difference. This almost century old method was adapted by technology and internet companies in the 1990s: differing web content was given to a randomized pool of users. According to WIRED, “[Google’s] engineers ran their first A/B test on February 27, 2000.”3 A few web analytics or online marketing companies such as Omniture (now part of Adobe) dominated the A/B testing market in the 2000s. Since then, online marketing tools have become more sophisticated as the data have gotten bigger. Today, A/B testing tools are not only more sophisticated but also so ubiquitous that many more competitors can provide different kinds of A/B and multivariate testing tools.

 

In addition to the sophistication and widespread use, another significant change has been big data. The amount of data targeted by these tools has exploded; many companies are now able to make business decisions quicker with more certainty based on bigger sample sizes or statistical significance. One of the A/B testing companies, Optimizely, writes, “As more visitors encounter your variations and convert, you’ll start to see Statistical Significance increase because Optimizely is collecting evidence to declare winners and losers.”4 Furthermore, Viswesan mentioned that there are over 300 A/B tests running concurrently at LinkedIn on a daily basis.1 The fact that such an enormous amount of testing is being conducted at companies such as LinkedIn shows how much data companies are collecting nowadays.

 

A/B testing began a long time ago and is now used to predict user behaviors and help make smaller business decisions more quickly. As the data get bigger, testing will surely be used for other applications as well. It would be interesting to see where this century-old method takes us in the future.

 

References

  1. Viswesan, Suja (2018), Remarks in MS&E 238 class, Summer Quarter 2018.
  2. Gallo, Amy (2017), A Refresher on A/B Testing. Retrieved from Harvard Business Review: https://hbr.org/2017/06/a-refresher-on-ab-testing.
  3. Christian, Brian (2012), THE A/B TEST: INSIDE THE TECHNOLOGY THAT’S CHANGING THE RULES OF BUSINESS. Retrieved from WIRED: https://www.wired.com/2012/04/ff-abtesting/.
  4. Optimizely (2017). How long to run an experiment. Retrieved from Optimizely: https://help.optimizely.com/Analyze_Results/How_long_to_run_an_experiment#statistical_significance.

 

 

0