Is A/B Testing Ethical?
In undergrad, I took a course called Data 100: Principles and Techniques of Data Science. One day we had a guest lecturer, who is a PhD student in the Statistics Department. He opened by asking us a question: “Have you ever been involved in a Silicon Valley experiment?” One or two people raised their hands. He paused then said, “Everyone should raise their hands. You’ll all been apart of a Silicon Valley experiment. You’ve all been apart of A/B testing.”
I was reminded of this moment and the ubiquity of A/B testing when last week’s speaker, Suja Viswesan, Director of Engineering at LinkedIn, spoke about LinkedIn’s use of A/B testing on People You May Know (PYMK). She explained how there were two variations of the PMYK card layout, and how she personally thought one was better than the other due to the size of the profile picture. However, the results of the A/B test showed that LinkedIn users significantly interacted with her personally least preferred option more. This shows that when you are designing a user interface, you must consider the users first and forgo your own personal bias!
We aren’t aware of it, but we are constantly participating in A/B experiments. Companies such as Netflix, LinkedIn, Facebook, Google, Twitter, and Snapchat are big advocators of A/B testing. Recently I was reading a medium post about how product designers at Netflix do A/B testing. The medium article recounts a quote from a Netflix blog, which says:
“The general concept behind A/B testing is to create an experiment with a control group and one or more experimental groups (called “cells” within Netflix) which receive alternative treatments. Each member belongs exclusively to one cell within a given experiment, with one of the cells always designated the “default cell”. This cell represents the control group, which receives the same experience as all Netflix members not in the test.” 
As soon as a test is live, Netflix gathers metrics they consider important, such as streaming hours and retention.  Once they have gathered enough data, they deem a “winner” out of the different variations. Common tests include the picture a user sees on the Netflix homepage, and artwork for a specific film/tv series.
Given the pervasiveness of A/B testing, should we be concerned about the morality of such tests? In 2014, Facebook released a paper studying the mood effects of presenting a user with either positive or negative posts in his/her newsfeed.  Facebook received a lot of backlash over the study because it made the general public consider if a/b testing on unaware participants is ethical. Although the experiment had almost no effect on the end-users of Facebook, it brought up questions about user consent and the need for higher regulations on A/B testing.  A TechCrunch article offers a couple of solutions to this dilemma:
- Make riskier experiments opt-in
- Security audit on tech companies
- Educating data scientists on more ethical A/B testing practices 
I hope technology companies are making an effort to put their customers first, and not subject their users to risky experiments without their permission.
If you interested in learning more about A/B testing, here is a link to a talk by Netflix Product Designer, Anna Blaylock, in which she discusses Netflix’s A/B testing strategy! (Her talk is referenced in the Medium article I listed.)
Users who have LIKED this post: