Is A/B Testing Ethical?
In undergrad, I took a course called Data 100: Principles and Techniques of Data Science. One day we had a guest lecturer, who is a PhD student in the Statistics Department. He opened by asking us a question: “Have you ever been involved in a Silicon Valley experiment?” One or two people raised their hands. He paused then said, “Everyone should raise their hands. You’ll all been apart of a Silicon Valley experiment. You’ve all been apart of A/B testing.”
I was reminded of this moment and the ubiquity of A/B testing when last week’s speaker, Suja Viswesan, Director of Engineering at LinkedIn, spoke about LinkedIn’s use of A/B testing on People You May Know (PYMK). She explained how there were two variations of the PMYK card layout, and how she personally thought one was better than the other due to the size of the profile picture. However, the results of the A/B test showed that LinkedIn users significantly interacted with her personally least preferred option more. This shows that when you are designing a user interface, you must consider the users first and forgo your own personal bias!
We aren’t aware of it, but we are constantly participating in A/B experiments. Companies such as Netflix, LinkedIn, Facebook, Google, Twitter, and Snapchat are big advocators of A/B testing. Recently I was reading a medium post about how product designers at Netflix do A/B testing. The medium article recounts a quote from a Netflix blog, which says:
“The general concept behind A/B testing is to create an experiment with a control group and one or more experimental groups (called “cells” within Netflix) which receive alternative treatments. Each member belongs exclusively to one cell within a given experiment, with one of the cells always designated the “default cell”. This cell represents the control group, which receives the same experience as all Netflix members not in the test.” [1]
As soon as a test is live, Netflix gathers metrics they consider important, such as streaming hours and retention. [1] Once they have gathered enough data, they deem a “winner” out of the different variations. Common tests include the picture a user sees on the Netflix homepage, and artwork for a specific film/tv series.
Given the pervasiveness of A/B testing, should we be concerned about the morality of such tests? In 2014, Facebook released a paper studying the mood effects of presenting a user with either positive or negative posts in his/her newsfeed. [2] Facebook received a lot of backlash over the study because it made the general public consider if a/b testing on unaware participants is ethical. Although the experiment had almost no effect on the end-users of Facebook, it brought up questions about user consent and the need for higher regulations on A/B testing. [2] A TechCrunch article offers a couple of solutions to this dilemma:
- Make riskier experiments opt-in
- Security audit on tech companies
- Educating data scientists on more ethical A/B testing practices [2]
I hope technology companies are making an effort to put their customers first, and not subject their users to risky experiments without their permission.
If you interested in learning more about A/B testing, here is a link to a talk by Netflix Product Designer, Anna Blaylock, in which she discusses Netflix’s A/B testing strategy! (Her talk is referenced in the Medium article I listed.)
Enjoy!
[1] https://uxdesign.cc/how-netflix-does-a-b-testing-87df9f9bf57c
[2] https://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/
Users who have LIKED this post:
4 comments on “Is A/B Testing Ethical?”
Comments are closed.
Great post Emily. I wonder if tech companies will eventually have ethics/institutional review boards like those that govern biomedical research.
Users who have LIKED this comment:
Thanks for the interesting and informative post, Emily! I was just thinking the same thing as Adam above. I know a lot of academic research requires review from the IRB (https://en.wikipedia.org/wiki/Institutional_review_board). Their approval process seems like such a stark contrast from tech companies’ A/B testing, which can be done immediately and with virtually any experiment. The difficulty though is that it will be hard for such a review board to emerge for tech companies… to be pessimistic, tech companies can simply conduct their own A/B tests and just not tell anyone. Hopefully companies figure out some way to integrate ethics into their research practices.
Users who have LIKED this comment:
Interesting and very important post, Emily! I’m glad you tackled this topic 🙂
It seems to be an ever-ongoing race for governmental entities to keep up with businesses and the rushing breakthroughs in technology. The GDPR in EU is a good example of how the rest of the world woke up in the reality of how data is being used as a means of power only recently (https://www.zdnet.com/article/gdpr-an-executive-guide-to-what-you-need-to-know/).
Even though there’s been discussion about the morality of A/B testing (see https://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/) it will be interesting to see when consumers are (or maybe they never won’t) be advised, educated and asked for permission about this type of testing that pretty much can be considered as a psychological test!
Users who have LIKED this comment:
I thought it was an interesting and important question since A/B testing is so ubiquitous now. While I have used A/B testing tools to research user behaviors, I had never asked myself whether conducting A/B tests is ethical. This type of test uses randomized samples to serve different tests, controls, or environments, so many of the testers would not even know if they had been targeted by this experiment. I am curious to see if this ethical question becomes more mainstream as the usage of this experimentation method continues to grow.