What is A/B testing?


“Take the guess work out of the equation” – Mahuya 

What is A/B Testing?

It’s a powerful technique that allows one to test and experiment with simple UI changes or complex flows and features.  The end goal of this exercise is to determine which version is best or ‘works’ with clear and actionable insights. In short, it is the thin line between “we think” to “we know”.

It is one of the user research technique that is applicable for late stage projects, where you have fair amount of knowledge on the problem but you need more objective & quantitative data to base your decision.  

Here’s a visual representation of the various User Research techniques & where A/B testing falls:

Overwhelmed? Well, the good news is you don’t necessarily have to use all of these techniques in one go.  Depending on the maturity of the product & the kind of insight(s) you are looking for, you would need to decide on the relevant technique(s).

Wondering, how can you benefit from A/B Testing?

  • If driven by analytics, it can accurately measure actual human behavior under real situations
  • If the sample size is good, it can measure very small performance differences with high statistical significance
  • It helps to resolve product capability trade-offs with factual data
  • And oh! Did we mention that it’s cheap? It is actually!

So, how do you do it?

“Simplicity is the ultimate sophistication” – Leonardo da Vinci

If we had to put it simply, A/B testing starts with 2 versions of a prototype. Then you find real users to take the test. This obviously results in sample split and the behavior is recorded. The real time findings are then used to proceed to the next step in the product funnel. Ideally, the process should have the following basic plan of action:

  • Form a testable hypothesis with clear goals which can be analytically measured
  • Identify the testable variables
  • Test by user segment
  • Test visitor flow with a goal of measuring which screen drives the greatest impact on retention (i.e., less drop off)
  • Look for patterns and quick wins

Here’s an A/B Variations for a non-software product to determine which variation of the jacket will be more relevant for dogs to use during winter:

If you are curious to know how A/B testing can significantly increase software product usage, sales with real world examples & tools that experts swear by, then stay tuned for the next edition!