DE
Investing in CRO or A/B Testing Software? You (Decision Maker) Must Know This
You have been hearing about A/B testing for a while. It always sounded useful, but was never quite the need of the hour. But now the sounds have gotten louder and you are reevaluating the need for conversion optimization.
If that seems familiar, this article is for you.
Download Free: A/B Testing Guide
In its 2016 Conversion Rate Optimization (CRO) Report, ConversionXL notes that more and more organizations have begun to care about CRO. Of the 673 responses they collected, here’s what the results showed:
If you are seriously considering CRO, here’s what you must know before investing in CRO or A/B testing software:
1) A/B Testing and CRO Aren’t the Same
A/B testing is a technique, and CRO is the process that involves the technique.
For sure, A/B testing has revolutionized the way online businesses look at growth. It has enabled them to make decisions reliably to improve conversion rates. The reliability seemed magical.
However, the industry hit disillusionment and then realized that A/B testing is only an enabler and not a panacea.
A/B testing is exactly what it says—a test for ideas. It’s not a magic wand nor a design roulette that works on luck.
From the initial experience, a series of crucial realizations should follow:
- A/B tests are only as good as the hypotheses they test.
- Hypotheses are only as good as the research behind them.
- Sustainable business growth is only as good as the management of this entire process called CRO.
A/B testing is an indispensable component of CRO, but it’s only one component. See below:
This distinction between A/B testing and Conversion Rate Optimization is crucial. Conversion optimization is a process that needs to be repeated, but A/B testing is a technique — a statistical tool. Both these terms are often confused for good reason. For seasoned, successful testers, good tests are inconceivable without a good process.
2) You Need More than an A/B Testing Tool
We have already established that whether you talk about “A/B testing” or “Conversion Optimization” in your internal discussions, what you need in the end is a process. A systematic approach to hunt down barriers to conversion on your website and eliminate these barriers. Such an approach requires:
- Research to construct testable hypotheses
- Prioritization of hypotheses to decide what to test first
- Testing and Analysis of results to derive learning for future tests
There are multiple tools that will help accomplish specific aspects of this process. Also, there are project management tools that help you set up a process for integrating these steps.
Here is a list of the stages involved in making a sound business decision.
The objective of such a detailed process is to ensure “sound” decisions. These decisions minimize the probability of loss and maximize the probability of gain.
You need to perform all the steps at one place with unrestricted access to data collection, manipulation, and validation.
A system that enables you to make such decisions requires a platform with connected capabilities. Think of any marketing automation software such as Hubspot, Marketo, or Eloqua. These are all platforms with a network of tools that communicate with each other such as email, social media, content, and paid channels.
3) Decide What You Want to Know through A/B Testing
Till some time ago, most A/B testing tools ran tests the same way. The underlying statistical framework used to interpret numbers and choose a “winning variation” was the same. Currently, A/B testing tools and CRO platforms such as VWO offer you a choice. With choice comes the responsibility to evaluate pros and cons and then choose what best suits your needs.
The choices are:
- Frequentist Method of Computing
- Bayesian Method of Computing
Both these methods present different approaches to testing and ask different questions from the data they collect. Therefore, these methods fundamentally affect the level of optimization you can achieve.
Are you looking to find the infallible truth, or are you looking to make the smartest business decision?
What the image says: The former conclusion of 16.6% (5 times/30 days) that “Glasses” found out first is the Frequentist way to look at things. (The example is a bit wrong because 5 times is probably too low to be statistically significant, but it works to demonstrate the thinking.) The latter conclusion of 75% is Bayesian reasoning. This reasoning considers supplementary information (that Glasses complimented her eyes) that was available. Which one seems a smarter, yet very reasonable, conclusion?
Let’s consider that your team sets up a test between a control (A) and a variation (B).
Here’s what these two methods find out for you:
Frequentist Method
“What are the chances of observing a result at least as large as this, if the test was done between A and an identical copy of A and on the same sample size?”
This question doesn’t sound very straightforward, because it is not.
This method helps statisticians and truth-seekers, incrementally reduce the improbability of a wrong conclusion to help arrive at the “truth.” The Frequentist method was designed to understand the probability of drawing the wrong conclusion. It is a cautious, and somewhat slower, approach to arrive at “the truth.”
Let’s say the p-value for a test is computed as 0.03. When the test is stopped, the conversion rates of A and B are Ca and Cb.
Frequentist testing tools interpret this figure and display the result as “Chance to beat A is 97%.” However, what the p-value indicates is that if you had conducted the same test a 1000 times between A and a copy of A, at 30 instances, you would observe a result at least as extreme as the difference between Cb and Ca.
This is not the same as saying “B will beat A by a margin of Cb minus Ca 97% of the times.” All that the p-value represents is the improbability of A and B to be identical.
The p-value does not discuss the probability of the variations to be different, or by how much. This value also does not tell you what you should do, although these are probably what you want to know.
Bradley Effron explains the Frequentist vs Bayesian thinking in his presidential address at the American Sociological Association:
“The frequentist aims for universally acceptable conclusions, ones that will stand up to adversarial scrutiny. The FDA for example doesn’t care about Pfizer’s prior opinion of how well its new drug will work, it wants objective proof. Pfizer, on the other hand, may care very much about its own opinions in planning future drug development.”
This analysis does not mean that the Frequentist results have no meaning in a business setup. These comparative results let you to be fairly sure that one choice is better than another. However, that’s an indirect way of saying it and any attempt to understand “by how much” leads to hidden fallacies.
Bayesian Method
Bayesian lets you ask the question:
“What choice should I make right now to make the most money? Continue the test, stop the test with A, or stop the test with B?”
This is made possible because Bayesian approaches the outcome—“conversion rate”—as a fluctuating variable. With additional data that it gathers, Bayesian updates the probability of the outcome—the conversion rate—to lie between a range.
Chris Stucchio,
Former Director of Data Sciences at VWO, explains:
For example, before observing any data, one might think all conversion rates (for two variations A and B) are equally likely. At this time, both A and B would have a 50% chance of being the winner.
At this time, the financial risk is also high—it’s totally plausible that B has a 20% conversion rate and A has a 5% conversion rate, so choosing A would be a bad decision! The opposite situation is also very plausible, in which case B is a bad decision. At this point, we should just wait and continue the test.
Then when new information is updated (such as an A/B test), that opinion changes. After obtaining data, we become confident that B has a 90% chance of beating A. We also come to believe that A has a conversion rate between, for example, 3% and 6%, while B has a conversion rate between 5.9% and 9%.
At this point, we’ve eliminated the risk involved in choosing B. It’s still theoretically possible that A is better—A might have a 5.97% conversion rate while B has only 5.93%. But the financial cost of that possibility is very low, so choosing B is a very safe business decision. Not the truth maybe, but a wise business decision.
Bradley needs to be quoted again where he summarizes the differences in the approach:
The Bayesian-Frequentist debate reflects two different attitudes to the process of doing science, both quite legitimate. Bayesian statistics is well suited to individual researchers, or a research group, trying to use all the information at its disposal to make the quickest possible progress. In pursuing progress, Bayesians tend to be aggressive and optimistic with their modeling assumptions. Frequentist statisticians are more cautious and defensive.”
Most tools in the market make use of Frequentist, while some like VWO have graduated to Bayesian; still others use both in conjunction.
Chris Stucchio, explains on ConversionXL why Bayesian trumps Frequentist approach when it comes to making business decisions, where the onus is not on finding the ‘truth’ but on making the most rational business decision. “One is mathematical—it’s the difference between “proving” a scientific hypothesis and making a business decision. There are many cases where the statistics strongly support “choose B” in order to make money, but only weakly support “B is the best is a true statement”. B has a 50% chance of beating A by a lot (say B is 15% better) and a 50% chance of being approximately the same (say 0.25% worse to 0.25% better). In that case, it’s a great business decision to choose B—maybe you win something, maybe you lose nothing.”
On the business front, here’s how VWO SmartStats, VWO’s Bayesian engine fares against our older Frequentist engine.
If you are interested to learn more about the Bayesian approach, you can find a detailed and intuitive approach to understanding the Bayesian method here.
And here’s Chris explaining the inherent limitation of Frequentist and how Bayesian saves the day.
Download Free: A/B Testing Guide
4) You Will Need More than a Platform; You Need People
Someday, we might have platforms that don’t just enable to you to make decisions, but also make decisions for you. Until then, you need more than just platforms—a set of people that possesses a unique set of skills to use the platform and move the needle.
The skills that a CRO team must possess can be broadly grouped into the following:
Strategy Development
You will need a plan for your CRO program from determining the goals or key metrics of the program prioritizing activities such as hypothesis building to A/B testing to leveraging the learning to fuel further tests. Such ownership requires context about your business and at the very least, your organization’s conversion funnels.
Data Analysis
CRO is as much about being proactive as it is about reacting to obvious conversion issues. Data analysis involves monitoring of website data and user behavior to identify areas of improvement—leaks and areas of friction in your conversion funnel. A data analyst needs to make sense of CRO jargon and ensure that everyone in the team understands the test results in simple terms.
Conversion-centered Designing
Design is a crucial aspect of CRO, as it is the culmination of the proposed changes and acts as the interface between your understanding of your users and their interaction with your business. Such responsibility requires you to have knowledge of persuasive design principles and psychological triggers.
Persuasive Copywriting
Copy is the other element that your visitors interact with apart from design, and it requires just as much thought and expertise. Some key aspects of conversion optimization that persuasive copy can help with are reducing visitors anxieties, reflecting their motivation, and delighting them.
Web Development
You also need a web developer. Translating your design and copy into an error-free experience lies in the hands of this developer. From creating variations and setting up and tracking events and goals to transferring data from one system to another, the developer’s role in a CRO environment is indispensable.
To provide an in-depth and comprehensive account of what it takes to run a successful optimization program, check out our interview with Michal Parizek, senior eCommerce and Optimization Specialist at Avast.
Wrapping Up
A/B testing is a test; CRO is a potent growth discipline.
Here’s what we covered in this post:
- The difference between A/B testing and Conversion Rate Optimization
- For sustained growth, you’ll need a connected platform rather than an A/B testing tool
- Different A/B testing platforms answer different questions, choose what you need answered
- CRO doesn’t end with a platform, you’ll need people who can run CRO.
Currently, most organizations that believe they are doing CRO are only doing patchy A/B testing. With time, it will evolve into the discipline it needs to be. When it becomes that, where do you want to be—in the gallery or in the race?
Carpe diem.
Categories:
