How to conduct A/B testing
Colleague #1:“Lets change the layout of this landing page.”
Colleague #2: “What’s the reason for the change?”
Colleague #1:“Just because. Maybe it’ll work better.”
How many times have you had or heard a conversation along these lines? It’s the marketing equivalent of shooting in the dark and hoping for the best.
Spoiler: there’s a better way to make big business decisions.
A/B testing, also sometimes referred to as ‘split testing’, is a type of experiment used by marketers to gauge which variation of a campaign works best.
The concept itself has been around for a long time, but is particularly relevant in the worlds of email and web marketing. It’s an inexpensive and reliable method to really understand whatworks for the correct audience.
When it comes to what you can test the possibilities are almost endless, but some common examples include:
Landing page copy
Call to actions (CTAs)
Headlines
Product descriptions
Advertisement imagery and colors
Email sender names
Personalization options
The benefits of A/B testing
How many times have you or a colleague made a decision based on instinct, a gut feeling, or best practice? It’s impossible to predict how people will react, and A/B testing will help you remove some of that guess work.
If done right, it can give you tangible insights that increase web traffic and conversions -- and decrease bounce rates and missed opportunities.
If done wrong, inaccurate results can be extracted which can result in performance-damaging decisions being made. So, let’s make sure you get it right!
How to get started
1. Pick your variable
First things first, you need to pick which variable you want to test. It’s important to focus your efforts on only one element or change at a time, otherwise you won’t know which is responsible for your surge or slump in performance.
For example, if you were focusing on improving the conversion rate of an email and changed the color of your CTA and the template, how would you know which change was behind the results?
That doesn’t mean you can’t fiddle with various elements of a single campaign, it just means they can’t be measured concurrently. Sticking with the email example, you’d need to make your CTA change, analyze your results, take action accordingly, and then experiment with your template
2. Set your goals
Once you’ve settled on your variable, you need to set its primary goal. Let’s say your experimenting with the text on a landing page, your primary goal could be to either:
Reduce the page’s bounce rate
Increase the average time spent on the page
Increase the page’s conversion rate
3. Create your variables
Next up, you need to create your two variations. Your control version is either what you already have (i.e. current webpage copy) or what you’d normally use (your standard email template, for example).
Your second variation - the ‘challenger’ - is the same asset, but with the isolated change you’re looking to test. For example, let’s pretend your testing if changing the color of your ‘Buy now’ button increases conversions.
Your control version would be simply leaving the webpage unaltered, and your challenger version would involve using the exact same page, but changing the color of the button to X, Y or Z. It’s as simple as that.
4. Split your data
When it comes to splitting your data, it’s important to divvy it up equally - i.e. 50/50. The reason for this, is that it’s the quickest and most reliable way to get statistically significant results.
To make a decision on which variable is more effective, each has to be viewed the same amount of times. So, if you were to split your data 30/70 (with 30% going to your challenger version and 70% to the original), for example, it’d take much longer for your challenger to rack up the numbers needed to complete your experiment.
If you’re not sure where to start with this bit, here are some of the best rated A/B testing tools to help you:
5. Set your sample size
This one will vary depending on what you’re testing and which A/B testing tool you use. If you’re testing a webpage, for example, you might want to set yourself a target number of visitors to base your experiment on.
Or, if you’re measuring a social media ad, you might choose to set your campaign to run until each variation has earned 3,000 impressions, for example.
For help on how to determine your sample size, check out this guide.
6. Analyze your results
Last but not least, you need to take the time to carefully read and understand your results. During the analysis stage, it’s important to keep your primary goal in mind and not get distracted by other metrics.
By this, we mean if your main goal was to improve email conversions by changing your template, try not to get too caught up with things like open rate, click through rate (CTR), and bounce rate.
Important things to remember
Before you get going with your A/B tests, here are a few final points to bear in mind:
Run both your variations at the same time
Run your tests for the same amount of time
Only run one test at a time
Give your experiment enough time to produce meaningful results
Hue & Tone Creative: Your Marketing Partner
If you need help producing your A/B testing assets, you know where we are. To discuss your needs, goals and requirements, contact the team at (336) 365-8559 or hannah@hueandtonecreative.com.