A/B Testing Guide: How To Perform AB Testing

In this A/B Testing (aka split testing or bucket testing) guide, we will discuss everything you need to know about AB Testing.

What is A/B Testing

A/B testing can be defined as comparing two variations of the same webpage with two sets of different visitors to see which one performs well. The comparison does not stop with just web pages, it can be app interface, landing pages, email marketing, or any other marketing assets.

The original design (A) is compared with the alternative design (B) to see which design converts well. The original design is called “Control” and the alternate version is called “Variation”.

A/B Testing Terminology

While conducting the A/B testing we obtain the data/statics from champions, challengers, and variations. These versions provide insight into your visitor’s conversion rates.

#1. Variant

For running A/B testing you would need different versions of web pages or any other marketing asset, there would be multiple forms of the same page with slight changes, these are called variants.

#2. Champion

Champion would be a web page that performs well or has performed well in the past. Usually, the champion would be tested against other variants and the variant with higher conversion rates becomes the champion variant.

#3. Challenger

Challenger is a new variant of your web page that is compared against your existing champion.

#4. Traffic

It refers to the users visiting your website, it is measured by the number of users spending time on your website.

#5. Conversion Rate

Conversion rate is the percentage of users who take the desired action that is intended in the website, like sign up for an email list to pay for a product divided by the number of visitors.

#6. Conversion Rate Optimization

Conversion rate optimization (CRO) is the practice of optimizing a website or landing page experience based on the behavior of website visitors. It helps a site owner to understand how the website visitors are taking desired actions (conversions) such as clicking on ‘add to cart’, purchasing a product, signing up for a service, filling out a form, or clicking on a link, becoming customers on the appropriate page and what’s stopping them from completing their goals.

#7. Hypothesis

A well-structured hypothesis will provide you insights into whether it is successful, failure or results are inconclusive. Ex: More people will click the button if it is blue because it contrasts with other colors on the page.

#8. Insights

It is a combination of reasoned ideas and conclusions established based on systematically collecting and analyzing data.

#9. Copy

Copy, also known as ad copy or sales copy is written content that aims to increase brand awareness to persuade an individual or group to take a particular action.

How A/B Testing Works

Basically, you would take a web page or any other interface that you want to and create another version of the same page. The change is as minor as changing an ad copy or a headline or changing the color of a signup button or it can be a complete rebranding of your landing page.

After creating that variation, divert half your traffic to the control i.e the original version (A), and the other half to the variation i.e the modified page (B). Observe the insights and conversion rates on your website.

The result determines the winning version out of the original version and modified version based on the most opens and clicks.

Why You Should A/B Test

• A/B testing can improve your landing page and increase your conversion rates through trial and error methods without spending too much time and effort.
• It helps you make changes in the interface and collect data on the user experience, so you know what works and what doesn’t.
• You can test one change at a time to which changes affected their visitors’ behavior, and which ones did not.
• You can implement these multiple successful changes and experiment with other parts of your website too.

How do you Perform an A/B Test?

A/B testing starts with a goal like get more sales, increase conversion rates, and more. So to test a different version of your website, you have to keep a measurable goal, an appropriate hypothesis, allocate your traffic between 2 different versions, and proper sample size.

#1. Analyze your insights

You should start evaluating how your website performs before you start your A/B testing. Start by collecting data on how many users visit your website, which page drives more traffic, what are the conversion goals for various pages in your website etc. You can use analysis tools like Google Analytics, which can offer you reports on how your site performs.

Collect data like pages that have low conversion rates or high drop-off rates

#2. Identify the end-goal

Create a specific goal for improving your website, make sure your goal is time-bound. Your goal can be anything: clicking a button, signing up for an email subscription, following you on your youtube channel, starting a free demo, more social shares, even paying for a product or service, etc.

By the end of the testing, you should have metrics on whether the variant improved your conversion rate or not.

#3. Create a variant

You have to focus on creating a variant for your existing control. Make sure you change a single component so that it is easier to measure if it performs better or not. You can change your headlines, calls to action text, call to action location, pop up, featured images, copy, the number of fields in a form, etc. Don’t forget to test your website while integrating a new variant.

You can create multiple variants and test them against your original webpage to see what works well.

#4. Run the test

Start running your test by dividing your traffic into two different variations of your website. Make sure that you’re not interfering with the traffic that reaches your page while experimenting with variants. Based on the average daily & monthly visitors, estimate the existing conversion rate along with the expected minimum improvement to calculate how long you can run the test.

Each user interaction with the web page should be measured, counted, and compared to determine how each performs.

#5. Interpret the results

Compare the result of how well the control and variant have performed. Check whether there is a statistically significant difference, i.e conversion rates between a given original page and the variant is not due to random chance. You will arrive at a data-driven conclusion.

Do not stop with a single test, experiment with more components, A/B testing is efficient when it is done continuously.

What Is Multivariate Testing

• In MultiVariate testing multiple elements in the page are changed in combination, it is compared with the current version of the web page.
• The objective of conducting multivariate testing is to measure the effectiveness of each design combination.
• Multivariate testing provides faster results but it can be complex for a beginner.

What Is Split URL Testing

• In Split URL testing, you create two or more variants for your webpage.
• Then testing those multiple versions of your website in different URLs.
• Here the variants are fully developed web pages and stored on the server, which is accessed via different URLs.

What Is Multi-Page Testing

• In Multi Page testing, we experiment by changing a particular element across multiple pages.
• Here you have to show users a mix and match of variation instead of showing consistent variations through a set of pages.
• So we can test one variation against another thoroughly.

A/B Testing vs Multivariate Testing

A/B Testing vs Split URL Testing

What Can You A/B Test

Remember that every element and content in your website contributes to your conversion ratio. So every piece of content in your website should be optimized to its maximum potential. This can help your business to make more sales and revenue. Conversion optimization is not easy as you think it is. Each and every element in your website translates to your conversion ratio which will directly impact your sales funnel.

Here is a list of things that can be used for variations in A/B testing:

#1. Landing pages

A landing page is a web page that a user lands after clicking on an ad, a link in your email campaign or any other digital location. Usually, these landing pages have a clear call to action (CTA), to buy a product or just joining your list.
Your Landing page needs to be optimized to convert users on whatever offer you present them with. Use a Heatmap ie.a visual representation of attention, engagement, and interactions generated by your visitors, before running your A/B testing to collect data and hypothesis, this helps you to focus on what needs attention.

#2. Headlines

Headlines can be very crucial, as it is the first thing any user would look at when they enter into your website. If your headline doesn’t catch their attention, don’t expect them to stay longer on your website. You have to be careful while crafting your copy, font, size, color, and message.

#3. Page layout

Your page layout should be simple yet powerful for your visitors to access it. A proper design and layout with clear information, straightforward content, a clear call to action, along with a media that enhances the visual aspect of the page would be enough.
Whether it’s your home page, landing page, blog post, or product description page make sure the UI/UX design is clutter-free and focuses on the conversions.

#4. Navigation

A visitor should navigate through your website seamlessly. They should not be overwhelmed or confused by different elements in your website. Place a navigation bar in the top left corner and your site’s logo in the top right corner, clicking on that should navigate back to the home page.

Follow these basic formats that most websites use so that your user quickly finds what they are searching for. The worst-case scenario here would be your visitor getting lost in your website.

#5. Forms

The form is a way of getting information about your prospect. Forms are the best way to get in touch with your potential customer. Make sure it is in the right size, if it’s too long, your visitor might just abandon it, if it’s too short, you may not gather the information to verify visitors are potential customers.

#6. Page length (Content depth)

Some visitors would like to have crisp and consumable content whereas some visitors like deep diving content. With A/B testing you can figure out the preference of your target audience and cater to their needs.

#7. CTA (Call To Action)

Call to Action is the single most important thing that directly impacts your conversion ratio. CTA prompts your visitor to take any action- to buy your product/service, subscribe to your email list, listen to your podcast, it can be anything. But you should have only one CTA on your page so that your visitors won’t be confused.

#8. Media

Media plays a huge part in your sales funnel. You may have a youtube channel, podcast, Instagram, etc that directs your audience to your website. These image/audio/video formats of content can give you insights on how many people in your audience checked your website and with that traffic how many got converted to your customer. So you can optimize your content which would drive traffic to your website.

What are the Mistakes to Avoid While A/B Testing

#1. Expecting big wins from small changes

One optimized copy in your webpage might increase your conversion rate but you should not expect $ 1000 revenue just with that.

#2. Testing too many items at once

You can get a proper insight into your conversion rate only when you test it with one or two elements.

#3. Not understanding Statistical significance

It is solid evidence that your change created to increase or decrease in conversion happened because your control and variation aren’t due to error or random chance.

#4. Running your test for an incorrect duration

Your testing timeline should be based on your traffic, goal, and hypothesis. It shouldn’t be too long or too short.

#5. Not following an iterative process:

You should continuously test different elements of your webpage, experiment on things that need more attention and, build upon the best result.

#6. Changing parameters in the middle of testing

This will be counter-intuitive for your testing and will end with the wrong data if you change your goal, hypothesis, variant or any other aspect of your testing.

What are the Challenges of A/B Testing

• Not having enough traffic on the website to conduct A/B testing.
• Testing based on your assumptions can bias the result.
• External factors can tamper with your results.
• It is hard to continuously invest in A/B testing.
• Having incorrect indicators of the variant as success.
• It is a one size fits all approach, it doesn’t cater to all your visitors.

A/B Testing Process

#1. Define a clear goal for your conversion rate optimization (CRO).
#2. Create two variations of the same element – landing page, ad copy, etc
#3. Calculate how long you are going to run the test. It is recommended at least two weeks once but it is completely dependent upon your traffic, goal, and other factors
#4. Select a testing tool that would help you run your test.
#5. Launch your A/B testing on your website and wait.
#6. After your assigned timeline of testing, analyze your result, check which version performed better
#7. Formulate a new hypothesis and come up with another element to test.
#8. Understand that A/B testing is most effective when done continuously.

A/B Testing Best practices

#1. Focus on optimizing the pages people visit most

According to Hubspot, visitors hang out on our Home page, About page, Contact page, and Blog page. So make sure that you focus on the conversion rates on those pages. You can also focus on the high conversion areas like the lead magnet page, landing page, and signup forms.

#2. The sample size is a prime factor in your testing

Not all marketers are good with statistics, you should have the right sample size for accurate results. To determine the statistical significance of the project you should have the right sample size.

#3. Have a proper pre-planned schedule

Your test data should not be affected by the peak seasons like holidays or sales. Check how your traffic has been for the past few months, understand the pattern and launch your test in the ideal period where external factors don’t tamper with your data.

#4. Never make changes during mid testing

If you are running your A/B test, forget about your conversion ratio and optimization factors till your test ends. You might be tempted to change your goals or variant, looking at the result of a few days. Be patient.

#5. Have a thumb rule of testing one element

Don’t test many variants at once in A/B testing. It takes a single piece of copy, calls to action, headline even an image that can impact your conversion rate. Having multiple items to test might not provide you with the data on what works and what doesn’t.

A/B Testing Examples

Example 1: eCommerce website- Wall Monkey

About the Company: Wall monkey is an eCommerce brand that sells wallpapers and stickers.

Goal: To optimize the home page (which contains a stock image, search bar, and a headline) to increase clicks and conversion.

Variation: They changed the image on their homepage to an alternative that would show visitors the opportunities they could enjoy with WallMonkeys products and replaced the headline slider with a search bar.

Result: Conversion rate increased up to 550 percent

You can checkout a depth case study Wall Monkey here

Example 2: SaaS website – WorkZone

About the Company: It is a web-based project management and document collaboration tool

Goal: To increase the number of demo requests by reducing distracting elements on the form page.

Change: They changed their customer testimonial information and logos in black and white.

Result: It increased the conversion rate to 34% with a statistical significance of 99%

You can check out a depth case study of WorkZone here

Example 3: Media Company – Electronic Arts – SimCity

About the Company: Electronic Arts is a popular media company that develops and delivers games, content, and online services

Goals: To increase sales and maximize revenue by promoting its pre-orders for their new game ‘SimCity’.

Change: First they created a banner for pre-order incentives(control version), with the thought that incentives might increase sales. It didn’t go well with the users and the company didn’t get pre-orders as they expected. So they decided to remove the $20 off promotion banner from the page. 

Result: The new variant increased the sales by 43.4 percent.

True fans of the game were not interested in the incentive, they just wanted to buy the game. A classic example of assuming the target audience’s behavior.

You can check out a depth case study of Electronic Arts here

Best A/B Testing Tools

There are many A/B testing tools on the market right now. Some of the market leaders, including Google Optimize, Optimizely, VWO, etc.

#1. Google Optimize

Google Optimize is a free split-testing tool from Google. It helps online marketers and webmasters to test variants of web pages and see how they perform against the goal they set.

Some of the key features of Google Optimize are A/B testing, A/B/n testing, Multivariate testing, Split URL testing, etc.,

#2. Optimizely

Optimizely is a cloud-based marketing solution that provides digital experience solutions. It is designed to help web developers with A/B and Multivariate Testing.

Some of the key features of Optimizely are Multi-page testing, Advanced targeting & segmentation

#3: VWO

VWO is a web testing and conversion optimization platform to analyze web visitor activity and increase conversions.

Some of the key features of the VWO are A/B Testing, Push Notifications, Website Heatmaps, Conversion Rate Optimization, Server Side Testing, Web Form Analytics, etc.,

Conclusion

A/B testing is very powerful in collecting information about your visitor’s behavior like what ads drive the most conversions, what offers your audience responds to, or what blog headlines drive the most traffic.

It is recommended to execute A/B tests constantly on websites. You can use multiple tools to get started with your A/B testing journey which we mentioned above. 

Related posts:

• Software Testing – In-depth Guide
• eCommerce Testing Guide
• Product Testing Guide
• Monkey Testing Guide
• Usability Testing Tutorial
• Accessibility Testing Guide
• Adhoc Testing Tutorial