What is A/B Testing | Maxymiser

What is A/B Testing

A/B Testing – The Complete Guide

A/B testing is one of the easiest ways to increase conversion rates and learn more about your audience, but it is still underutilized by many otherwise very switched on, online marketing professionals. This is often because any form of testing is wrongly assumed to be very technical, time consuming and difficult to implement; however this is just not the case. When you consider what a key part of the sales equation conversion rate is and how valuable customer insight can be it is surprising that so many people ignore such a simple form of testing.

What are A/B testing and conversion rates?

A/B testing does pretty much what the name suggests, it tests a control, version A, against a different version, version B to measure which is the most successful based on the metric you are measuring.

In the online world A/B testing allows you to split traffic on your website so that visitors experience different web page content on version A and version B of a page while you monitor visitor actions to identify the version that yields the highest conversion rate. A conversion rate is the rate at which visitors perform a desired action on your site. By testing with live visitors on your site you learn from real users which experience they prefer.

AB Testing

Through A/B testing you also learn about the visitors themselves, such as visitor segments which consistently perform better with specific content. For example, new visitors may prefer different content than repeat visitors.

A page where traffic is sent to, to perform a specific conversion or call to action is often called a landing page; sales and lead generation landing pages are often the target of A/B tests although it is possible to test any page so long as you have a conversion metric to measure. For example you may be looking to increase the stickiness of a content page where you have a high bounce rate (visitors leaving the page directly after landing on it without visiting any other pages on your website). In this example you could set up a measurable goal or conversion where visitors view at least one other page after landing on your content page.

A/B testing, also referred to as split testing, starts with an hypothesis of the types of content changes that could impact your conversion rates. For example will a download button result in more downloads than a download link. The different web page content, or variants, is configured for a test and traffic is split between the variants. The test results indicate the conversion rate of one variant over another and are monitored until a statistically sufficient number of visitors have been included in the test.

Conversion rates and what to measure

To perform an A/B test you will need to measure a conversion rate; the objective of the test being to increase this conversion rate. The most obvious form of conversion rate is sales and can be worked out as the number of sales per 100 visits; so if you average 2 sales per hundred visits your conversion rate is 2%. Raising this conversion rate from 2% to just 2.5% would mean a 25% increase in sales, when viewed this way conversion rates really should be something worth paying a lot of attention to.

Conversion rates can also be measured in terms of revenue. Instead of the number of sales you can measure the impact of a change on sales revenue.

Revenue Report

However conversion rates can be any measurable action and are not just restricted to ecommerce sites and sales. Conversion rates can include:

  • Sales
  • Leads (e.g. booking a test drive or requesting an information pack)
  • Newsletter sign ups
  • Clicking on revenue generating banners or affiliate links
  • Spending a minimum amount of time on the site (this is great for detecting low quality pages where visitors are not engaged)

What to A/B test

Once you have decided what conversion rate you want to improve the next stage is to work out what to change on the page to try and increase conversions. Look at the various elements you have on the page in question that would be changed, these may include:

  • Headings – size, color wording
  • Images – placement, different images
  • Content – amount, wording, font, size and placement of content on the page
  • Call to action buttons such as: buy now, sign-up and subscribe buttons can be different sizes, colors, in different places on the page and have different wording.
  • Social media buttons – placement, size and wording are all worth testing
  • Logo and strapline
  • Use of trade association and online trust seals such as VeriSign

Setting up a test

  • Decide what to test
  • Set up two designs you want to test (one is the control which is often the original version)
  • Don’t be afraid to be bold and test big changes to start with
  • Ideally you want a 95% confidence level that your test is statistically significant
  • Chose a tool to run your tests or work with a professional company specializing in testing

How many conversions should I test and are my results accurate?

Statistical significance is reached when your test has matured and is accurately telling you which version (A or B), is best. This depends on the confidence level and the higher the confidence level the higher the chances are that your test results are accurate and not random. A/B testing is often aimed to deliver a 95% confidence level meaning that there is only a 5% chance the results are random.

A/B Testing platforms will set the confidence level and highlight when the test has reached maturity and is statistically valid. The confidence level may be automatically lowered if the test will take an enormous amount of time to reach maturity at that level.  In this case results are reported which have reached statistical significance but with a lower confidence level.

It is important to remember that having a high volume of traffic to the target pages does not necessarily mean you will have enough data for accurate results; the important metric is not the amount of traffic to the pages you are measuring but the number of conversions and the difference in the number of conversion between versions A and B.

What tools and services can I use to A/B test?

Those on a budget may be drawn towards a free way to test using Google’s Website Optimizer.  However, there are hidden costs with anything that is free, and those costs impede your ability to test. The solution puts a burden on IT to ensure each test is set up on your site, and you then also wait for the next opportunity to update the site so that you can start your test.

Maxymiser has been a market leader in A/B and multivariate testing for years, offering a platform that requires only a single line of code on your site to enable your marketing team to test anything, anywhere and without the need for IT resources.  In addition, Maxymiser’s services team will guide you through the process, starting with a clear optimization strategy that is aligned with business objectives and set of measurements, through to developing and executing on a roadmap of campaigns. Maxymiser provides all of the resources needed throughout the process or can train you to fill some or all of the roles. Having designed and run a huge number of tests with a wide range of different types of sites in different industries we can apply years of experience to help our client attain the biggest increases in conversions.

Conversion Report

Maxymiser Testing Case Studies

After a site redesign a few years ago Fragrance Direct found that their conversion rate had actually dropped. A key page for any ecommerce site is the check-out page which was identified as any area where with good potential to improve conversions and sales. Here’s how the original page looked before any changes.

Original Version of the checkout page:

Checkout Page

By keeping prominent images ad information on the product even at this late stage in the buying process we were able to maintain visitor excitement about Fragrance Directs products, this resulted in a huge 23% increase in orders placed on the winning check-out page.

Winning page with a 23% increase in orders

Winning Page

A/B testing is limited to single elements of a page and often tests are far more extensive than what a simple A/B test should handle. If you’re changing multiple parts of a page or testing a process that involves multiple pages, then Multivariate Testing is used to determine the combination of changes that yield the highest conversion rate.

Ready to start A/B Testing

Take the first step and speak with Maxymiser about how we can help you achieve your A/B testing goals.

Let's get in touch

Fill out the form below and a Maxymiser optimization expert will contact
you within one business day.


Give us a call

US:+1 212 201 2359            DE:+49 (0) 211 66966 0

UK:+44 (0) 203 375 0100    FR:+33 (0) 176700210