Srikant Kotapalli
Dec 8, 2022
Srikant Kotapalli
Dec 8, 2022
A key performing index for any marketing campaign or digital asset is to understand how it is delivering in terms of actual conversions. For digital marketers, it’s imperative to rely on tools to optimize their campaigns and boost conversion rate (CR). There is no one true way to optimize CR, as the dynamics of every digital marketing asset is different along with varying needs of various customers which remains unique. In such a heterogeneous environment, the only ground-rule companies can follow is continuous experimentation and testing with tools such as A/B Testing.
A/B Testing has proven to be an effective data-gathering tool for companies, as they try to interpret how people behave when they land on a webpage, open an email or respond to a Facebook ad. It allows the marketer to experiment with various versions of their collateral to see which one performs better before they launch the actual campaign or marketing asset like a website, app, landing page among others.
In this blog, we will take a 360 degree tour of how A/B Testing – also called Split testing – works, how companies stand to gain by using this tool and how can companies conduct accurate and constructive A/B Tests. A quick overview:
What is A/B Testing or Split Testing
Why Companies Should A/B Test
Why A/B Testing Tools are Vital For Growth Managers
Step By Step Guide to Do A/B Testing or Split Testing
Examples of A/B Testing
How Insider Can Help You Start A/B Testing Marketing
Think of it as a contest where two players face-off to test their strength. You put two versions of your marketing asset against one another to see which one emerges the winner. Often A/B Testing and Split Testing are used interchangeably and rightly so, as the principles and objective of both remain the same, where you can consider the former to be a subset of the latter.
The players in the A/B Test arena are categorized as:
Challenger: When you start a test, you create a new version (called Variant) to challenge your existing champion page. This is the Challenger.
Champion (Winner): After running the test, the version with the best conversion performance is crowned the winner. It becomes the Champion variant.
In an A/B test, a web page or a mobile app screen is amended to create a second model of the same page. This can be a simple change in the headline or hero shot, or a complete redesign. Then, half the website’s traffic is shown the original version (the control) and the other half the modified version (the variation). Audience engagement is analysed and measured to find which fared better. The winning version is deployed as good-to-go and targeting the wider audience pool.
It is well established that even minor changes can impact conversion rates. The colour of the call-to-action button, for instance, is a hotly debated topic among conversion rate optimization (CRO) experts. Some say red works wonders. Others vouch for green. There could be other changes like the size of the CTA button or removing the banner from the top to changing form on the sign up page and then assessing performance. Or it could be a change in the Add to Cart/Purchase button’s text or even font to see if conversion rates increase.
A/B testing is not new and perhaps being used for experimental results in other fields since the 1920s, but brought to marketing in the 1960s and 1970s to evaluate direct response campaigns (source: HBR). One big benefit of A/B Testing is that it’s low on cost, but high on returns. There are several other reasons why companies should make A/B Testing a part of their testing portfolio:
Boost your bottom line: Controlled testing and analysis can determine what marketing strategies create high user engagement and work best for your product. In the current landscape of digital which is highly dynamic, continuous experiment will help maximize ROI.
Improve conversion rate: Any positive change – be it in the text or the CTA colour – can enhance user experience and garner more clicks. This would mean more conversions and more lead generation. E-commerce companies often find users abandoning items in their shopping carts. This trend can be reversed by A/B Testing.
Lower bounce rate: Are visitors leaving – or bouncing – from your website too quickly? A/B Testing can help make the right changes to retain them.
Solve pain points: Users may face common problem areas while navigating a web page. It may be a difficult-to-comprehend text or an inability to find the CTA button. A/B Testing helps figure out and resolve these pain points.
Go by statistics, not guts: Don’t rely on guesswork. Take data-driven decisions on which headline, image or video works best on your landing page.
Test new features for optimum outcome: Want to introduce a new feature change to your webpage? If you A/B Test, the outcome becomes predictable and value-adds to your page. This is especially relevant if the change affects customer data or the purchase funnel. A feature change done without testing can be a gamble – it may or may not pay off. Testing brings certainty to the outcome.
Manage a balance between the marketing team and developers: Since the changes for A/B testing do not require hard coding, it makes life simpler for both marketers and coders. Coders are not constantly making changes without any data to back that request for change. Marketers are able to manage variations all by themselves and optimize performance.
As innovative businesses find new ways to optimize conversion rates, A/B Testing has moved from being an occasionally used strategy, to a continuous, structured activity in a growth managers kitty. In fact, it is being used by both small size companies as well as enterprise to get higher conversions.
For best results, it’s ideal for marketers to work with service providers who specialise in this space. Running in-house testing, without the right technology back-up, can be a challenge for companies. The effectiveness of the tests would be lower, the processes longer, the analysis not fully reliable and the post-test actions taken may not be completely accurate. With a specialist tech vendor, companies can run 5 to 10 tests in a week. Alternatively, if they opt for in-house testing, they would probably manage 2 or 3 tests a month, at best.
When companies work with a reputed service provider, they don’t just gain access to technology – but also buy into their knowhow, experience, consultancy prowess, agility and benchmarks with other industries and countries. It is critical to select a tech provider who comes with all these capabilities and supports you in every step of your testing journey:
Coming to A/B Testing tools, there are tools galore in the market. As a growth marketer, if you were to go shopping, pick your tool with care. Besides prioritizing on effectiveness and ease-of-use, opt for a tool that best fits your business size, resources and learnings from past experiences with using A/B Testing tools.
There are few key features that fully-loaded A/B Testing tools come with:
A/B/n Testing: For multiple versions of a page. You may want to test more than just one version of your page – say, an A/B/C Test. Your tool must cater to this.
Multivariate Testing: For multiple elements on the same page.
Visual Editor: For building variations without any code
Full Stack: For A/B Testing both web pages as well as mobile apps.
Autopilot: Automatically manages traffic, to shift more traffic to the pages that are performing better.
Heatmaps: To decipher how visitors are actually using a webpage.
From start to finish, here are the steps to follow:
Step 1: Do your homework
Know everything about your website’s current performance – how much traffic it generates, the most checked-out pages, conversion rates among others.
Step 2: Get an A/B testing tool
Find an A/B testing tool that is comprehensive, intuitive and easy to use. The tools should be designed specifically for the Web, Mobile Web and Mobile App Platform. Keep a checklist ready and see if it matches your requirements.
Step 3: Create a challenger
As you work on optimizing your web page, they may be many variables you want to test. Pick one or two at a time. Otherwise, you won’t know what worked. Build a Variant or Challenger page incorporating this change.
Step 4: Split your sample group or wait for the numbers to add up
Split your sample group equally and randomly if you’re testing an email. If you’re testing a web page, there’s no fixed group. In that case, let the test run its course. Let it be long enough to clock in a definitive number of views.
Step 5: Run the test. Test both variations at the same time
If you test Variant 1 today and Variant 2 a week later, you’ll be left guessing whether your web page’s performance change was driven by design or timing.
Step 6: Measure the test results using an A/B Testing Calculator
Measure which Variant emerged the winner. You can either do this manually or feed the results into an A/B testing calculator. Deploy the winning variation.
A/B Testing allows you to test almost anything in your marketing material. But since time and resources are limited, it’s best to test the most impactful elements. A checklist would comprise:
Test Hypothesis: Cart Page With Top Menu Bar vs. Cart Page Without Top Menu Bar
Let’s take an example of an e-commerce company. It is a new entrant in the market, has just launched its website and is looking at attracting visitors to its webpage to drive sales. E-retail is a crowded marketplace and the company wants to create a website that stands out and garners eyeballs.
The current version of the company’s cart page comes with a menu bar running across the top of the page. The bar lists all the product categories that users can browse through before they arrive at a purchase decision. It is a handy tool, but is it making the page look cluttered? The company decides to find out.
It runs an A/B Test, testing two versions of the cart page – one with the menu bar and one without. The results start flowing in over the next three weeks, as visitors are randomly assigned to visit one of the two pages. The KPI being measured is Purchase performance and basis the data, the company can take an informed decision. Also, going forward, it can apply this insight on future cart page designs, avoiding using menu bars.
When this e-commerce company launches a mobile web application, it uses A/B Testing again to find the best fit Call to Action button. In the original version of the webpage, the CTA button read `Buy Now’. It’s an unambiguous CTA, but the marketing team wants to test it against something more subtle. So it sets up an A/B Test, where two versions of the web page are built – one with a `Buy Now’ CTA and another with a `Cart Icon’ CTA.
The results provide the company an invaluable insight into consumer psyche – whether subtle works better than a direct call to buy, again the measurement metric is purchase performance.
A/B Testing can help companies make important decisions on what visuals or texts appeal to consumers on the mobile application platform as well. Let’s take an instance. A new food delivery company launches an app and begins its marketing campaign, to get consumers to download it. However, two months down, it finds that traction isn’t high, although the demand for food delivery is spiralling.
The company decides to go back to the drawing board to see if it can make any changes to its application to attract more downloads. The app currently has a text based on-boarding process. Are consumers finding it too boring to proceed with? The company decides to A/B Test to find answers. An alternate, banner based on-boarding process is created for the app. The two can be tested for registration conversion and see which one performs better.
Insider has worked with companies across the board, to help them test new features and add-ons on their webpage and mobile apps, increase conversion rate and deliver unique user experiences.
Partnering with Insider goes beyond a testing tool. The partner organization benefits with strategy, expertise and Account Management. They also tap into Insider’s multi-dimensional experience, knowhow, consultancy expertise, acumen and benchmarks across industries. Every step of the journey – from ideation and strategizing, to implementation, analysis and iteration – is covered.
Let’s look at some examples.
airBaltic, a Latvia-based regional airline, came onboard with Insider, to use its A/B Testing tools to optimize user experience on its digital channels. Here’s how it benefited:
Goal 1: Build a unique experience for each traveller on their digital channels.
Goal 2: Improve main KPIs like conversion rate and average session duration on their website.
The A/B Test process:
– Personalize the home page for different audience segments.
– A/B/n testing the home page where it used 3 different variations, also keeping the original version.
Test result: The 3-week experiment showed an alternate page was performing better than the original, both in terms of user experience and average session duration.
Number crunching: A conversion rate uplift of 5.8% on the winning variation.
(https://useinsider.com/case-studies/airbaltic/)
A Dubai-based property developer, First Group, worked with Insider to A/B Test its investment guide:
Goal 1: Get visitors to their website, so they can start talking with the brand. Get the visitors to engage with the content. Nurture leads.
Goal 2: Get visitors to download a copy of their investment guide.
Goal 3: A/B Test notifications for better conversions.
The A/B Test Process:
– Personalize all pages of the website.
– Send notifications to visitors, suggesting they download the guide.
– A/B Test five different messages, sent to different portions of visitors.
Test result: Substantial increase in conversion rates for customers who received notifications to download the guide.
(https://useinsider.com/case-studies/the-first-group/)
To make the process seamless, Insider has two platform-specific tools, designed to optimize A/B Testing on the Web and on Mobile apps:
Content Optimizer:
Built for mobile apps, this tool A/B Tests in many interesting ways:
Onsite Experiment:
Catering to Web platforms, Onsite Experiment enables businesses to make changes on the elements, test new ideas and improve user experience.
Its feature portfolio include:
A/B Testing makes one thing crystal clear to marketers – you don’t know what works with your end-user until you test. A/B Testing can safely stand for, `Always Believe in Testing’.
If you want to start A/B Testing on your web/mobile page or mobile application to optimize conversion rate and enable users to have a smooth navigation experience, we can help. Talk to our experts, request a personalized demo.
Written by
Srikant Kotapalli
With 12+ years of experience in consulting, building, and marketing technology products for clients across industries, Srikant is a product leader, storyteller, data fanatic, and UX/usability enthusiast. He often appears as a speaker on panels about personalization and optimization and has a passion for building simple solutions to complex problems, and is currently pursuing that at Insider.
