1. INTRODUCTION
A/B testing refers to a randomized experimentation process wherein two or more versions of a variable are tested, to understand which version results in the maximum desired impact. This testing methodology traces its roots back to the 1920s, when a biologist and statistician Ronald Fisher, performed various agricultural experiments which laid the foundations for what is now known as A/B testing. Some of these experiments analyzed what would happen if different fertilizers or different quantities of fertilizers were used in agriculture.
In later decades, this testing approach was adopted for various sectors including marketing and advertising. While sending out mail-order brochures and postcards, marketing companies used different variations to see which of them resulted in more calls and sales. A/B testing, which is also known as split testing, if often used in political campaigns, as well. For Barack Obama’s 2012 re-election campaign, his team used various A/B tests to eventually increase the online donation conversions by 49%, but also boost online sign-up conversions by 161%.
The common misconception is that for online businesses A/B testing revolves around optimizing the layouts, size, position and colors of webpages. However, the applications for online businesses are far more-wide ranging and impactful than what most people expect it to be.
For example, game development companies use A/B testing to validate different hypotheses like “How frequently should in-game advertisements be run?” or “What should be the price of in-game bundle purchases?” or “What sort of in-game rewards lead to higher customer retention?”.
The online streaming service Netflix has long used A/B testing to drive the engagement levels with their customers. Back in 2017, when the company moved from the user star rating system to the thumbs up and thumbs down system, the company found that the ratings activity went up by a massive 200%. More recently, Netflix found out through surveys that this approach of asking users to rate shows on a thumbs up or a thumbs down basis, wasn’t proving to be as effective as they anticipated it to be. In order to understand a better way of engaging with the customer, the company used extensive A/B testing to find that a two-thumbs up rating system led to quantifiably higher engagement levels, as compared to a hearts-based variant of rating shows.
2. WHY A/B TESTING
A/B testing’s rise to prominence has stemmed from the realization that as competition increases, so does the cost of customer acquisition. Consequently, it has become imperative for all online businesses such as eCommerce, travel, SaaS, education, media, and publishing businesses to optimize their customer conversion rates (the number of conversions divided by the total number of visitors). The primary objective of all these businesses is to attract visitors on the website who would ultimately transform into actual customers.
The key challenge for most of these businesses lies in accomplishing more with the limited resources at their disposal. This translates to enhancing website conversion rates while maintaining a steady flow of traffic, rather than perpetually increasing traffic without addressing the conversion rates. However, A/B testing goes beyond merely mitigating the repercussions of continually rising customer acquisition expenses.
The major organizational advantages of using A/B testing are:
2.1 Shortening your decision-making process
In the e-commerce industry, customer buying habits change rapidly and competition is fierce. Organizations need to make decisions quickly in order to stay competitive and have to adapt to frequent changes in their markets. Since A/B testing is entirely data-driven with no room for guesswork, gut feelings, or instinct based decision making, companies can quickly determine a “winner” and a “loser” based on statistically significant improvements. Measuring metrics like time spent on the page, number of demo requests, cart abandonment rate, click-through rate, and so on prove the success or failure of a planned change.
2.2. Get better RoI from existing traffic
As most experienced optimizers have come to realize, the cost of acquiring quality traffic on a company website is huge. A/B testing enables companies to make the most out of their existing traffic and helps increase conversions without having to spend additional dollars on acquiring new traffic. A/B testing can result in high Return on Investment (RoI) as sometimes, even the minutest of changes on a company website can result in a significant increase in overall business conversions.
2.4. Adopts a RoI focused approach
While A/B testing is a valuable method for understanding the preferences of visitors on company websites, it’s not the only available option. Techniques like user panels, focus groups, and eye tracking can also offer crucial insights. However, A/B testing uniquely enables companies to interact with their visitors in authentic settings. It enables the gathering of unbiased data about their preferences, both conscious and subconscious, without the customers being aware that the company is collecting essential behavioral insights.
When choosing between different versions of a website, there’s a long history of examples where the choice that seems obvious doesn’t always work out. In the Netflix two thumbs A/B testing, many within the company expected that the Hearts option would prove to be a lot more popular among users than the two thumbs up variant. However, the extensive tests led to an unexpected data driven outcome.
One of the most important metrics to track a company’s website performance is its Bounce rate (the percentage of visitors that leave a webpage without taking an action, such as clicking on a link, filling out a form or making a purchase) There can be many reasons behind a website’s high bounce rate, such as too many options to choose from, a mismatch of expectations, confusing navigation, use of excessive technical jargon and so on.
Since each website serves different goals and caters to various customer segments, there is no one-size-fits-all solution to reducing bounce rate. However, running an A/B test can assess multiple variations of an element of the website until the best possible version is identified. This not only helps companies identify friction and visitor pain points but also helps improve the overall experience of website visitors, making them spend more time on the site and possibly even converting them into paying customers.
Making minor, incremental changes to a web page with A/B testing, instead of getting the entire page redesigned can reduce the risk of jeopardizing the current conversion rates. An example of this is product description changes on a website. Companies can perform an A/B test when planning to remove or update the existing product descriptions. Another example of a low-risk modification is the introduction of a new feature change. Before adding the new feature, performing an A/B test can help companies understand whether the new changes will help the website users.
3. KEY ELEMENTS OF A/B TESTING
There are three dimensions that companies use to create and test new variations. Some companies tend to focus on just one dimension, but market-leading conversion rates can be achieved by mastering all the following dimensions. They are,
- Value Proposition: Is it easily understood, logically structured, and consistently presented across pages? Should the emphasis be on product quality, benefits, or price?
- Information Clarity: Does the website provide the right level of detail? Should there be more information, or should it be further streamlined to what is essential?
- Presentation Format: Would a concise video be more effective than a lengthy one?
- Product or Service Impact: Is one product or service overshadowing the sales of another? Is the pricing aligned with the market?
- Guiding Customers: Are visitors directed and reassured by appropriate messages? Are promotional offers prominently featured?
- Design, placement, and overall appearance of text, text blocks, and CTAs
- Wording, especially in CTAs
- Forms
4. HOW TO APPROACH A/B TESTING
Since companies need to have a flexible approach for A/B testing, it is vital that you start with the right toolkit. Interestingly, not all A/B testing solutions are the same. Some are complex and only used by experts. Your company needs to carefully choose a testing platform, especially if you are starting out with A/B testing.
4.1. Prioritize the right tests
A common challenge with A/B testing is the inclination to begin with complex scenarios like multivariate tests (Multiple sections are tested simultaneously, and one or more variations are created for each section) for page redesigns. It is advisable to start with simpler tasks before tackling more intricate ones. Specifically, your company should focus on tests that offer realistic improvements in conversion rates and are easy to implement.
A simple test typically entails minimal alterations to the design of one or more pages. With an efficient A/B testing tool, changes like adjusting text, colours, product images, button sizes, rearranging elements, linking to a new page, or creating banners can be executed by anyone without specific technical expertise.
An advanced test doesn’t involve major functional changes but encompasses more intricate design adjustments. This can involve tasks like shifting from a three-column layout to a two-column layout for product listings, reorganizing a navigation menu, removing fields from a form, URL adjustments etc.
Complex tests often necessitate significant functional changes and demand proficiency in web technologies, particularly JavaScript. Examples include transitioning from page redirection to an overlay window with a pop-up, redesigning product data sheets, breaking a form into multiple steps without a URL change etc.
A/B testing is a part of a wider holistic Conversion Rate Optimization (CRO) (the practice of increasing the percentage of users who perform a desired action on a website) program. An effective optimization program involves analysing your existing website data and gathering visitor behaviour data, then moving on to preparing a test backlog (a list of tests that are planned to be carried out in the future) of action items, further prioritizing each of these items, running tests, and then drawing insights. To achieve this, you need to make an A/B testing calendar which consists of the following:
This includes measuring your website’s performance in terms of how visitors are reacting to it. Your company needs to figure out what is happening on the website, why it is happening, and how visitors are reacting to the webpages. Everything that goes up on the website should correspond to your company’s business goals and Key Performance Indicators (KPIs) of the website. Additionally, you need to collect and understand you visitor behaviour data. Based on this, your company can prepare the test backlog. With a data-backed backlog ready, the next step is formulating a hypothesis for each backlog item. For example, after analysing the data gathered using quantitative and qualitative research tools, you can conclude that not having multiple payment options leads to maximum prospect customers dropping off on the checkout page. So, the hypothesis is “Adding multiple payment options will help reduce drop off on the checkout page.”
While the test is running, make sure it meets every requirement to produce statistically significant results before closure, like testing on accurate traffic, not testing too many elements together, testing for the correct duration, and so on.
- A particular variation won with statistical significance.
- The existing option was the better version and hence, won over the variation/s.
- The test failed and produced insignificant results.
In the first 2 scenarios, you should not stop testing just because there is a clearcut winner. Your company has to make further improvements to the versions and continue the testing. In the 3rd scenario, you need to identify where the process went wrong and re-do the tests after rectifying the mistakes.
When scaling your company’s A/B testing program, there are different aspects that need to kept in mind.
Once your team has tested each element or most elements in the backlog, they should revisit each successful as well as failed campaign to determine whether there is enough data to justify running another version of the test. Also, the team will have to plan in such a way that none of the tests being performed end up affecting other tests or the website’s performance. One way to do this is by running tests simultaneously on different pages of the website or testing elements of the same web page at different time periods.
Testing too many elements of a web page together makes it difficult to pinpoint which element had the most influence on the success or failure of the test. Most companies measure an A/B test’s performance based on a single conversion goal. Sometimes, the winning variation affects other website goals as well. To scale an A/B testing program, tracking multiple metrics will help you draw more benefits with less effort.
- Anyone interested in gaining a deep understanding of customers, including CRM managers, marketing managers and interface designers should be informed of the results.
- This leads to wide-ranging discussions based on actual experiences, thereby optimizing internal processes as part of a culture of continuous improvement.
- It validates the significance and value of A/B testing within the organization.
- This fosters the development of a data-driven culture within the organization.
5. CONCLUSION
A/B testing is a well-established and proven methodology which has resulted in significant benefits for companies that have correctly implemented it for their businesses.
Prescience Decision Solutions has successfully conducted A/B testing for a large Indian retail company in the eCommerce space. The marketplace platform had algorithms to ensure that products were price matched against other prominent eCommerce websites. In order to understand whether sales of certain categories would be negatively impacted by removing the built-in price matching functionality, Prescience Decision Solutions conducted A/B testing for over 2000 products. The hypothesis was – removing the price match would not negatively affect the sales of their selected products (in this case, books). The study found that any decline in the product sales could not be attributed to the lack of price matching with competitor eCommerce websites. This led to the company removing the price matching functionality on this product range and improving its revenues and profits for the same.
