Understanding Consumer Behaviour Through A/B Testing Prescience Decision Solutions November 20, 2023

Understanding Consumer Behaviour Through A/B Testing


In today’s highly competitive world, business leaders must rely on empirical evidence and an array of data points to support the short-, medium- and long-term decision making for their companies. This is evidenced in the ever-growing reliance that these business leaders have on Business Intelligence (BI) reports, market study reports, A/B testing etc.

A/B testing refers to a randomized experimentation process wherein two or more versions of a variable are tested, to understand which version results in the maximum desired impact. This testing methodology traces its roots back to the 1920s, when a biologist and statistician Ronald Fisher, performed various agricultural experiments which laid the foundations for what is now known as A/B testing. Some of these experiments analyzed what would happen if different fertilizers or different quantities of fertilizers were used in agriculture.

In later decades, this testing approach was adopted for various sectors including marketing and advertising. While sending out mail-order brochures and postcards, marketing companies used different variations to see which of them resulted in more calls and sales. A/B testing, which is also known as split testing, if often used in political campaigns, as well. For Barack Obama’s 2012 re-election campaign, his team used various A/B tests to eventually increase the online donation conversions by 49%, but also boost online sign-up conversions by 161%.

Today, A/B testing is widely used for online businesses. It is a part of the standard customer conversion (when a visitor completes an objective defined by the merchant during the course of his / her visit) toolkit of all online businesses, working on the principle that, if you don’t test, you are wasting your website traffic. A/B testing is within the reach of all businesses, regardless of size, provided that the solution used and the processes that are deployed are sufficiently agile to ensure that the benefits are realized.

The common misconception is that for online businesses A/B testing revolves around optimizing the layouts, size, position and colors of webpages. However, the applications for online businesses are far more-wide ranging and impactful than what most people expect it to be.

For example, game development companies use A/B testing to validate different hypotheses like “How frequently should in-game advertisements be run?” or “What should be the price of in-game bundle purchases?” or “What sort of in-game rewards lead to higher customer retention?”.

The online streaming service Netflix has long used A/B testing to drive the engagement levels with their customers. Back in 2017, when the company moved from the user star rating system to the thumbs up and thumbs down system, the company found that the ratings activity went up by a massive 200%. More recently, Netflix found out through surveys that this approach of asking users to rate shows on a thumbs up or a thumbs down basis, wasn’t proving to be as effective as they anticipated it to be. In order to understand a better way of engaging with the customer, the company used extensive A/B testing to find that a two-thumbs up rating system led to quantifiably higher engagement levels, as compared to a hearts-based variant of rating shows.


A/B testing’s rise to prominence has stemmed from the realization that as competition increases, so does the cost of customer acquisition. Consequently, it has become imperative for all online businesses such as eCommerce, travel, SaaS, education, media, and publishing businesses to optimize their customer conversion rates (the number of conversions divided by the total number of visitors). The primary objective of all these businesses is to attract visitors on the website who would ultimately transform into actual customers.

The key challenge for most of these businesses lies in accomplishing more with the limited resources at their disposal. This translates to enhancing website conversion rates while maintaining a steady flow of traffic, rather than perpetually increasing traffic without addressing the conversion rates. However, A/B testing goes beyond merely mitigating the repercussions of continually rising customer acquisition expenses.

The major organizational advantages of using A/B testing are:

2.1 Shortening your decision-making process
In the e-commerce industry, customer buying habits change rapidly and competition is fierce. Organizations need to make decisions quickly in order to stay competitive and have to adapt to frequent changes in their markets. Since A/B testing is entirely data-driven with no room for guesswork, gut feelings, or instinct based decision making, companies can quickly determine a “winner” and a “loser” based on statistically significant improvements. Measuring metrics like time spent on the page, number of demo requests, cart abandonment rate, click-through rate, and so on prove the success or failure of a planned change.

2.2. Get better RoI from existing traffic
As most experienced optimizers have come to realize, the cost of acquiring quality traffic on a company website is huge. A/B testing enables companies to make the most out of their existing traffic and helps increase conversions without having to spend additional dollars on acquiring new traffic. A/B testing can result in high Return on Investment (RoI) as sometimes, even the minutest of changes on a company website can result in a significant increase in overall business conversions.

2.3. Learning more about your visitors
While A/B testing is a valuable method for understanding the preferences of visitors on company websites, it’s not the only available option. Techniques like user panels, focus groups, and eye tracking can also offer crucial insights. However, A/B testing uniquely enables companies to interact with their visitors in authentic settings. It enables the gathering of unbiased data about their preferences, both conscious and subconscious, without the customers being aware that the company is collecting essential behavioral insights.

2.4. Adopts a RoI focused approach
While A/B testing is a valuable method for understanding the preferences of visitors on company websites, it’s not the only available option. Techniques like user panels, focus groups, and eye tracking can also offer crucial insights. However, A/B testing uniquely enables companies to interact with their visitors in authentic settings. It enables the gathering of unbiased data about their preferences, both conscious and subconscious, without the customers being aware that the company is collecting essential behavioral insights.

2.5. Promoting a data-oriented culture

When choosing between different versions of a website, there’s a long history of examples where the choice that seems obvious doesn’t always work out. In the Netflix two thumbs A/B testing, many within the company expected that the Hearts option would prove to be a lot more popular among users than the two thumbs up variant. However, the extensive tests led to an unexpected data driven outcome.

In addition to these benefits, companies can also gain major website level benefits by using A/B testing.
A website visitor can typically face common pain points while navigating including confusing marketing material and hard to find the CTA Button (Call To Action which refers to a button or link that suggests a particular action to the visitor)  like Watch Now, Buy Now, Request a Demo etc. Not being able to achieve what they want to directly leads to a bad user experience. This, in turn, increases customer friction and eventually impacts a company’s conversion rates. To solve common visitor pain points, companies use A/B testing and data gathered through visitor behavior analysis tools such as heatmaps, Google Analytics and website surveys.

One of the most important metrics to track a company’s website performance is its Bounce rate (the percentage of visitors that leave a webpage without taking an action, such as clicking on a link, filling out a form or making a purchase)  There can be many reasons behind a website’s high bounce rate, such as too many options to choose from, a mismatch of expectations, confusing navigation, use of excessive technical jargon and so on.

Since each website serves different goals and caters to various customer segments, there is no one-size-fits-all solution to reducing bounce rate. However, running an A/B test can assess multiple variations of an element of the website until the best possible version is identified. This not only helps companies identify friction and visitor pain points but also helps improve the overall experience of website visitors, making them spend more time on the site and possibly even converting them into paying customers.

Making minor, incremental changes to a web page with A/B testing, instead of getting the entire page redesigned can reduce the risk of jeopardizing the current conversion rates. An example of this is product description changes on a website. Companies can perform an A/B test when planning to remove or update the existing product descriptions. Another example of a low-risk modification is the introduction of a new feature change. Before adding the new feature, performing an A/B test can help companies understand whether the new changes will help the website users.


There are three dimensions that companies use to create and test new variations. Some companies tend to focus on just one dimension, but market-leading conversion rates can be achieved by mastering all the following dimensions. They are,

3.1. The Product
A/B testing provides a cost-effective means to assess different hypotheses regarding how companies present their offerings:
  • Value Proposition: Is it easily understood, logically structured, and consistently presented across pages? Should the emphasis be on product quality, benefits, or price?
  • Information Clarity: Does the website provide the right level of detail? Should there be more information, or should it be further streamlined to what is essential?
  • Presentation Format: Would a concise video be more effective than a lengthy one?
  • Product or Service Impact: Is one product or service overshadowing the sales of another? Is the pricing aligned with the market?
  • Guiding Customers: Are visitors directed and reassured by appropriate messages? Are promotional offers prominently featured?
3.2. The Data
Companies often have a substantial amount of information about their visitors’ behaviour, aggregated in a web analytics tool, perhaps without even utilizing it. If there is an unusually high bounce rate on a landing page, it’s a clear candidate for a series of A/B tests.
3.3. The User Interface
Visitor behaviour and UX (User eXperience, which is the set of responses and perceptions that result from the use of a service or a website) trends are rapidly evolving. Conducting a user experience audit is often the right approach to pinpoint areas of the website which may be underperforming. What was a good decision in one year may not hold true in the next. Analysing the path which customers take (behaviour flow, browsing, etc.) also demands thorough attention.
Companies that do not have UX experts carry out a wide range of tests on various elements of their website including,
  • Design, placement, and overall appearance of text, text blocks, and CTAs
  • Wording, especially in CTAs
  • Forms

Since companies need to have a flexible approach for A/B testing, it is vital that you start with the right toolkit. Interestingly, not all A/B testing solutions are the same. Some are complex and only used by experts. Your company needs to carefully choose a testing platform, especially if you are starting out with A/B testing.

4.1. Prioritize the right tests
A common challenge with A/B testing is the inclination to begin with complex scenarios like multivariate tests (Multiple sections are tested simultaneously, and one or more variations are created for each section) for page redesigns. It is advisable to start with simpler tasks before tackling more intricate ones. Specifically, your company should focus on tests that offer realistic improvements in conversion rates and are easy to implement.

A simple test typically entails minimal alterations to the design of one or more pages. With an efficient A/B testing tool, changes like adjusting text, colours, product images, button sizes, rearranging elements, linking to a new page, or creating banners can be executed by anyone without specific technical expertise.

An advanced test doesn’t involve major functional changes but encompasses more intricate design adjustments. This can involve tasks like shifting from a three-column layout to a two-column layout for product listings, reorganizing a navigation menu, removing fields from a form, URL adjustments etc.

Complex tests often necessitate significant functional changes and demand proficiency in web technologies, particularly JavaScript. Examples include transitioning from page redirection to an overlay window with a pop-up, redesigning product data sheets, breaking a form into multiple steps without a URL change etc. 

4.2. Have an A/B testing calendar

A/B testing is a part of a wider holistic Conversion Rate Optimization (CRO) (the practice of increasing the percentage of users who perform a desired action on a website) program. An effective optimization program involves analysing your existing website data and gathering visitor behaviour data, then moving on to preparing a test backlog (a list of tests that are planned to be carried out in the future) of action items, further prioritizing each of these items, running tests, and then drawing insights. To achieve this, you need to make an A/B testing calendar which consists of the following:

Step 1: Measure

This includes measuring your website’s performance in terms of how visitors are reacting to it. Your company needs to figure out what is happening on the website, why it is happening, and how visitors are reacting to the webpages. Everything that goes up on the website should correspond to your company’s business goals and Key Performance Indicators (KPIs) of the website. Additionally, you need to collect and understand you visitor behaviour data. Based on this, your company can prepare the test backlog. With a data-backed backlog ready, the next step is formulating a hypothesis for each backlog item. For example, after analysing the data gathered using quantitative and qualitative research tools, you can conclude that not having multiple payment options leads to maximum prospect customers dropping off on the checkout page. So, the hypothesis is “Adding multiple payment options will help reduce drop off on the checkout page.”

Step 2: Prioritize
Not every action area has equal business potential. So, it becomes imperative to weigh out the different backlog options before picking the right ones to test. There are a few dimensions to keep in mind while prioritizing items for the test campaign like the potential for improvement, page value and cost, the importance of the page from a business perspective, traffic on the page, and so on.

Step 3: A/B test

While the test is running, make sure it meets every requirement to produce statistically significant results before closure, like testing on accurate traffic, not testing too many elements together, testing for the correct duration, and so on.

Step 4: Repeat
While analysing the test results, one common finding is that some versions had performed better than the others and hence, “won”. There can be 3 outcomes of the tests:
  • A particular variation won with statistical significance.
  • The existing option was the better version and hence, won over the variation/s.
  • The test failed and produced insignificant results.

In the first 2 scenarios, you should not stop testing just because there is a clearcut winner. Your company has to make further improvements to the versions and continue the testing. In the 3rd scenario, you need to identify where the process went wrong and re-do the tests after rectifying the mistakes.

4.3. Scaling the tests

When scaling your company’s A/B testing program, there are different aspects that need to kept in mind.

Once your team has tested each element or most elements in the backlog, they should revisit each successful as well as failed campaign to determine whether there is enough data to justify running another version of the test. Also, the team will have to plan in such a way that none of the tests being performed end up affecting other tests or the website’s performance. One way to do this is by running tests simultaneously on different pages of the website or testing elements of the same web page at different time periods.

Testing too many elements of a web page together makes it difficult to pinpoint which element had the most influence on the success or failure of the test. Most companies measure an A/B test’s performance based on a single conversion goal. Sometimes, the winning variation affects other website goals as well. To scale an A/B testing program, tracking multiple metrics will help you draw more benefits with less effort.

4.4. Implement relevant organizational processes
In our experience, it is vital to designate a Project Manager (PM) to oversee and coordinate the testing efforts. This PM will define tests, ensure their proper integration, monitor outcomes, gather data from the production environment, adhere to schedules, and accurately analyse results. The project manager should also advocate for A/B testing internally. An ideal choice for the PM role is a technically adept collaborator, with experience as a front-end / back-end developer or a web marketer, with technical knowledge of CSS and HTML as prerequisites.
4.5. Share the test results
Communicating the results of A/B testing offers several benefits for your organization:
  1. Anyone interested in gaining a deep understanding of customers, including CRM managers, marketing managers and interface designers should be informed of the results.
  2. This leads to wide-ranging discussions based on actual experiences, thereby optimizing internal processes as part of a culture of continuous improvement.
  3. It validates the significance and value of A/B testing within the organization.
  4. This fosters the development of a data-driven culture within the organization.

A/B testing is a well-established and proven methodology which has resulted in significant benefits for companies that have correctly implemented it for their businesses.

Prescience Decision Solutions has successfully conducted A/B testing for a large Indian retail company in the eCommerce space. The marketplace platform had algorithms to ensure that products were price matched against other prominent eCommerce websites. In order to understand whether sales of certain categories would be negatively impacted by removing the built-in price matching functionality, Prescience Decision Solutions conducted A/B testing for over 2000 products. The hypothesis was – removing the price match would not negatively affect the sales of their selected products (in this case, books). The study found that any decline in the product sales could not be attributed to the lack of price matching with competitor eCommerce websites. This led to the company removing the price matching functionality on this product range and improving its revenues and profits for the same.

If you would like to know more about A/B testing and how Prescience Decision Solutions can assist you on this A/B testing journey, do reach out to us on through our company website. One of our senior executives will reach out and guide you through the process.