The company runs a large online auction platform and shopping website that connects buyers and sellers in over 190 countries. Their website has millions of sellers with 1.9 billion global listings and over 132 million active buyers.
THE CHALLENGE
Every month, the company sends out a satisfaction survey to different sets of buyers and sellers on their marketplace, to better engage with their key stakeholders, understand what issues they are facing, identify areas of improvement and correlate their own findings with the feedback that they receive. This survey consists of multiple-choice questions, objective questions and subjective questions. The primary customer satisfaction metric is measured by calculating the total number of respondents that had rated them the highest (on a scale of 1 – 5), divided by the total number of respondents. There was no process to analyze the subjective feedback that was collected from the monthly satisfaction surveys.
If the results of a monthly survey were unexpected or if key business stakeholders wanted details on why certain sections of buyers or sellers had provided low scores, they reached out to their analysts for further details. The analysts had to go through hundreds of survey responses from that month, as well as thousands of past responses to provide the requested information. These ad-hoc requests typically took 2 to 3 days to complete and the pattern of requests was not consistent.
The company needed an analytics partner to build a machine learning (ML) model for sentiment analysis of the monthly satisfaction surveys and provide summarized reports for the top management.
THE SOLUTION
The Prescience team studied the historical results of the satisfaction survey process and recommended the Zero Shot Classification model for Natural Language Processing (NLP). This model does not require the creation of a training dataset, which can be highly effort intensive, time consuming and costly. Our analysts identified all the required labels that the model would use to classify the subjective responses from each survey.
There were over 5,000 responses from satisfaction surveys that were conducted in the previous years. These were fed into the Zero Shot Classification model to help the company’s management team understand historical trends of sentiment analysis for both buyers and sellers, while also creating a long-term benchmark for future surveys. The collated entries of each month’s satisfaction survey were entered into the ML model.
These details were also entered into the company’s proprietary Artificial Intelligence (AI) tool, which then summarized the overall sentiments and findings for that data set, into the top 5 themes. Our team created a standardized template in Microsoft Excel for reporting the findings of the Zero Shot Classification model. This is submitted each month to the top management, along with the key themes that were identified by the company’s AI tool.
By automating the process using the ML model and the proprietary AI tool, the monthly sentiment analysis report can now be generated in less than an hour.
The different technologies used for this engagement included,
- SQL
- Python
- Microsoft Excel
- Proprietary AI tool
THE IMPACT
With this new automated ML model for sentiment analysis, the company has immediate access to key insights into customer satisfaction and dissatisfaction levels and can take remedial action for issues that get highlighted through the surveys. They also have visibility into past customer sentiment trends which were previously unavailable to them.
The overall process is being replicated for understanding the seller sentiment from their communications to each other through the marketplace.
Prescience Team
