Proto Personas
Reviews Analysis
TL;DR
After the proto-personas were created, we still needed stronger evidence to validate some of the assumptions behind them. A survey produced almost no useful responses, so I proposed using publicly available product reviews instead.
I gathered and cleaned several thousand reviews of Elastic Email and three close competitors from G2, Capterra and Trustpilot. I then used three different LLMs to analyse recurring areas of customer importance and compared the results. The analysis supported some of our key persona assumptions and challenged one of the developer profiles.
My role: full ownership
The problem
We had a set of useful proto-personas, but much of their content was still based on assumptions.
Direct research was not working
My first attempt at validation was a customer survey, but engagement was extremely low. We needed another source of customer evidence.
Time and budget were limited
Primary research would have taken more time and money than we had available. I wanted to find a way to learn from data that already existed.
We needed evidence beyond our own customers
Looking at competitors offered another advantage. If similar themes appeared across several products serving the same market, they were more likely to reflect broader customer priorities rather than quirks of Elastic Email alone.
What I did
Building a review dataset
I selected three competitors that targeted customer groups similar to ours, giving me four companies to analyse in total.
I gathered reviews from G2, Capterra and Trustpilot, with the goal of collecting as much material as possible.
Extracting the data was fairly laborious. I saved review pages as HTML and used LLMs to extract the individual reviews. The dataset was large enough to push against the limits of our AI plans, but I eventually managed to clean and prepare it for analysis.
Looking for importance, not sentiment
I created a detailed prompt that instructed the model to identify the areas customers considered most important.
The key idea was to ignore whether a topic was mentioned positively or negatively.
Someone praising customer support and someone complaining about poor support were expressing opposite opinions about quality, but both were signalling that support mattered.
This allowed me to look past simple sentiment and focus on the themes customers cared enough about to mention.
Cross-checking the AI analysis
Because the dataset was too large to manually verify in full, I did not want to rely on the output of a single model.
I ran the analysis through three different LLMs and compared their conclusions.
All three produced broadly consistent results, which gave me more confidence that the strongest patterns were not simply artefacts of one model.
Outcome
The analysis gave us stronger evidence for several assumptions used in the personas.
The clearest confirmation was that ease of use and time saving were important across a large part of the market. This supported the idea that many of our customers did not want email to become a complex or time-consuming part of their work.
At the same time, the evidence challenged one of our original persona profiles.
We found little support for the SaaS microfounder as a representative developer type. A web developer, often working in an agency context, appeared to fit the available evidence much better.
This finding was later reinforced by the customer case study series and contributed to a change in the developer persona.
The analysis also produced some useful side findings, including differences in how competitors approached spam policies.
The main value of the project was that it gave us a relatively fast and inexpensive way to test assumptions against a much larger body of customer language than we could have gathered through direct research alone.