Persona Validation
Persona Validation
TL;DR
Our proto-personas were useful, but many of their goals, challenges and behaviours were still based on assumptions. Before relying on them for targeting and positioning decisions, I wanted to test whether those assumptions held up against real customer evidence.
My first attempt, a paid customer survey, failed almost completely. I then changed course and used two less direct sources of qualitative evidence: a large-scale analysis of customer reviews and a series of customer case studies. Together, these gave us enough evidence to validate much of the original persona framework and make two significant changes.
My role: full ownership
The problem
The first version of our personas gave the Marketing team a much clearer picture of who we thought our customers were. But I was very aware of their limitations.
Without primary research, some of the most important parts of the personas (articularly their goals, challenges and attitudes) were still based partly on assumptions.
Untested assumptions
The proto-personas gave us a useful starting point, but many of their goals, challenges and behaviours were still based partly on assumptions rather than direct customer research.
Risky marketing decisions
We wanted to use the personas to guide targeting and positioning. If our assumptions were wrong, we risked focusing on the wrong audiences and wasting an already limited marketing budget.
“Ease of use” needed evidence
One pattern appeared across several personas: email was usually a supporting task rather than the focus of their work. This suggested that ease of use could become an important part of our positioning, but we needed evidence before acting on it.
I decided to look for stronger customer evidence that could either support these assumptions or show us where the personas needed to change.
Challenges
The process of creating the personas presented several challenges from the very beginning.
Limited access to customers
We had thousands of active customers, but getting them to participate in research turned out to be much harder than identifying them.
No established research panel
There was no existing pool of customers willing to take part in surveys or interviews, so every research attempt required recruitment from scratch.
Testing broad behavioural assumptions
The questions we wanted to answer were not about individual features. We were trying to understand which goals and challenges genuinely mattered to different types of customers.
The process
The process which led me to the final result was long and took some unexpected turns, however it involved three distinctive elements.
Trying direct validation with a survey
My first approach was the most straightforward one: ask customers directly.
I created separate surveys in Tally for each of the four personas. Each survey included screener questions to filter out respondents who did not match the target profile, followed by questions designed to measure the importance of the goals and challenges we had assigned to that persona.
Using data from our onboarding survey, I identified customer accounts that appeared to match each profile. I also secured enough budget to offer a $25 gift card as an incentive.
The campaign failed.
Despite contacting several thousand customers, only a handful completed the surveys. One respondent even appeared to use several accounts in an attempt to claim the reward multiple times.
There was nowhere near enough reliable data to draw conclusions, so I abandoned the survey as a validation method and started looking for evidence we already had access to.
Looking for patterns in customer reviews
My second approach was review analysis.
I collected several thousand reviews of Elastic Email and a group of closely related competitors from G2, Capterra and Trustpilot. AI helped me extract the reviews from website HTML and process the resulting dataset.
I then analyzed the reviews with three different LLMs using prompts designed to identify recurring customer priorities. The important distinction was that I treated both positive and negative mentions as evidence of importance.
Someone praising customer support and someone complaining about poor support were expressing opposite opinions about its quality, but both were signalling that support mattered to them. Rather than asking only what customers liked or disliked, I was trying to identify what they cared enough about to mention at all.
The analysis surfaced several recurring themes and broadly supported many of the goals and challenges already present in the personas.
Using case studies as another source of qualitative evidence
At the same time, the company wanted to build a new series of customer case studies. I volunteered to lead the project because it created another opportunity to learn directly from customers.
We offered participants a $50 incentive and received a much stronger response than we had with the survey. After screening out unsuitable candidates, we completed approximately eight case studies.
Most participants answered a structured set of questions by email, while two took part in live interviews.
The primary goal was to create publishable case studies rather than validate the personas directly. However, the conversations gave us something equally useful: richer descriptions of customers' businesses, workflows, priorities and reasons for using the product.
When I compared these customers against our persona framework, most mapped surprisingly well to the groups we had previously identified.
Outcome
The combined evidence from customer reviews and case studies gave us much more confidence in the overall persona framework, but it also revealed two areas where our original assumptions were too narrow.
Replacing the SaaS microfounder
We found little evidence that the independent SaaS founder we had originally imagined represented a particularly important customer group. A much more common developer profile was someone working in a web or digital agency and managing email infrastructure for multiple clients. We therefore changed the persona to reflect this more prevalent customer type.
Broadening the small-business persona
The original version of Linda represented a lawyer working in a small law firm. The research suggested that this scenario was unnecessarily specific. Similar needs appeared across many small service businesses, including travel, beauty, fitness and consulting.
We changed Linda into a secretary working at a small travel agency: a more neutral example of the broader small-business segment she was intended to represent.
The project did not turn the personas into fully validated research personas. But it moved them further away from assumption and closer to evidence. More importantly, it showed which parts of the original framework were robust enough to keep and which needed to change.