Murphy Knows Your Users Better Than You Do

Github: https://github.com/ProsusAI/Murphy Contributors: Kevin Ngo, Isha Agrawal, Ambra Mihu

Murphy is an open-source evaluation agent that tests your product the way real users do, clicking through flows, filling forms, and failing in ways your test suite never anticipates. But the quality of that evaluation depends entirely on one thing: how realistic the personas are.

Murphy shipped with 10 hand-authored personas. Each defined upfront as a fixed trait vector. Useful, but hand-authored personas will always miss behaviors nobody thought to imagine.

So we built something different: a pipeline that reads your product analytics and generates personas from scratch. Session data in, named and operationally ready personas out. We also built an evaluation layer that measures how closely Murphy’s actual test behavior matches the real users each persona was derived from

The problem with static personas

When you define personas by hand, you’re encoding your own assumptions about how users differ. You think the interesting axes are technical sophistication, patience, and intent. But what if your product’s actual failure mode is that a significant group of users never backtracks, they proceed linearly, skip verification steps, and submit whatever’s in front of them? Or that another group reads everything carefully, then hesitates for two minutes before taking any action?

These aren’t generic UX patterns. They’re specific to your product’s design, your onboarding, your user base. Static personas can approximate them. They can’t discover them.

The idea: let the data define the axes

The core insight behind dynamic personas is that behavioral trait dimensions should emerge from session data, not be imposed on it.

If you give an LLM a compressed timeline of a real user session and ask it to describe what it observes, in its own words, no schema, it notices things. “This user backtracks constantly before committing.” “Rage-clicked three times during the loading screen, then left.” “Opened five features in sequence without engaging deeply with any of them.”

Do that across many sessions, cluster the observations semantically, and patterns emerge. Not the patterns you expected, the ones actually in your data. Those become the trait dimensions. Score each session against those dimensions, cluster the score vectors, and ask the LLM to name each cluster.

The result is a persona set grounded in your actual users. Run the same pipeline on a different product and you get different axes, different names, different operational guidance, because different users behave differently.

How it works

The pipeline runs in three phases.

Phase 1 — Discovery: finding the axes

Sessions are pulled from PostHog and compressed into structured plain-text timelines. Each timeline covers what the user did (navigation, interactions, uploads, feedback), how they did it (dwell times, backtrack rate, time-to-first-action), and contextual signals like rage clicks and device info. PII is stripped at this stage, message content, emails, and sensitive properties never reach the LLM.

Each event timeline is sent to an LLM with a deliberately open-ended prompt: describe the behavioral characteristics of this user. No fixed schema. No suggested axes. The model describes whatever it sees.

This produces a large set of free-form trait observations. They’re embedded and clustered to find which behaviors co-occur and which are semantically related. The clusters are handed back to an LLM to synthesize into named trait dimensions, each with a name, a description, and anchors for what low and high scores look like.

Here’s what that produced on a real run across sessions from Toqan, a Prosus-built enterprise agentic platform whose users range from technical builders to non-technical business partners:

Discovered behavior trait schema

Phase 2 — Scoring: placing each user on the axes

With the trait schema in hand, a fresh batch of sessions is scored against it. Each session gets a single LLM call: here’s the timeline, here are the seven dimensions, score each from 1 to 5 with brief reasoning. The output is a score vector per session, a point in seven-dimensional trait space.

This phase is fast and cheap. The expensive open-ended discovery only runs once. Scoring is structured and bounded.

System and user prompt for scoring user sessions on the discovered behavior trait dimensions

Phase 3 — Clustering: the personas

K-Means on the score matrix groups sessions into clusters. Each cluster centroid is a characteristic score profile representing a type of user. An LLM is given all the centroids and asked to name each one: an archetype name, a description, and the dimensions where this cluster diverges most from the others.

But here’s what makes the output operationally useful rather than just descriptive. Each persona also comes with:

• Execution hints — concrete behavioral instructions for the browser agent running the test
• Judge questions — what a separate AI judge should ask when evaluating a test run under this persona
• Success criteria guidance — what counts as pass vs. fail, calibrated to this persona’s traits

These aren’t marketing personas. They’re agent role specifications.

High-level overview of the dynamic personas generator pipeline

What came out of a real run

Here are two of the eight personas that emerged from a run on the Toqan platform. They weren’t designed to contrast, they’re just what the data produced.

That’s the value. The axes that matter aren’t always the ones you’d reach for first.

What this enables

Tests that ask the right questions

When a Murphy test runs under the Impatient Minimalist persona, the judge isn’t asking whether the operation succeeded. It’s asking whether the site handled early abandonment gracefully — no orphaned state, no silent failures, no flows that break when a user disappears halfway through. That’s the question that matters for that cluster of users.

Different persona, different verdict criteria. Same site, same interaction, but evaluated differently because different users experience it differently.

No configuration for new products

A product integrating Murphy for the first time doesn’t need to decide which personas are relevant. Run the pipeline on a few months of PostHog data and they emerge. This is especially useful across the Prosus portfolio, where products span very different user bases and domains. Toqan, built by Prosus, is one of them: an enterprise agentic platform whose users range from technical builders to non-technical business partners, exactly the kind of behavioral diversity dynamic personas are built to capture.

Personas that stay current

User behavior shifts as products evolve. New features attract new user types. Onboarding changes reshape how first-time users engage. Dynamic personas can be regenerated on a schedule — a weekly run on fresh session data keeps your test suite calibrated to who your users actually are right now, not who they were at launch.

Measuring how well Murphy mimics real users

Once we have persona archetypes derived from real user sessions, the natural question becomes: when Murphy runs tests as one of those personas, does it actually behave like one? This is where the evaluation layer comes in.

Setting up the comparison

Every time Murphy completes a test run, it produces a detailed action history: every click, navigation, form fill, error, and moment of deliberation, annotated with timing. We convert this into a behavioral timeline that deliberately mirrors the format we use to represent real user sessions .

Scoring Murphy like a real user

The same LLM scorer used to extract behavioral trait scores from real sessions is then applied to Murphy’s session timeline. It scores each dimension against the trait schema we discovered from the data.

At this point, we have two sets of scores in the same space: Murphy’s scores for this test run, and the persona’s centroid scores, which represent the average across all real user sessions that were grouped into that cluster. The difference between them, per dimension, tells us where Murphy drifted in representing that trait.

Two similarity signals

The evaluation report captures this as two complementary metrics.

• Trait similarity — a 0–1 score computed from the per-dimension deltas between Murphy’s trait scores and the persona centroid.
• Embedding similarity — cosine similarity between the vector embedding of Murphy’s session timeline and the centroid embedding of the persona, which is the mean of the embeddings of all real user sessions in that cluster.

What the numbers show

To test how persona fidelity holds up under pressure, we designed three goals at increasing levels of complexity. A simple goal — “test the agent creation functionality”, a medium goal — “test the creation of a RAG agent using startup_handbook.pdf” and a difficult goal — “test the creation of a RAG agent that runs on a schedule, using startup_handbook.pdf and the scheduling functionality”.

Each level adds complexity, as Murphy needs to go through more pages, understand more features and click more.

Across all three, Murphy achieves strong persona fidelity for task-oriented, methodical archetypes. Focused Implementer, Workflow Perfectionist, and Precise Tester consistently score 85–92% LLM trait match across all conditions, with the Workflow Perfectionist hitting a near-perfect match on Methodical Thoroughness in every single run.

However, as goal difficulty increases, Murphy’s LLM match degrades — from a largely HIGH-tier distribution on simple goals down to a contracted, predominantly MEDIUM profile on difficult ones.

Murphy is open source. If you’re building something with a UI and want honest, user-grounded evaluation, take a look: https://github.com/ProsusAI/Murphy