Most Health Scores Are Built Backwards

I got a call on a Friday afternoon with a brief and a Tuesday presentation. I gave myself 48 hours, spent Monday doing QA, and got it to them before COB.

The role didn’t move forward. It happens. But this was a serious piece of work and I think it’s worth sharing, so here it is. The deliverable they received was three pages. The working notes and implementation plan behind it were eleven. What you’re reading now is somewhere in between. Imagine what 90 days looks like.

The Brief

PLG SaaS. Multi-product. Incomplete telemetry. Build a customer health scoring system that can identify churn risk before it happens.

That last constraint is the one worth sitting with. In any PLG model, product usage is the primary signal. It’s the closest thing you have to an objective truth about whether a customer is getting value. And they didn’t fully have it yet. Granular event-level telemetry was still being built.

What they did have: seat data, support history, VoC tooling, NPS responses, billing events, and post-churn feedback from customers who had already left. Good signal environment. Just not the one most people would anchor the whole model around.

Most teams in this situation do one of two things. They wait for the telemetry to be ready, or they build something fragile on top of whatever data is cleanest. Both are wrong. The first costs you accounts you could have saved. The second gives you a score nobody trusts.

Signal Architecture First. Score Second.

Before you touch a weighting model, you have to decide what you’re actually measuring and be honest about what each signal can and can’t tell you.

Seat activation delta became the primary proxy. Seats allocated versus seats activated versus seats still active over a rolling 30-day window. It’s not the same as usage telemetry, but a meaningful drop in active seats is a behavioral signal you can act on even without event-level data.

Imperfect and directional is better than perfect and unavailable.

Secondary signals layered in from there: support volume and sentiment trends, NPS movement over time, billing events like payment failures, auto-renew toggles, and renewal date shifts, plus account-level data from the CRM.

Then there’s CSM input, and this is where most health scoring models either get it right or fall apart completely.

CSM signal belongs in the model. A good CSM carries context that no data source captures. Relationship dynamics, upcoming organizational changes, a champion who just went quiet. That information is real and it matters. But it has to be explicit and measurable. Not a vibe, not a gut feeling, not “this account feels fine.” If a CSM wants to influence a health score, they submit a specific input against a defined rubric. Renewal conversation went well, log it. Champion confirmed budget is approved, log it. Anything that can’t be articulated concretely doesn’t touch the score.

Here’s why that matters in practice. The moment you ship a health score, a CSM is going to walk in and ask why their favorite account is trending toward churn. That conversation is exactly what you want. The score forced a discussion that might not have happened until it was too late. But if you let CSMs override scores based on relationship comfort, you’ve built an expensive system that confirms whatever people already believed. The score has to be trustworthy enough that arguing against it requires data, not feelings.

Starting With What You Know

Before the model runs a single live account, there’s a question worth asking: how much do we already know?

Most companies have data they’re not using. Churned accounts from the past 12 months. Support histories. Billing event timelines. NPS trends leading up to cancellation. That’s a retroactive picture of what churn actually looked like, and it’s the most honest starting point available.

Accounts have churned before — otherwise nobody would be asking for this system. The question is why. You go back through the history and look for the signal. Turns out support volume spiked six weeks before cancellation on a consistent pattern. NPS dropped and nobody followed up. Seats went quiet and the renewal still got booked as a surprise loss. That’s not a guess, that’s a pattern, and that pattern becomes the foundation for the initial weights before the model ever runs live.

If there’s a year of account history, you can review historical patterns to inform initial weights — which signals actually preceded churn, how far in advance, which ones were noise. You enter with a grounded starting point rather than a set of educated guesses.

Post-churn verbatims don’t feed the live score. They’re post-mortem by definition. But churned customers telling you exactly why they left is the clearest retroactive test of whether the model would have caught them. That data refines the weights over time rather than influencing real-time scoring.

Two Parallel Scores as a Built-In Audit

Here’s a step most teams skip entirely.

For the first 30 to 45 days, the proposal calls for two simultaneous health scores fed by slightly different signal combinations. Score A pulls support data directly at the source. Score B trusts the VoC tool to have already digested and classified that same signal.

If both scores track closely across accounts, the VoC tool is doing its job and you consolidate to the simpler architecture. If they diverge, you have something concrete to investigate. Is the tool missing signal? Misclassifying sentiment? Or is it actually catching something the raw volume data doesn’t show?

This turns the calibration period into an accuracy audit of your most expensive tooling. Most teams just pick a methodology and run it. Running two in parallel for 45 days costs almost nothing and tells you a lot.

How the Model Gets Smarter Over Time

A score that doesn’t improve against real outcomes is just a dashboard.

The calibration layer runs separately from the scoring layer. Its only job is checking whether the model is catching churn early enough and surfacing which signals need adjustment.

Scoring runs continuously, triggered by events. Calibration runs monthly. It looks at every account that cancelled, walks back through their score history, and identifies what the model missed. Were they trending red far enough in advance to act? Which signals moved first? Which ones were noise?

Weight adjustments happen one at a time, documented, measured against the following month’s outcomes. This isn’t a controlled experiment — it’s an operational feedback loop. No fishing for improvements, no sweeping overhauls. Small deliberate changes with a clear feedback loop.

At the account volumes typical of a growing SaaS, a human runs this process. Human judgment outperforms automated optimization when you’re working with a small set of cancellation events. The path to automating calibration exists as data matures, but that’s a future problem. Start with someone who understands the accounts, the signals, and the business. The model earns its accuracy over time.

The Architecture: Practical, Owned, and Additive

The expected answer was probably another Zapier workflow. That’s what most CS teams reach for. Fast to stand up, connects tools without writing code, and everyone can at least open it and see what it does.

I went a different direction and the reasoning was practical.

Zapier is a recurring cost that compounds as you add steps. The logic lives in a third-party platform, version control is limited, and if the person who built it leaves, what’s inherited is a series of zap descriptions and a prayer that nothing breaks on a Sunday night.

The scoring engine is a Python script. It runs on whatever infrastructure the company already has. AWS, GCP, Azure, on-prem, or the founder’s favorite Mac Mini. The script is the asset, not the runner. No new platform, no new vendor, no additional licensing conversation at renewal time.

The data layer is Postgres. It’s one of the most widely adopted databases in the SaaS world and any engineer worth their salt has worked with it. If the company is already running MySQL, MongoDB, or anything else at scale, the same logic applies — we use what’s there. Account scores, signal history, weight configurations, and delta state all live in a single database anyone on the team can query directly. No new tooling, no black box, no vendor lock-in.

The pipeline is event-driven rather than a scheduled refresh. Webhooks from billing systems, support platforms, and NPS tooling push signal changes into the pipeline as they happen. A payment fails, a webhook fires. An NPS score drops, a webhook fires. The script processes only the accounts where something actually changed. At 50 accounts this feels like a nice-to-have. At 500 it’s the difference between a system that stays lean and one that becomes a maintenance problem.

This is entirely additive to current operations. CSMs don’t change their workflow. Nothing new is asked of the support team. The score surfaces in the CRM where CS teams already live, populated automatically by events they’re already generating just by doing their jobs. The pipeline runs in the background. The overhead is essentially zero.

Compliance Wasn’t an Afterthought

When I looked at this company’s customer base I could see the logos. Healthcare. Finance. Legal. Education. A platform like this doesn’t have to be in a regulated industry. It just has to sell into one. And at the growth stage they were at, that expansion was inevitable.

I didn’t wait for a compliance team to hand me a requirements doc. I designed to the major standards from the start. HIPAA, GDPR, SOC 2 principles, COPPA where EdTech exposure existed. Built the architecture to satisfy them before anyone paid for a certification to confirm it.

You can’t get SOC 2 overnight. But you can build something a SOC 2 audit wouldn’t reject. Those are very different conversations to have with a prospect’s legal team. And in my experience there is always an overcaffeinated IT admin or CTO sitting in the corner of that room ready to ask. The best part about this architecture is that by the time they ask, there’s nothing left to move.

No PII in the scoring pipeline by design. The system operates on account-level aggregates. Everything lives in infrastructure that’s owned and controlled. Access controls on the database. Every weight adjustment documented. Every change version controlled. A full audit trail from day one, not because a regulator asked for it, but because that’s what a trustworthy system looks like.

Designing to compliance standards before you need to isn’t just risk management. It’s a sales conversation.

What the Naysayers Will Say

“Our CSMs won’t trust a score.” They will if it’s right more often than it’s wrong and if they were part of defining what goes into it. The goal isn’t to replace CSM judgment. It’s to give them a forcing function for conversations that might otherwise happen too late. The CSM who walks in asking why their favorite account is trending red is the system working exactly as intended.

“This is too technical for a CS team to maintain.” The CS team doesn’t maintain it. It runs on existing infrastructure, the logic lives in a repo, and any engineer who touches it can read exactly what it does. The CS team consumes the output in their CRM. That’s it.

“We don’t have enough data to build this yet.” You have more than you think. Start with what exists, instrument the model to learn from outcomes, and improve the weights as the data matures. A directional score that improves over time beats a perfect score that never ships.

“What about compliance?” Already answered above. That was the first question I asked, not the last.

What This Doesn’t Solve

This is a first generation model, not a final one. It will misfire. It will flag accounts that are fine and miss accounts that are quietly eroding. The calibration loop exists because the initial weights, however well-informed by historical data, are still a hypothesis until live outcomes confirm them.

The goal in the first 90 days isn’t a perfect score. It’s a score that’s directional enough to change behavior. Get CSMs having conversations they wouldn’t have had. Surface billing risk before it becomes a churn event. Give leadership something concrete in a revenue review.

As telemetry gets built out, event-level usage data replaces the seat activation proxy. Signal weights get updated based on what actually predicted outcomes. The model earns its accuracy over time.

The churn clock doesn’t pause while engineering finishes the data pipeline. You build with what you have, you instrument the model to learn, and you improve it as the data catches up.

A Note on the Tools

This wouldn’t have come together in 48 hours without AI. I used Claude Code inside my VSCode environment for the heavy lifting, and I’ll be honest — there were a few rogue PowerShell terminals open before I realized I was making a mess of my own environment.

But I want to be clear about what that means and what it doesn’t. I didn’t vibe code this. I have some Python chops, but Codemonkeys was a better TV show than a job title for me. Every signal, every architectural decision, every tradeoff was deliberate. AI was the execution layer. The orchestration was mine from the first line of the brief to the last page of the implementation plan.

It’s the same way I’d work with a strong developer on hand — except the feedback loop is measured in minutes instead of days. The thinking still has to come from somewhere. The AI just removes the ceiling on how fast you can build once you know what you’re building.

Why Give This Away?

Because this framework, as clean as it reads here, means almost nothing without the judgment to deploy it.

Every org has politics. A VoC tool the CS team doesn’t trust. A founder’s friend who built a sentiment platform that’s now somehow in the stack. A CSM who’s been there five years and isn’t interested in being told their favorite account is at risk. An overworked engineer who’s heard “quick integration” before and is already bracing for impact.

The framework doesn’t navigate any of that. Experience does.

A script doesn’t make anyone trust a score. Someone trustable has to build it, run it, and navigate every conversation that comes after it. The framework gets you to the door. Execution is what happens after someone walks through it.