The Number That Lies to Your Entire Leadership Team

Simpson’s Paradox — the statistical trap hiding inside your business reviews

Simpson’s Paradox

Picture this.

Your Monday business review. Slides are up. Overall conversion rate is trending positive — up 3% this quarter. The room is pleased. Someone says “great work team.” The meeting moves on.

What nobody noticed: conversion dropped in every single customer segment.

Not flat. Not mixed. Down. Across the board. Every segment, every cohort, every slice of the business — going in the wrong direction. And yet the aggregate number — the one on the slide everyone looked at — showed growth.

This is Simpson’s Paradox. And it is not a rare statistical curiosity. It is hiding inside business reviews at companies right now, including probably yours.

What Is Simpson’s Paradox?

Simpson’s Paradox occurs when a trend that appears in aggregated data completely reverses — or disappears — when you break the data into subgroups.

The aggregate tells one story. Every segment tells the opposite story. Both are mathematically correct. Only one is real.

It sounds impossible. It isn’t. It happens because of a shift in the composition of your data over time — a change in the mix of who or what you’re measuring — that distorts what the overall number actually means.

A Product Analytics Example

Let’s make it concrete.

You run a two-sided marketplace. You have two customer segments: casual users and power users. Last quarter vs this quarter, here’s what happened:

Casual users: Conversion dropped from 4% to 3% Power users: Conversion dropped from 25% to 22%

Every segment is down. Meaningfully.

But here’s the catch. This quarter, your marketing team ran a big acquisition push that brought in a flood of new power users — a segment that converts at a much higher rate than casual users. So even though conversion fell within each group, the overall mix shifted toward the higher-converting group.

Result: your blended conversion rate went up.

The slide says the business is improving. The data says the opposite. Leadership makes decisions based on the slide.

This is the paradox. And the damage it does isn’t just analytical — it’s strategic. You might double down on a product change that’s actually hurting conversion. You might reward a team for a metric that’s being flattered by a mix shift they had nothing to do with.

Why It Happens So Often in Business Reviews

The answer is mundane: most business reporting aggregates by default.

Your dashboard shows total conversion, total retention, total revenue per user. It’s clean. It fits on a slide. Executives can absorb it in thirty seconds.

What it hides is composition. And composition changes constantly in a live business — new markets open up, acquisition channels shift, product changes attract different user types. Every time the mix changes, your aggregate metric becomes a different animal without anyone noticing.

The paradox is particularly common in:

• Retention analysis when a new cohort of less-engaged users joins alongside your core base
• Revenue per user when you expand into a lower-ARPU market segment
• Conversion rates when channel mix shifts between high and low intent traffic
• Clinical or health outcomes when treatment group demographics change over time

The underlying performance in each group can be deteriorating while the top-line number quietly flatters everyone in the room.

How to Catch It

The fix is not complicated. It’s just disciplined.

Always segment before you aggregate. Before you report any top-line metric, cut it by your most meaningful dimensions — user type, acquisition channel, geography, cohort, product tier. If the story changes when you segment, the aggregate is lying to you.

Watch for mix shifts. Whenever your aggregate metric moves, ask: did the composition of what I’m measuring also change? A rising overall conversion rate means nothing if you added a million high-intent users this quarter. Strip out the mix shift and look at like-for-like performance.

Use standardised or weighted metrics. If you want to track performance over time without composition distorting the picture, hold the mix constant. Measure each segment against itself, then report segment-level trends alongside — not instead of — the aggregate.

Make segmentation a default, not an afterthought. The best analytics teams don’t dig into segments when something looks wrong. They segment first, every time, and treat the aggregate as a summary to be explained — not a fact to be reported.

The Broader Point

Simpson’s Paradox is a symptom of something deeper: the gap between what a number says and what it means.

Aggregate metrics are convenient. They compress complexity into something everyone can look at in thirty seconds. But that convenience comes at a cost — the cost of hiding the mechanics underneath. A business that only looks at top-line numbers is a business that can be fooled by its own data, consistently, without ever knowing it.

The job of a good analyst isn’t to make numbers look clean. It’s to make sure the story the numbers are telling is the right one.

And sometimes that means walking into a business review, pointing at the green number on the slide, and saying — “actually, let me show you what’s happening underneath.”

That’s uncomfortable. It’s also exactly what the room needs.

This is part of a series on rigorous analytics thinking. Previous pieces cover causal inference and the most common A/B testing mistakes.