Why You Can’t Just Cluster Your Raw Data

The problem

Say you have a dataset of patients, each described by a set of symptom scores, and you want to group them into “types”- clusters of people who look similar. The obvious move is to throw the scores into a clustering algorithm (k-means, hierarchical, whatever) and read off the groups.

I tried exactly this early in a research project. The clusters came out clean and confident. There was just one problem: they were clustering the wrong thing.

When I looked closer, the groups weren’t separating by symptom profile at all. They were separating by how long ago each person had been assessed. Patients measured recently landed in one cluster; patients measured years earlier landed in another. The algorithm had found structure - it just wasn’t the structure I cared about.

This is the trap: a clustering algorithm clusters whatever variation is loudest in your data. And time was loud, because the groups I cared about were measured at different times. So the algorithm latched onto when people were assessed instead of how they presented.

An example

We want to cluster patients into certain Phenotypes. Here, the severity groups were measured at different times, each severity group having certain healing trend.

The milder patients happened to be assessed early (about a year after injury), while the more severe patients were assessed late (years later). And symptoms tend to ease over time, so early scores run higher and late scores run lower.

So every score is a mix of:

real symptoms (what you want) and a time effect (when they happened to be measured)

A difference you see between mild and severe might not be about injury severity at all. It might just be that one group was measured earlier than the other.

Clustering groups people who look similar. If time pushes everyone’s scores up/down and that shift is bigger than the real symptom differences, the algorithm groups by WHEN people were measured instead of by their actual symptom pattern.

The fix : Remove confounding

Remove the time effect from each score first. Once you do, the early/late gap disappears and the patients look like this:

The culprit here is a confound — a variable that’s mixed into your measurements and isn’t the thing you’re studying.

In my case, symptom scores naturally changed with time since assessment. So “time” was baked into every score. Two people with identical underlying symptom profiles could have different raw scores purely because one was measured earlier than the other. Cluster on the raw scores, and time leaks in.

The same thing happens with age, sex, or any background variable that influences your measurements. A useful way to think about it: confounding is defined relative to your question. Time-since-assessment is a confound for the symptom-profile comparison, even though it’s a perfectly real and interesting effect on its own.

Removing the confound

The standard tool is regression. You model each score as a function of the confound, then subtract off the confound’s estimated contribution — leaving an “adjusted” score that no longer carries that variation.

Here’s the idea:

import numpy as np
import pandas as pd
import statsmodels.api as sm

np.random.seed(0)
n = 300

# a confound: time since assessment, unrelated to type
time = np.random.exponential(3, n)

# observed score = true-type effect + a TIME effect (the confound) + noise
score = (
    np.array([50, 60, 55])[true_type]   # real signal
    + 4 * np.log1p(time)                 # confound leaking in
    + np.random.normal(0, 3, n)
)

df = pd.DataFrame
({
"score": score, 
"time": time, 
})

If you cluster on score directly, the time term contaminates everything. So first, regress it out:

# fit one model: score ~ log(time), across EVERYONE
X = sm.add_constant(np.log1p(df["time"]))
model = sm.OLS(df["score"], X).fit()

# subtract the time effect → adjusted score
df["score_adj"] = df["score"] - model.params.iloc[1] * np.log1p(df["time"])

Now score_adj reflects the true-type signal with the time contamination removed — and clustering on it recovers the real groups instead of the confound.

The takeaway

An important part of clustering is — making sure your data is clusterable — that the variation driving your groups is signal, not a confound.