Claude Can’t Run Regression. A 200-Year-Old Theorem Proves It Never Will.

Why the oldest statistical method on Earth is still the one thing LLMs cannot replace

You ask Claude to predict house prices. It writes you a beautiful regression model. Perfect syntax. Clean code. Sensible feature engineering.

You ask Claude why the model predicts what it predicts. It gives you a paragraph about square footage and location.

You ask Claude if adding a swimming pool causes prices to rise. It hedges. It qualifies. It lists assumptions.

Here’s what Claude will never tell you: It has no idea what any of this means.

Not because it’s dumb. Because regression isn’t prediction.

It’s inference. And inference requires something LLMs fundamentally lack.

TL;DR

The theorem: Regression estimates conditional expectations. Not predictions. Not causes. Expectations.

The proof: E[Y|X] is the function that minimizes squared prediction error. That’s it. That’s what you’re computing.

The problem: Every interpretation — causality, significance, generalization — requires assumptions outside the data. Assumptions that require domain knowledge. Human judgment. Things you cannot learn from text.

What LLMs do: Pattern match statistical syntax. Generate plausible-sounding interpretations. Miss the entire point.

What humans still do: Decide what the numbers mean. What they imply. Whether they matter.

The uncomfortable truth: Claude can write a thousand regression models. It cannot tell you if a single one is correct.

Why Regression Is Not Machine Learning

Every data scientist learns regression first. Then they learn neural networks. Then they forget regression.

Big mistake.

Regression is not an algorithm. It’s a framework for scientific reasoning.

When you run lm(y ~ x), you are not training a model. You are estimating a parameter. That parameter has meaning independent of prediction accuracy.

Here’s what I mean.

Linear regression finds the line:

Y = β₀ + β₁X + ε

The coefficient β₁ answers a question: “If X increases by 1, how much does the average Y change?”

Notice I said average. Not “the next observation.” Not “Y in general.” The conditional mean.

This is Gauss-Markov (1821). The least squares estimator gives you the best linear unbiased estimate of E[Y|X].

Not the best predictor. Not the true causal effect. The conditional expectation.

Machine learning models predict. Regression models estimate parameters that have interpretations.

That’s why you can’t replace regression with XGBoost and call it done.

The First Thing Claude Gets Wrong: Prediction vs. Estimation

Ask Claude: “Why is my R² only 0.3? Is this a bad model?”

It will probably say something like: “An R² of 0.3 means your model explains 30% of the variance. You might need more features or a different model.”

This is wrong.

Not factually wrong. Conceptually wrong.

R² measures how much variance you explained. It does not measure whether your model is “good.”

Here’s the thing data scientists forget: You are not trying to predict Y perfectly. You are trying to estimate the relationship between X and Y.

If your R² is 0.3, that means 70% of Y’s variance is due to things other than X. That’s not a model failure. That’s information.

Let me be more concrete.

You’re estimating the effect of education on income. You run:

Income = β₀ + β₁·Education + ε

Your R² is 0.15.

Does this mean your model is bad? No. It means 85% of income variance comes from things other than years of schooling. Family wealth. Luck. Geography. Timing.

But β₁ is still meaningful. It tells you: holding everything else random, one more year of education is associated with β₁ more dollars of income.

That’s not a prediction. That’s an inference about the world.

Claude writes code that computes β₁. But it cannot tell you what β₁ means in context. Because meaning requires assumptions. Assumptions about what “holding everything else random” means. Assumptions about what you’re trying to learn.

The Second Thing Claude Gets Wrong: Statistical Significance Is Not About Truth

You show Claude your regression output:

Coefficient: 1.23
Std. Error:  0.45
p-value:     0.006

Claude says: “The coefficient is statistically significant at the 0.05 level, indicating a strong relationship.”

This is not what p-values mean.

A p-value is not “the probability the coefficient is real.” It’s not “the probability the relationship exists.” It’s not even “the strength of the relationship.”

A p-value is this: If the true coefficient were exactly zero, how surprised should I be to see data this extreme?

That’s it. That’s all it is.

If p = 0.006, it means: “If there’s actually no relationship, I’d see data this unusual less than 1% of the time.”

It does not mean:

• The relationship is strong (that’s the effect size)
• The relationship is important (that’s domain knowledge)
• The relationship is causal (that requires identifying assumptions)
• The relationship will replicate (that requires external validity)

Here’s why this matters.

With n = 10,000 observations, even tiny, meaningless relationships become “statistically significant.” A coefficient of 0.001 with p < 0.001 might be “significant” but completely useless.

With n = 50 observations, large, important relationships might be “non-significant” simply because you don’t have enough data.

Statistical significance is about sample size and noise. Not about truth. Not about importance.

Humans know this. Humans look at effect sizes, confidence intervals, and domain context.

Claude just pattern-matches “p < 0.05 means significant.”

The Third Thing Claude Gets Wrong: Residuals Tell You Where You’re Wrong

After running regression, every stats textbook says: “Check your residuals.”

Claude will dutifully tell you to plot residuals. It might even generate the code. But it won’t tell you why this matters more than any other diagnostic.

Your residuals are everything you didn’t explain.

If you modeled:

Y = β₀ + β₁X + ε

Then ε is everything Y depends on except X. Your error term. Your ignorance term.

When you plot residuals, you’re asking: “What does my ignorance look like?”

If your residuals have patterns, you’re missing something systematic.

• Residuals increase with X? You’re missing nonlinearity. • Residuals clump in groups? You’re missing a categorical variable. • Residuals correlate over time? You’re missing time trends. • Residuals have huge outliers? You’re missing measurement error or rare events.

This is diagnostic reasoning. Looking at the structure of what you got wrong to figure out what you’re missing.

Claude cannot do this.

It can generate residual plots. It can describe patterns. But it cannot reason backward from patterns to missing variables. Because that requires knowing what variables exist in your domain. What mechanisms could be at play. What theory says should matter.

Residual analysis is detective work. Claude is not a detective. It’s a transcript generator.

The Fourth Thing Claude Gets Wrong: Causality Requires Assumptions You Cannot Learn From Data

Here’s where it all falls apart.

You run:

Sales = β₀ + β₁·Advertising + ε

You get β₁ = 1.5, p < 0.001.

You ask Claude: “Does advertising cause sales to increase?”

Claude will hedge. It will say things like:

• “Correlation does not imply causation.”
• “There may be confounding variables.”
• “You would need a randomized experiment to establish causality.”

All true. All useless.

Here’s what Claude won’t tell you: Whether you can make a causal claim depends on assumptions you must state explicitly.

The key assumption: No omitted variable bias.

For β₁ to be a causal effect, you need: All variables that affect both advertising and sales are either included in the model or uncorrelated with advertising.

This is untestable from data alone. You cannot check this assumption by running more regressions. You have to argue for it. Using domain knowledge. Theory. Institutional context.

Example:

• If companies with better products advertise more and have higher sales anyway, β₁ is biased upward. Advertising looks effective when it’s not. • If companies advertise because sales are dropping (trying to recover), β₁ is biased downward. Advertising looks ineffective when it’s not.

To argue causality, you need to tell a story about why these confounders either don’t exist or are controlled for. That story requires knowledge outside the dataset.

Claude has no such knowledge.

It has seen millions of regression outputs. It has seen millions of papers saying “we control for X, Y, Z.” But it doesn’t know why X, Y, Z matter in your context. It doesn’t know what you forgot. It doesn’t know what’s unobservable in your setting.

Causal reasoning is counterfactual reasoning. “What would have happened if advertising were different, holding everything else fixed?”

That question cannot be answered by looking at data. It requires a model of the world. A theory of how things work. Human judgment.

The Proof. In Five Lines.

Let me state this more carefully so the weight of it lands.

Theorem (Gauss-Markov, 1821): The least squares estimator β̂ minimizes:

Σ(Yᵢ — β₀ — β₁Xᵢ)²

This is a mathematical fact. Calculus. Anyone can verify it.

What this gives you: An estimate of E[Y|X]. The conditional mean.

What this does NOT give you:

• Whether E[Y|X] is the right thing to estimate
• Whether the linear form is correct
• Whether X causes Y
• Whether your estimate generalizes to new settings
• Whether your estimate is practically meaningful

All of those require additional assumptions. Assumptions that are not in the data. Assumptions that must come from humans.

This is not a limitation of least squares. This is a fundamental property of inference.

You cannot learn assumptions from data. Assumptions are prior commitments. Beliefs about how the world works. Constraints you impose based on theory, context, or domain expertise.

LLMs do not have domain expertise. They have text.

The Part That Keeps Me Up

Here’s what bothers me.

Regression has been around for 200 years. It’s the foundation of statistics. Economics. Medicine. Social science. Policy analysis.

And yet, every year, I see more data scientists who treat it like deprecated technology. “Just use XGBoost.” “Neural networks are better.” “Regression is too simple.”

They’re missing the point entirely.

Regression is not about prediction accuracy. It’s about interpretable parameter estimation. You’re not trying to build the best predictor of Y. You’re trying to estimate β₁ and understand what it means.

When you switch to XGBoost, you gain prediction accuracy. You lose interpretation. You can no longer say “β₁ measures the effect of X on Y.” You can only say “X was important for prediction.”

Those are not the same thing.

Example: You’re estimating the effect of a drug on recovery time.

• Regression gives you: “On average, the drug reduces recovery time by 2.3 days (95% CI: 1.1 to 3.5 days), controlling for age, severity, and comorbidities.”

• XGBoost gives you: “The model predicts recovery times with 85% accuracy. The drug was the 4th most important feature.”

Which answer helps you decide if the drug works?

The first one. Because you need an effect size. With uncertainty. And you need to know what you controlled for.

LLMs will write both models for you. They will not tell you which question you’re actually asking.

What I Actually Think

Regression is not going away.

Yes, neural networks dominate computer vision. Yes, LLMs dominate NLP. Yes, gradient boosting dominates tabular prediction.

But none of those methods answer the questions regression answers:

• What is the average effect of X on Y?
• What is the uncertainty around that effect?
• What assumptions are we making?
• Can we interpret this estimate causally?

These questions still matter. They matter in medicine. They matter in policy. They matter in social science. They matter anywhere you need to make decisions based on estimated effects, not just predictions.

And these questions require humans.

Not because humans are smarter. Because humans understand context. Humans know what assumptions are plausible. Humans know what the numbers mean in the world.

Claude can write regression code. It can summarize output. It can generate plausible-sounding interpretations.

But it cannot tell you if your assumptions are reasonable. It cannot tell you if your estimate is meaningful. It cannot tell you what to do with the result.

That still requires a human who understands both the math and the domain.

Tell Me

Which statistical concept do you think LLMs misunderstand most: causality, uncertainty, or generalization?

Tags: #DataScience #Statistics #Regression #MachineLearning #AI #LLM #Claude #Causality #LinearRegression #Inference #Mathematics

References

The Gauss-Markov theorem is foundational. If you want the formal treatment:

Linear Models with R — Julian Faraway, 2014 — Chapman and Hall/CRC

For causal inference and why regression alone isn’t enough:

Causal Inference: The Mixtape — Scott Cunningham, 2021 — Yale University Press

For why R² is misunderstood:

The ASA Statement on Statistical Significance and P-Values — Wasserstein & Lazar, 2016 — The American Statistician

For the philosophy of statistical inference:

Statistical Rethinking — Richard McElreath, 2020 — Chapman and Hall/CRC