The Rooster Problem: Why Correlation Isn’t Causation and How to Test for the Difference.

Two analysts are given the same task: predict next week’s coat demand. Analyst R gets historical sales data. Analyst U gets the same data, plus Google search trends. Both make their predictions. Both are wrong sometimes. But one is consistently less wrong, and the gap between them is the entire point of this article. Does knowing what people are searching for actually help you predict what they will buy? That question has a formal, testable answer. It’s called Granger causality.

For 2,000 years philosophers argued about what “cause” actually means. In 1969, an economist named Clive Granger sidestepped the entire debate with one practical move. He said: forget philosophy. If X causes Y, then knowing the past of X should help you predict the future of Y better than you could from Y’s own past alone. Cause equals improves prediction. That’s it. Applied to your shop: does knowing last week’s Google searches for “long coat” help predict this week’s coat sales better than past sales alone? If yes — searches Granger-cause demand. If no — whatever pattern you saw was already inside the sales data.

Before you fall in love with this idea, learn its one honest limitation. Granger causality proves X reliably comes before and informs Y. It does not prove X physically makes Y happen. The classic example: a rooster crowing reliably precedes the sunrise and would improve your prediction of it, yet the rooster obviously doesn’t cause the sun to rise. Both are driven by a hidden third thing, Earth’s rotation. So why is it still useful? Because for forecasting, predictive precedence is exactly what you need. You don’t need trends to philosophically cause sales. You need them to help you predict sales before you place an inventory order. That’s what this test measures.

The race between Analyst R and Analyst U produces a number called the F-statistic. It answers one question: how big was U’s improvement over R, relative to what random noise would give you? F near 1 means the improvement is about what chance would produce, nothing real here. F much larger than 1 means the trend is doing genuine work. As always, the F-statistic comes with a p-value.

The rule is simple: p < 0.05 means the trend Granger-causes demand; p ≥ 0.05 means no detectable signal, failing to reject the null is not proof of zero effect, just absence of evidence.

This whole comparison is only valid if both series are stationary, drift-free (Article: Stationarity and the ADF Test) . Two series that both trend upward over time look related even when they aren’t, which would make the F-test scream “causality!” at a coincidence. So before any of the math below runs, both demand and the trend signal are passed through differencing until stationary. Stationarity is the entry ticket; everything below assumes it.

The math behind the race

The two competing models written out formally:

Restricted model (Analyst R — demand history only):

Yₜ = α + β₁Yₜ₋₁ + β₂Yₜ₋₂ + … + βₚYₜ₋ₚ + εₜ

Unrestricted model (Analyst U — demand + trend history):

Yₜ = α + β₁Yₜ₋₁ + … + βₚYₜ₋ₚ + γ₁Xₜ₋₁ + γ₂Xₜ₋₂ + … + γₚXₜ₋ₚ + εₜ

Where Y is demand, X is the Google Trends signal, and p is the number of lags tested. The γ coefficients are the ones being tested, if they’re all zero, trends add nothing.

The F-statistic:

F = [(SSRᵣ − SSRᵤ) / p] / [SSRᵤ / (n − 2p − 1)]

Where SSRᵣ is the restricted model’s total error, SSRᵤ is the unrestricted model’s total error,

p is the number of lags, and n is the number of observations used in the regression (the series length minus the p rows consumed by lagging). The denominator’s n − 2p − 1 is the unrestricted model's degrees of freedom: it has 1 intercept + p demand lags + p trend lags = 2p + 1 parameters.

A large F means SSRᵤ dropped far below SSRᵣ , Analyst U won by a real margin.

Lag selection by AIC:

AIC = n · ln(SSR/n) + 2k

We test whether last week’s searches predict demand, then two weeks ago, all the way back to eight weeks — two months of history. Rather than reporting whichever lag produces the best p-value, we let AIC automatically select the most honest depth: the one that fits well without using more history than necessary.

Where k is the number of parameters in the model. AIC rewards fit but penalises complexity, adding lags always reduces SSR slightly, but AIC charges a penalty for each one. The lag with the lowest AIC is the honest choice, not the lag with the lowest p-value.

from statsmodels.tsa.stattools import grangercausalitytests

# Stack as [demand, trend] — statsmodels convention
data = np.column_stack([demand_series, trend_series])

# Test lags 1 through 8
results = grangercausalitytests(data, maxlag=8, verbose=False)

# Pick lag by lowest AIC, not lowest p-value
aic_by_lag = {lag: results[lag][1][1].aic for lag in range(1, 9)}
optimal_lag = min(aic_by_lag, key=aic_by_lag.get)

# Extract F-stat and p-value at optimal lag
f_stat, p_value, _, _ = results[optimal_lag][0]["ssr_ftest"]
is_significant = p_value < 0.05

Two details matter here. First, the column order is deliberate, statsmodels tests whether column 1 (trend) predicts column 0 (demand), so swapping them runs the reverse test automatically. Second, ssr_ftest is exactly the SSR-based F formula above , the library computes both models and returns the comparison directly.

One forward test isn’t enough. What if demand actually predicts trends, people buy coats, then search to compare and review? That’s reverse causality, and it looks identical to the forward case if you only run one direction.

• To catch reverse causality, the pipeline runs all three directions- forward, reverse, and both simultaneously, and reports which pattern holds.

Separately, as a one-time validation of the method, not a step in the per-product pipeline — a placebo test feeds the model an irrelevant category’s trend paired with the real demand series. If the test found “causality” there, the whole approach would be suspect. When it correctly finds nothing, it confirms the real positives aren’t just the test saying yes to everything. (It uses a deliberately stricter bar — flagging anything below p = 0.10, not 0.05 — because a negative control should catch even a faint false signal.)

Granger causality is a rung above raw correlation: it demands that X precede Y and measurably improve the forecast, not merely move alongside it. But it stops short of proving mechanism, the rooster still can’t summon the sun. For a forecasting system, that’s a feature, not a flaw. You don’t need search trends to make people buy coats; you need them to help you predict the buying before you place the order. Predictive precedence is exactly the property your inventory decision depends on , so it’s exactly the property you should test.