I Fed a Berkeley Prof’s Anti-CP Article to Grok — Every Single Claim Flagged as Fake News

I pasted Ben Recht’s ‘Maybe You’re Wrong’ post into Grok and asked it to evaluate every major claim. The verdict? Overhyped, misleading, or just plain wrong.

The article “Maybe You’re Wrong” (Oct 23, 2025) by Ben Recht is a blog post based on Lecture 16 from his UC Berkeley graduate ML course Patterns, Predictions, and Actions. It uses a hypothetical autonomous EV taxi fleet to argue for prediction intervals (via quantiles) over point estimates for battery usage, then pivots to a critique of conformal prediction (CP) as the dominant modern framework for such uncertainty quantification (UQ).

Recht’s core thesis: CP is overhyped “rigor” that boils down to a simple hold-out quantile computation. It promises more than it delivers (“guarantees much less than you’d hope”), relies on hidden assumptions, and fetishizes mathematical guarantees that are practically weak or irrelevant. He contrasts this with everyday heuristics (e.g., trusting a coffee scale without theorems) and his broader view that “there are no distributions in machine learning.”

The post is part of Recht’s ongoing series questioning prediction intervals and CP hype (see his earlier “Cover Songs” and “Statistical Intervals” posts). Some points are valid cautions about overclaiming, but the article misrepresents or understates CP’s actual contributions, rigor, and practical value. Below, I evaluate and debunk the main claims one by one with evidence from the literature and responses to similar critiques.

Claim 1: CP “just amounts to using a test set to compute a quantile rather than a point prediction.” It is not innovative or special — just a “post-hoc justification” for risk-averse quantiles.

This is misleading and partially false. Split conformal prediction on residuals does look like “quantile of hold-out errors + add/subtract to prediction,” but that overlooks the key innovation:

• It delivers an exact finite-sample marginal coverage guarantee — P(Y{n+1} ∈ C(X{n+1})) ≥ 1−α (or up to 1−α + 1/(n+1)) — under only exchangeability of the data, even if the underlying model is completely misspecified or wrong. A naive empirical quantile of residuals (or plain quantile regression) has no such guarantee without additional assumptions (e.g., correct model, homoscedasticity, or asymptotics).
• CP is a general framework, not tied to quantiles or regression. It works with any nonconformity score (residuals, likelihood, distance to class prototypes, etc.) and extends naturally to classification (“I don’t know” sets), structured outputs, risk control, etc. The split version is simple, but full conformal, jackknife+, CV+, etc., avoid data splitting for better efficiency.
• Conformalized Quantile Regression (CQR, Romano et al. 2019) explicitly combines quantile regression (adaptive, heteroscedasticity-aware intervals) with CP to get both adaptivity and rigorous coverage — shorter/more efficient intervals than vanilla CP while preserving guarantees. This directly refutes “just quantiles.”

Recht’s EV battery example is exactly the use case CP was designed for: risk-averse decisions (e.g., “reserve extra charge so we cover 95% of trips”) where you want valid coverage without parametric assumptions. The method provides that rigorously where ad-hoc quantiles do not.

Claim 2: CP is not truly “distribution-free” — it still assumes data comes from a distribution with exchangeability, which is a “strong” assumption. “There are no distributions in machine learning.”

This is a semantic sleight-of-hand that conflates terminology. “Distribution-free” in statistics/CP literature specifically means no parametric assumptions (no Gaussianity, no specific form of P). The guarantee holds for any joint distribution whatsoever, as long as the (n+1) points (calibration + test) are exchangeable — a minimal, nonparametric assumption far weaker than what most UQ methods require (e.g., Bayesian methods need a correctly specified prior and likelihood; bootstraps often assume i.i.d. + regularity).

Exchangeability can be violated in practice (distribution shift, non-stationarity), and the literature openly acknowledges this — there are now many extensions (weighted CP, covariate-shift CP, online CP, etc.). But dismissing the entire framework because “the future might not be like the past” applies equally to every statistical method. Recht’s “no distributions” stance is a philosophical preference from his blog series, not a technical debunking.

Claim 3: CP guarantees are “ex ante” (probabilistic over training set + new point), not “ex post.” They are “deeply misleading” and not what practitioners actually want.

Partially true but overstated as a fatal flaw. Yes — the standard guarantee is marginal (averages over the randomness of the calibration set) and not conditional on the specific X_{n+1}=x or on a fixed calibration set. For small calibration sets, realized coverage after deployment can deviate. This is well-known and discussed in the CP literature.

However:

• With even moderate calibration sizes (hundreds of points, common in practice), the guarantee concentrates well.
• Extensions target conditional/group-conditional/asymptotic conditional coverage (e.g., via localized scores, Mondrian CP, or learn-then-test frameworks). Pure distribution-free exact conditional coverage is provably impossible without vacuous sets, but approximate versions exist and are useful.
• Compare to alternatives: most “ex post” methods (Bayesian credible intervals, neural uncertainties) have no finite-sample guarantees or rely on stronger assumptions that are routinely violated.

Responses to Recht’s similar earlier posts (e.g., David Stutz, Jessica Hullman) note that marginal coverage still provides a meaningful reference absent from ad-hoc methods, especially for safety-critical abstention, deferral, or selective prediction.

Claim 4: The 95% guarantees “regardless of sample size” are suspicious; you need “astronomically large” samples (PAC-style) for tight/useful intervals, per law of large numbers or old DKW inequality (1956).

True in spirit but not the gotcha Recht implies. Finite-sample CP is conservative with small n (the quantile order statistic can force wider sets), and tight adaptive intervals do benefit from larger calibration data. This is explicit in every CP paper and tutorial — no one claims “magic tight intervals from 10 points.”

But:

• The guarantee itself is non-asymptotic and holds for any n (unlike most bootstrap or CLT-based intervals).
• DKW gives uniform bands for empirical CDFs; CP generalizes this flexibly to any black-box predictor and nonconformity score.
• In practice, modern datasets often have thousands+ calibration points, and efficiency improves with better base models/scores (e.g., CQR).

Claim 5: Proponents “fetishize rigor”; quantiles are just heuristics, and not all UQ needs theorems (coffee scale example).

This is opinion, not debunkable fact, but it downplays why the rigor matters. In low-stakes settings, sure — heuristic intervals are fine. In high-stakes ones (the EV fleet, medical diagnosis, autonomous systems), a provable coverage guarantee gives a reference point that empirical calibration or “trust the scale” lacks. CP requires no extra assumptions or compute beyond what you’d do for a plain quantile — it is the “free lunch” when exchangeability approximately holds.

Literature (including work by Candès, Tibshirani, Angelopoulos, Bates, etc.) shows CP enabling concrete applications: controlling FDR in segmentations, selective classification with risk bounds, LLM factuality calibration, etc. It is not “just rigor for rigor’s sake.”

Overall Evaluation

Recht raises legitimate cautions that appear in the CP literature itself: marginal (not conditional) coverage isn’t always ideal, exchangeability can fail, guarantees don’t substitute for good modeling or domain knowledge, and hype sometimes overstates “distribution-free” as a panacea. His EV example usefully illustrates when any interval (quantile or CP) beats a point estimate.

However, the post systematically understates CP’s value by reducing it to “just quantiles,” ignoring the finite-sample validity it uniquely provides under weak assumptions, the extensions that address its limitations, and its proven utility in decision-making pipelines. It is not “much less than you’d hope” — it is often exactly what you’d hope for when you need model-agnostic, guaranteed coverage without strong parametric faith. The “fetishization” charge is more rhetoric than technical analysis; the framework has been stress-tested and refined precisely because practitioners and theorists take its guarantees seriously.

In short, the article is a provocative classroom provocation, but its claims do not hold up as a thorough debunking of conformal prediction. CP remains one of the most principled, practical tools for UQ in modern ML