Decision-Grade Uncertainty: Conformal Prediction, Calibration, and Ensembling the Modeling Ladder

Why the layer above the modeling ladder is where senior modeling practice actually lives — and how to build it honestly.

Part 2 of the corn forecasting series.

Context

Part 1 of this series built the modeling ladder for forecasting corn futures: naive baselines through classical statsforecast through LightGBM through neural and foundation models, evaluated honestly with walk-forward cross-validation. The ladder produces point forecasts and, where models support it, distributional outputs.

What the ladder does not do, on its own, is make those distributional outputs trustworthy. A model that reports an 80% prediction interval but actually covers 55% of observations is worse than a point forecast that admits its limitations honestly. The next layer above the ladder — the layer that distinguishes accuracy-grade from decision-grade modeling — is uncertainty quantification done correctly.

This piece covers the four moves that constitute decision-grade uncertainty: diagnosing calibration, applying conformal prediction as a unifying distribution-free layer, handling the time-series violations of exchangeability that break standard conformal guarantees, and ensembling across the ladder in a way that compounds skill without corrupting calibration. It closes with the decision-modeling layer that translates forecasts into commercial action.

Calibration vs sharpness: the two qualities of a forecast

A probabilistic forecast has two distinct quality dimensions, and the modeling literature consistently conflates them.

Calibration asks whether the forecast distribution matches the empirical frequency of outcomes. If a model issues 80% prediction intervals, those intervals should cover the true value 80% of the time across many predictions. Calibration is about honesty.

Sharpness asks how concentrated the forecast distribution is. Narrower intervals are sharper. Sharpness is about information content — a forecast that says “the price will be between $4 and $6” is sharper than one that says “between $2 and $8.”

The two trade off. A trivially calibrated forecast can always be produced by widening intervals until coverage hits the target — but at the cost of all information content. The right objective is to maximize sharpness subject to calibration. Proper scoring rules like CRPS implement this trade-off automatically by penalizing both miscalibration and excessive width.

A model that is sharp but miscalibrated is dangerous in commercial use; the failure mode is silent overconfidence. A model that is calibrated but unsharp is honest about not knowing much, which is informative in its own right. The combination of well-calibrated and sharp is what decision-grade forecasting actually means.

Calibration diagnostics: how to know what you have

Three diagnostics catch most calibration failures.

Empirical coverage at multiple nominal levels. For a nominal α-level interval, report the fraction of test points that actually fall inside it. Aggregate across walk-forward folds. A well-calibrated 80% interval covers 78–82% empirically; a 95% interval covers 93–97%. Deviations larger than these bounds on a sample of ~150 folds indicate real miscalibration, not noise. Reporting coverage at multiple levels matters because a model can be well-calibrated at 80% and badly miscalibrated at 95% — the tails are where most miscalibration lives.

Reliability diagrams. Plot predicted quantile level against empirical frequency below the predicted quantile. A perfectly calibrated model traces the diagonal. Systematic deviation above the diagonal indicates the model’s quantiles are too low (overconfident in the lower tail); below indicates the opposite. The shape of the deviation diagnoses the failure: bowing toward the diagonal is mild miscalibration, while a sigmoid-shaped reliability curve indicates that the model is overconfident in the center and underconfident in the tails (a common pattern for neural methods on heavy-tailed data).

PIT (Probability Integral Transform) histograms. For each test observation, compute the cumulative probability the model assigns to a value at or below the actual outcome. If the model is calibrated, these PIT values are uniformly distributed on [0,1]. A U-shaped PIT histogram indicates underdispersion (intervals too narrow). An inverted-U indicates overdispersion (intervals too wide). A skewed PIT indicates biased forecasts.

def pit_values(y_true, quantile_grid, quantile_preds):
    """PIT values from a quantile-grid forecast.
    quantile_preds: shape (n_samples, n_quantiles).
    """
    pit = np.zeros(len(y_true))
    for i, y in enumerate(y_true):
        below = quantile_preds[i] <= y
        pit[i] = quantile_grid[below].max() if below.any() else 0.0
    return pit

In production forecasting, calibration almost always degrades faster than point accuracy when distribution shift occurs. A model that has been deployed for six months may still show acceptable RMSE while its 95% intervals now cover 70% of observations. Calibration diagnostics belong on the first page of any forecasting model’s monitoring dashboard, ahead of point-accuracy metrics.

Conformal prediction: the distribution-free unifying layer

The traditional way to produce prediction intervals — assume residuals are Gaussian, scale by 1.96 — works only when the assumption holds. For most real models on most real data, it does not. Conformal prediction provides distribution-free prediction intervals with finite-sample marginal coverage guarantees that hold for any underlying model.

The split conformal procedure is straightforward:

1. Split the training data into a proper training set and a calibration set.
2. Fit the model on the proper training set.
3. Compute conformity scores (typically absolute residuals) on the calibration set.
4. The (1−α)-quantile of these scores is the half-width of the prediction interval.
5. New prediction: model output ± this half-width.

from mapie.regression import MapieRegressor

mapie = MapieRegressor(estimator=base_model, method="plus", cv=10)
mapie.fit(X_train, y_train)
y_pred, y_pis = mapie.predict(X_test, alpha=[0.05, 0.20])
# y_pis shape: (n_samples, 2, n_alpha_levels)

The coverage guarantee holds in finite samples under one assumption: exchangeability between calibration and test data. For IID data, this is satisfied. For time series, it is not — and that is the central difficulty of applying conformal methods to forecasting.

One important caveat that senior readers should hold onto: standard conformal prediction guarantees marginal coverage (averaged over the test distribution), not conditional coverage (at every covariate value). A model can be marginally well-calibrated while being badly miscalibrated for specific market regimes or specific feature values. Conditional coverage requires additional structure and is generally not achievable distribution-free without further assumptions.

Conformalized Quantile Regression (CQR) extends the approach to heteroskedastic settings. Rather than wrapping a point predictor with constant-width intervals, CQR wraps a quantile regression model and adjusts its existing quantile predictions:

1. Train two quantile regression models at α/2 and 1−α/2 (e.g., 0.05 and 0.95).
2. On the calibration set, compute conformity scores: max(q_lo(x) − y, y − q_hi(x)).
3. Take the (1−α)-quantile of these scores.
4. Adjust the interval bounds outward by this offset.

CQR produces intervals that widen where the underlying model is uncertain and narrow where it is confident. For the corn forecasting ladder, CQR wrapped around quantile-objective LightGBM is the natural fit — the model already produces quantile estimates, and CQR repairs whatever miscalibration the raw quantiles exhibit.

Conformal for time series: the exchangeability problem

Standard conformal prediction requires that calibration and test data come from the same distribution. Time series data almost always violates this. Regime changes, evolving volatility, structural breaks — all break the exchangeability assumption that underwrites the coverage guarantee.

Three approaches address this.

Block conformal prediction uses contiguous blocks rather than individual observations for calibration, preserving local temporal structure. Coverage becomes approximate rather than exact, but holds reasonably well under mild non-stationarity.

Adaptive Conformal Inference (ACI) updates the target miscoverage level α online based on recent coverage. If the model has been under-covering, α decreases (producing wider intervals); if over-covering, α increases (narrower intervals). The update rule:

α_{t+1} = α_t + γ · (target_miscoverage − miscovered_t)

where miscovered_t ∈ {0,1} is whether the most recent interval missed coverage, and γ is a learning rate (typically 0.005 to 0.05). ACI provides long-run coverage guarantees even under arbitrary distribution shift, though instantaneous coverage may deviate during regime transitions.

Weighted conformal prediction weights calibration observations by their similarity to the test point’s covariates. Observations from different regimes or older periods receive lower weights. Useful when regime structure is known or estimable.

For commodity series with regime shifts (corn in 2012, 2020, 2022; natural gas in 2022; many energy series in 2008), ACI is the most defensible default. Block conformal is simpler to implement and adequate for slowly-changing series. Weighted approaches require domain judgment about what “similar” means.

The empirical coverage diagnostic from the calibration section becomes the operational signal: if a conformal procedure produces intervals that miss coverage on recent test folds, the exchangeability assumption is being violated and the method should switch from plain conformal to ACI or block-conformal.

Ensembling the ladder

Ensembling across the modeling ladder is the highest-leverage move available after each individual model has been honestly evaluated. The intuition is simple: different model families make different errors. A combination that exploits the decorrelation of errors typically beats every individual model.

Three ensembling approaches, in increasing sophistication:

Simple averaging. Equal weights on all models. Robust, no parameters to tune, surprisingly hard to beat in practice. For point forecasts, average the predictions. For distributional forecasts, average the quantile predictions at each level.

Inverse-error weighting. Weight each model by the inverse of its validation error. Models with lower walk-forward CV error get higher weight. Computationally trivial and usually improves over simple averaging when models differ substantially in skill.

Stacking. Train a meta-model that takes individual model predictions as features and learns to combine them. Most powerful but most prone to overfitting; requires out-of-fold prediction generation to avoid information leakage.

def weighted_quantile_ensemble(quantile_preds, weights):
    """Combine quantile forecasts from multiple models.
    quantile_preds: dict[model_name -> array (n_samples, n_quantiles)]
    weights: dict[model_name -> float]
    """
    norm = sum(weights.values())
    pooled = np.zeros_like(next(iter(quantile_preds.values())))
    for model, preds in quantile_preds.items():
        pooled += (weights[model] / norm) * preds
    return pooled

Two warnings deserve explicit emphasis. First, ensemble evaluation must use out-of-fold predictions for each component model, not in-fold predictions. Using in-fold predictions overstates ensemble gain because the meta-model effectively sees the labels through leakage. This error is endemic in published forecasting work and produces ensemble performance estimates that fail to replicate in production.

Second, ensemble weights estimated on the validation set should be regularized — toward equal weights for inverse-error schemes, toward zero for stacking coefficients. Unregularized stacking on small validation sets reliably finds noise; the resulting ensemble underperforms simple averaging out-of-sample.

A useful property of distributional ensembling: linear pooling of forecast CDFs produces a mixture distribution that inherits calibration from its components. If every component is calibrated, the linear pool is calibrated, though typically less sharp than the components. The same is not automatically true for quantile averaging or stacked predictions, which is a reason to prefer CDF-level pooling when full distributional output is available.

From forecast to decision

The terminal value of a forecast is the decision it enables. A forecasting exercise that stops at “the model has RMSE 3.2” has not yet produced commercial value. The decision-modeling layer translates distributional forecasts into actions, accounting for the asymmetric costs that real commercial problems exhibit.

The canonical example is the newsvendor problem. A buyer faces an uncertain demand for a perishable commodity. Overage costs c_o per unit unsold; underage costs c_u per unit of foregone sales. The optimal order quantity is the quantile of the demand distribution at level:

q* = c_u / (c_u + c_o)

For commodity hedging contexts, the asymmetry can be severe. The cost of being short physical supply in a tight market may be 5× the cost of carrying excess inventory. This implies hedging or purchasing should target the 83rd percentile of the demand forecast distribution, not the median.

Two implications follow.

The relevant model output is a specific quantile, not a point estimate. Forecasting infrastructure built around point predictions and symmetric loss is structurally mismatched to most commercial use cases. Quantile forecasting infrastructure — via quantile regression, conformal methods, or Bayesian posteriors — is the right substrate.

The relevant evaluation metric depends on the decision. Pinball loss at the decision-relevant quantile matters more than RMSE for a forecasting model whose output drives quantile-based decisions. A model that beats the ladder average on RMSE but loses at the 90th-percentile pinball loss is the wrong model for hedging applications. The metric the model is selected on should match the cost structure the decision faces. This is rarely the case in default forecasting infrastructure, and represents one of the most common sources of the “the model is accurate but the business doesn’t care” outcome.

Failure modes worth cataloging

Five patterns recur in real-world deployment.

1. Silent miscalibration. Point accuracy holds while interval coverage degrades. Detection requires continuous coverage monitoring, not RMSE dashboards alone.
2. Regime-change blowup. Models trained on a stable regime fail when conditions shift — commodity supply shocks, energy crises, policy changes. Mitigation: adaptive conformal, regime-aware features, ensemble diversity across model families.
3. Ensemble overfitting. Stacking coefficients estimated on small validation sets fit noise. Mitigation: regularize toward equal weights, use longer validation periods, cross-validate the meta-model.
4. Decision-grade vs accuracy-grade confusion. A model selected on RMSE is used in a context requiring extreme-quantile accuracy. Mitigation: align the evaluation metric with the decision cost structure from the start.
5. Foundation model overconfidence. Zero-shot foundation models often produce intervals that are systematically too narrow, especially on series with high volatility. Mitigation: wrap them in conformal layers before any production use; do not deploy raw foundation-model intervals.

Conclusion

The modeling ladder built in Part 1 is a necessary foundation but not a sufficient one. What makes the ladder commercially valuable — what makes its output decision-grade rather than merely accurate — is the uncertainty quantification layer that sits above it: calibrated intervals, distribution-free guarantees, ensembling that compounds skill without corrupting honesty, and decision modeling that closes the loop from forecast to action.

The competitive advantage referenced at the end of Part 1 — judgment about what to predict, how to evaluate, what to trust, and what to do with the answer — operates almost entirely in this upper layer. The model-fitting work has commoditized; uncertainty quantification, calibration diagnosis, and decision linkage have not. They are where senior practitioners earn the difference between accuracy and value.

Part 3 will report empirical results from the corn forecasting ladder: calibration diagnostics for each tier, conformal-wrapped intervals, ensemble performance, and the cases — if any — where decision-grade evaluation reorders the ranking that RMSE alone would suggest.

Part 3 — empirical results from the full ladder — forthcoming.

About the Author

Brian Curry is a Kansas City-based Data Scientist, AI expert, and founder of Vector1 Research, where he explores the frontier of Artificial Intelligence, Causal Inference, Agentic Systems, and Economic Modeling.

His work focuses on building intelligent systems that reveal the causal structures driving performance across marketing, content, and economics. He is the creator of PyCausalSim (causal discovery through simulation), Papilon (complex system optimization), Daedalus (causal and economic analysis), and Memory-Node Encapsulation (data structures for artificial episodic memory).