Why Calibrated Forecasts Beat Point Estimates Every Time

Ask a planning team what they want from a forecast and most will say accuracy. Ask what decision they are about to make and the answer is almost always about risk: how much safety stock to hold, how many people to schedule, how large a budget to commit. Those are questions about the tails of a distribution, not about its center. A point estimate, no matter how accurate on average, throws away exactly the information the decision depends on.

The cost of asymmetry

Consider inventory. If demand comes in below your forecast, you carry excess stock and tie up working capital. If it comes in above, you stock out and lose the sale, and possibly the customer. These costs are rarely symmetric. Stocking out of a high-margin product during peak season costs far more than holding a little extra. A point forecast at the mean is optimal only when your loss function is symmetric and the distribution is well-behaved, which is almost never true in practice.

The correct object to forecast is the full predictive distribution. Once you have it, you can choose the quantile that matches your economics. For a product where stockouts are ten times more costly than overstock, you plan to a high service-level quantile, not the mean. The forecast stops being a number to be right about and becomes an input to an explicit optimization.

Calibration is the property that matters

A distributional forecast is only useful if it is calibrated. Calibration means that when the model says there is a 90 percent chance demand falls below some level, demand actually falls below that level 90 percent of the time across many forecasts. A model can be sharp, producing narrow intervals, while being badly miscalibrated, and narrow-but-wrong intervals are worse than honest wide ones because they inspire false confidence.

We evaluate calibration with reliability diagrams and with proper scoring rules. Pinball loss, also called quantile loss, rewards forecasts that place their quantiles correctly and penalizes both overconfidence and underconfidence. Continuous ranked probability score generalizes this across the whole distribution. Optimizing these scores directly, rather than mean squared error, produces forecasts whose intervals you can actually trust.

Temporal fusion transformers, with baselines

For the conditional mean and quantiles, attention-based sequence models like the temporal fusion transformer are powerful. They handle known future covariates such as holidays and promotions, they learn seasonality across many related series jointly, and their attention weights offer a degree of interpretability that pure recurrent models lack. But power is not a license to skip rigor. We benchmark every deep model against classical baselines: exponential smoothing, ARIMA, and seasonal-naive. If the transformer cannot beat seasonal-naive on backtested pinball loss for a given series, we ship the baseline. Many series are genuinely simple, and a simple model that is calibrated beats a complex one that is not.

Conformal prediction for honest intervals

Even a good model can produce intervals that are systematically too narrow. Conformal prediction offers a distribution-free correction: using a held-out calibration set, it adjusts interval widths so that empirical coverage matches the nominal level. The appeal is that conformal guarantees hold regardless of whether the underlying model is well-specified. We layer conformal calibration on top of quantile forecasts to get intervals that are both sharp and honest, which is the combination planners need.

Reconciliation keeps the story coherent

Forecasts rarely live in isolation. You forecast demand at the SKU level, but you also need coherent totals by category, region, and channel. Forecast independently and the parts will not sum to the whole. Hierarchical reconciliation resolves this by projecting the base forecasts onto a coherent structure, typically with a minimum-trace optimal combination that also reduces error by borrowing strength across levels. The result is a set of forecasts that are internally consistent, so the number the executive sees at the top matches the numbers the planners work with at the bottom.

From forecast to decision

The final step is the one teams most often skip: connecting the distribution to the decision. A forecast that ends in a chart has not done its job. The distribution should flow into an explicit optimization that accounts for your costs, constraints, and service-level targets, and it should return the quantity to order, the shifts to schedule, or the budget to set. When forecasting is wired directly into the decision this way, calibration stops being an academic nicety and becomes the difference between a plan that holds and one that fails at the worst possible moment.

Point estimates feel comfortable because they are simple. But the future is a distribution, and pretending otherwise does not make it so. Teams that forecast distributions, calibrate them honestly, and plan against explicit service levels consistently outperform teams that chase a single accurate number.