Brier Reliability Diagram
A station that says “95% confident” and is right 95% of the time is calibrated. A station that says “95% confident” and is right 70% of the time is miscalibrated, and the lie compounds downstream.
Companion to The Verification Loop is the Bottleneck. The Brier score decomposes formally into reliability + resolution + uncertainty.
Drag a forecast point vertically. The dashed drop-line to the diagonal is the calibration error.
The decomposition
The Brier score for a probabilistic forecaster admits a clean decomposition (Murphy 1973):
BS = REL - RES + UNC
REL = mean weighted (predicted - observed)^2 [calibration error]
RES = mean weighted (observed - baseRate)^2 [resolution / sharpness]
UNC = baseRate * (1 - baseRate) [irreducible uncertainty]REL is what most teams miss. A station can have high accuracy on a held-out set and still emit posteriors whose confidence claims are systematically wrong. Held-out accuracy hides the calibration error completely. The downstream consumer believed the upstream's confidence and sized accordingly. The downstream pays the cost when the truth comes in.
RES is sharpness. A station that always predicts the base rate has zero resolution and is useless even when calibrated. The target is high resolution with low REL: sharp and calibrated. The preset “sharp + miscalibrated” is what most production AI pipelines actually look like.
See also
- The Verification Loop is the Bottleneck - this widget's parent post
- The Knowledge-Work Assembly Line - the framing piece
- Yield Collapse - why per-gate accept matters at all