Ask someone how good their predictions are and you'll get a story. The time they called the election. The stock they bought before it ran. The match they knew was an upset. What you almost never get is a number — because a number can't be spun, and a story can. The Brier score is that number. It's the closest thing forecasting has to a lie detector, and once you understand it, you'll never look at a confident prediction the same way again.
We Brier-score our own AI against prediction markets every night, in public, so this isn't abstract for us. Let me show you exactly how it works, using a match we scored last night — including the part where we lost.
The one-sentence definition
A Brier score is the squared error of a probability. You made a forecast, reality happened, and the Brier score measures how far apart the two were. It runs from 0 to 2 for a three-way outcome like a football match: **0 is a perfect, God-tier call; higher is worse.** That's the whole thing. The genius is in what "squared" does to you.
A real example: France vs Morocco
Last night we published a locked, Bitcoin-timestamped forecast for France vs Morocco before kickoff. Our AI model said:
- France win **56%**, Draw **25%**, Morocco win **19%**
The prediction market, priced by thousands of people betting real money at the same instant, said:
- France win **62%**, Draw **25%**, Morocco win **14%**
France won. Now we grade both forecasts. The outcome as a vector is France = 1, Draw = 0, Morocco = 0. You take the difference between each probability and what actually happened, square it, and add them up.
**Our model:** (0.56 − 1)² + (0.25 − 0)² + (0.19 − 0)² ≈ 0.19 + 0.06 + 0.04 = **0.288**
**The market:** (0.62 − 1)² + (0.25 − 0)² + (0.14 − 0)² ≈ 0.14 + 0.06 + 0.02 = **0.225**
Lower is better, so the market beat us last night. It was more confident in the favourite, the favourite won, and it got rewarded for that conviction. We were more cautious on France — and on this particular night, caution cost us. That's on the public scoreboard, losses and all, because a track record you can edit isn't a track record.
Why the squaring matters
Here's the part most people miss. Why square the error instead of just measuring it? Because squaring punishes overconfidence far more harshly than modest mistakes — and overconfidence is the cardinal sin of forecasting.
Say two forecasters both back France, who then wins. One said 62%, the other said 99%. Under a naïve "were you right?" scoring, they tie — both "called it." Under Brier, the 62% call scores a gentle 0.14 on that outcome while a 95% call scores 0.0025. Fine — but flip the result. When the underdog wins, the 62% forecaster eats a 0.38 penalty and the 99% forecaster eats a brutal 0.98. The square turns a confident miss into a crater. That asymmetry is the point: it forces you to mean your numbers. Say 90% only when you'd stake real credibility on being wrong one time in ten.
This is why Brier scoring quietly demolishes pundits. The whole pundit business model is loud confidence with no downside for being wrong. Attach a Brier score and the loud-and-wrong get buried by the quiet-and-calibrated within a season.
Right once tells you nothing
Notice what a single match doesn't tell you. The market beat us on France–Morocco, but that's one data point drawn from a noisy process. A 56% call that loses isn't a "bad" forecast — a favourite priced at 56% is supposed to lose plenty of the time. You cannot grade a probability on one outcome any more than you can prove a coin is fair with a single flip.
That's the trap behind every "the model was wrong!" hot take. One result is a coin flip. The honest test is the average Brier across dozens or hundreds of forecasts — that's where skill separates from luck, and where a genuinely good forecaster pulls clear of a lucky one. Which is also why our board runs continuously through the tournament instead of stopping on any night we happen to look good.
Calibration, the thing Brier is really measuring
Underneath the arithmetic, Brier rewards one specific virtue: **calibration**. A calibrated forecaster is one whose 70%s happen about 70% of the time and whose 30%s happen about 30% of the time. Not bold. Not exciting. Just honest about uncertainty. Over a long run, the best possible Brier score belongs not to the boldest forecaster but to the most calibrated one — the person whose confidence matches reality.
That's a genuinely uncomfortable idea in a culture that celebrates the pundit who "called it." The Brier score doesn't care who called it. It cares whether your probabilities, taken as a whole, told the truth about how likely things were. Confidence is cheap; calibration is the actual skill.
How to use this yourself
You don't need to run the numbers to think in Brier terms. Three habits do most of the work:
- **Attach a probability to your claims,** then remember them. "Probably" is unfalsifiable. "70%" can be scored. - **Judge forecasts on the full record, not the highlight reel.** Anyone can be right once. Ask what they say across a hundred calls. - **Distrust the crater-makers.** When someone is 99% sure of anything contingent on the future, they're not being rigorous — they're setting themselves up for a Brier catastrophe and hoping you won't keep score.
Keeping score is the entire game. It's why we anchor every forecast into Bitcoin before the event and grade it in public afterward: so nobody, including us, can pretend a loss was a win. Last night the market was sharper than our model. Tonight might go the other way. The scoreboard — not the story — is what will say so.
*Educational content about forecasting and probability — not betting or financial advice.*