Say you forecast a 70% chance of rain and it stays dry. Were you wrong? Not obviously — 30% things happen constantly. That awkward question is the whole reason probabilistic forecasting needs its own grading system. You can't mark a probability "right" or "wrong" the way you mark a multiple-choice test. You need a score that rewards forecasts for being both confident and correct, and quietly punishes them for being confident and wrong.
Two scores dominate the field: the Brier score and log loss (also called logarithmic loss or cross-entropy). They agree on the big picture and disagree in the details, and those details matter a lot when you're deciding which forecaster — human, model, or market — is actually better. Here is how each one works, where they differ, and why we score our public experiment in Brier.
Two Scores, One Job
Both metrics belong to a family called proper scoring rules. A scoring rule is "proper" if a forecaster minimizes their expected penalty only by reporting what they truly believe. In plain terms: honesty is the optimal strategy. If you think an event is 80% likely, no proper score lets you game a better grade by publishing 90% or 60%. That property is what makes these metrics trustworthy for comparing forecasters — nobody can win by bluffing.
Both scores are also penalties, so lower is better. Zero is a perfect score: you said 100% and it happened, or 0% and it didn't. Everything above zero is the cost of your uncertainty and your mistakes. The difference between the two is entirely in how they convert a probability and an outcome into that cost.
How the Brier Score Works
The Brier score is the simpler of the two. For a single yes/no forecast, you take the probability you assigned to "yes," subtract the actual result (1 if it happened, 0 if it didn't), and square that gap. Then you average across all your forecasts.
Say you predicted 0.7 for an event that happened. The error is 0.7 minus 1, which is minus 0.3; squared, that's 0.09. If instead the event had not happened, the error is 0.7 minus 0, squared to 0.49 — a much bigger penalty for the same forecast, because you leaned the wrong way. Average those squared errors over a season of predictions and you get a Brier score.
Because it's a squared difference between two numbers each living between 0 and 1, the Brier score for binary events is bounded between 0 and 1. A forecaster who blindly says 50% every time lands at 0.25 — a useful mental benchmark. Anything below 0.25 means you're adding real information; anything above means you'd have been better off shrugging.
How Log Loss Works
Log loss takes a different route. Instead of measuring distance, it measures surprise. For each forecast, it looks only at the probability you assigned to the outcome that actually occurred, then takes the negative natural logarithm of that number.
If the event happened and you'd said 0.7, your penalty is minus the log of 0.7, which is about 0.36. If you'd said 0.95, the penalty shrinks to about 0.05 — confidence that pays off is cheap. But if the event happened and you'd said only 0.1, the penalty balloons to about 2.30. You averaged these log penalties across all forecasts, and again, lower is better.
The logarithm is asking how "surprised" your forecast should have been by reality. A probability near the truth barely raises an eyebrow. A probability far from it is a shock, and the log function makes shocks expensive in a hurry.
Why Log Loss Hits Harder
Line the two up on the same mistake and the contrast is sharp. Suppose the event happens and your forecast was a stubborn 0.05. Brier charges you (0.05 − 1) squared, which is about 0.90 — bad, but capped. Log loss charges you minus the log of 0.05, roughly 3.00 — several times worse, and climbing.
That's the core personality difference. Brier's penalty grows with the square of your error, so it's steep but always finite. Log loss's penalty grows without bound as your stated probability for the true outcome approaches zero. Log loss cares intensely about your worst, most overconfident calls; Brier spreads its attention more evenly across every forecast you make.
The p = 0 Problem
Log loss has a famous failure mode. If you ever assign a probability of exactly 0 to something that then happens — or exactly 1 to something that doesn't — the log of zero is negative infinity, and your score is destroyed. One overconfident call can ruin an otherwise excellent record permanently.
In practice, people patch this by "clipping" probabilities away from the extremes, forbidding forecasts more certain than, say, 0.999 or 0.001. The patch works, but it's a reminder that log loss treats absolute certainty as a reckless bet, not a bold one. Brier has no such cliff: predict 0 for something that happens and you're charged exactly 1, the maximum possible single-forecast penalty and no more. For a public scoreboard where one dramatic miss shouldn't send a number to infinity, that bounded behavior is a genuine feature.
What Each One Is Good For
Neither score is "correct" — they answer slightly different questions.
Log loss is the natural choice when overconfidence is the sin you most want to punish, and when you need a metric that plugs into machine learning. It's smooth and differentiable, which is why it's the default loss function for training classifiers, and it has a clean information-theoretic reading in bits of surprise. If a single catastrophic error should dominate the evaluation — think safety-critical or tail-risk settings — log loss speaks your language.
Brier shines when you want interpretability, robustness, and a fair head-to-head between forecasters who occasionally blow a call. It reads on the same 0-to-1 scale as the probabilities themselves, it's hard to explode with one bad prediction, and it decomposes cleanly into components you can actually act on.
The Link to Calibration
That decomposition is where scoring meets calibration. The Brier score can be broken into three parts — reliability, resolution, and uncertainty. Reliability measures calibration: do your 70% forecasts happen about 70% of the time? Resolution measures how boldly and correctly you separate likely from unlikely events. Uncertainty is just how hard the events themselves were to call.
Log loss decomposes along similar lines, into a calibration term and a sharpness term. So both scores ultimately reward the same two virtues: being calibrated (your stated probabilities match real frequencies) and being sharp (you commit to strong probabilities when the evidence warrants). A good score is impossible without good calibration — which is exactly why calibration, not any single flashy prediction, is the honest way to judge a forecaster over time.
Why We Score in Brier
At NeuPortal we score every locked, timestamped forecast against prediction-market prices, and we use the Brier score to do it. Three reasons. It's bounded, so a public leaderboard can't be blown to infinity by one overconfident miss. It's interpretable, sitting on the same scale as the probabilities everyone can already read off a market. And it decomposes into calibration and resolution, which lets us show not just who's ahead but *why*.
Being ahead, we should add, is not currently us. Across 15 scored logs, the market leads the model 11 to 4 — the market's aggregated crowd is a brutally good forecaster, and our published record says so plainly. You can read every graded call at neuportal.ai/experiment. Whichever metric you prefer, the discipline is the same: lock the forecast before the event, score it honestly afterward, and let the number — not the narrative — decide who was actually good.
Educational content — not financial advice, and not a betting tip.