7,426 forecasts against Polymarket: exactly where the market beat our models

Since September our statistical models have made 7,426 probability calls on Polymarket questions. Each call is stored next to the market price at the moment it was made, and the file holding it is timestamped into Bitcoin with OpenTimestamps. When the market resolves, both numbers are scored with the Brier score (lower is better).

On the 7,048 calls that have resolved, the market scores 0.161 and our models score 0.194. The market wins. This post is about where it wins, because the answer is more specific, and more useful, than "markets are efficient".

Everything below can be reproduced from the public dataset in a few lines of pandas (code at the end).

Headline numbers

Weather is 87% of the sample, so it dominates the total. Without weather the gap is small (0.130 vs 0.123 on 920 calls). The only track where we are ahead is politics, on 23 calls, which proves nothing.

One more number worth keeping in mind: our call scored better than the market price on 50.5% of individual questions. We win about half the coin flips, and lose the total anyway. The losses are bigger than the wins, and the rest of this post is about why.

Finding 1: we were far too sure that unlikely things would not happen

Take every call where our model said 10% or less. There are 2,176 of them.

• our average probability: 3.8%
• the market's average price on the same questions: 16.5%
• how often the event actually happened: 17.0%

The market was almost exactly right; we were off by a factor of four. Most of these are weather "band" questions (will the high temperature land in a given 2-degree range). Our models put a narrow, confident distribution on the temperature and gave neighbouring bands almost nothing. Reality was wider than the model.

The market's own calibration, by contrast, is close to the diagonal everywhere it has data:

And ours:

So the error runs both ways: too low at the bottom, too high in the middle. That is the signature of a model that is overconfident about which band wins, not one that is biased in a single direction.

Finding 2: the more we disagreed with the market, the worse we did

Group the calls by the gap between our probability and the market price:

When we roughly agreed with the price, we tied it. Every step of disagreement cost us, and the large disagreements, the ones that would have looked like "edge", were the worst calls in the set.

Finding 3: mixing our model into the price only made it worse

A natural rescue is to blend: take some weighted average of our probability and the market's. If the model carried any information the price did not, some blend should beat the price alone.

Every amount of our model hurt. On this sample, the honest conclusion is that the models added nothing the price did not already know.

What we are not claiming

• This is one set of models, mostly on one kind of question. It says little about forecasting in general and nothing about other teams.
• Our first batch (calls from 14 to 26 September, 1,613 of them) was sealed late, on 26 September, after 1,394 had already resolved. Those rows are flagged sealed_before_resolution = false and are not evidence of forecasting in advance. They do not flatter us: on the 5,654 resolved calls sealed before their outcome, the result is the same (0.196 vs 0.160). Since 26 September every call is sealed minutes after it is made.
• Nothing here is a trading signal or advice. It is a record of misses, kept so it cannot be edited later.

Reproduce it

import pandas as pd
from datasets import load_dataset

f = load_dataset("neuportal/forecasts-vs-market", split="train").to_pandas()
f = f[f.status == "final"]

# headline
print(f[["brier_model", "brier_market"]].mean())

# finding 1: our 0-10% calls
low = f[f.our_p <= 0.10]
print(len(low), low.our_p.mean(), low.market_p_at_call.mean(), low.outcome.mean())

# finding 2: by disagreement
gap = (f.our_p - f.market_p_at_call).abs()
bins = pd.cut(gap, [0, .05, .1, .2, .3, 1], include_lowest=True)
print(f.groupby(bins)[["brier_model", "brier_market"]].mean())

# finding 3: blends
for w in [0, .25, .5, .75, 1]:
    p = w * f.our_p + (1 - w) * f.market_p_at_call
    print(w, ((p - f.outcome) ** 2).mean())

To check that a call existed when we say it did, open its proof_file in the proofs repository and drop the .ots file, then the .json, on opentimestamps.org. The Bitcoin block it reports is the latest moment that call could have been written.

• Data (DOI, CC BY 4.0): https://doi.org/10.5281/zenodo.24051884
• Proof files: https://github.com/neuportalai-ui/experiment-proofs
• Project page: https://neuportal.ai/experiment

Educational transparency data, not financial advice.