An AI agent that issues your refund correctly once is a demo. An AI agent that issues it correctly every single time is a product. The distance between those two sentences has been the quiet problem of the whole agent boom, and until recently nobody published it as a number.
A benchmark for AI agents called **Thinkingbox** now does. It takes 507 business workflows, runs each one **20 separate times**, and reports not only how often an agent succeeds, but how often it succeeds on **all twenty attempts**. The headline pair, from the paper's own results table: the strongest model scores **66.50% pass@1** and **47.53% pass^20**.
That second number is the one your operations team will live with.
The two numbers, and the sharper pair underneath
Thinkingbox-bench is published as arXiv:2608.19741, under a title that is itself the argument: *One Success Isn't Reliability*.
Here is the pass@1 table as reported:
| Model | pass@1 | |---|---| | Claude Opus 5 | 66.50% | | GPT-5.4 | 65.36% | | GPT-5.6-sol | 61.91% | | Claude Sonnet 4.6 | 58.45% | | Qwen3.8-27B | 51.70% | | DeepSeek-V4-Pro | 43.26% | | Claude Opus 4.6 | 37.91% |
Ranked this way it is an ordinary leaderboard, and ordinary leaderboards are why the agent conversation has been stuck. Now add the second axis. Claude Opus 5 goes from 66.50% on a single attempt to **47.53%** when all twenty attempts must succeed - the smallest collapse in the group, and still nineteen points of it.
The sharper pair belongs to a smaller model. **Qwen3.8-27B posts the highest pass@20 in the field at 89.35%, and a pass^20 of 7.50%.** Given twenty tries it finds a winning path on nearly nine tasks in ten. Asked to repeat that path every time, it manages it on one task in thirteen.
Two models, two completely different failure shapes, and a single-attempt score cannot tell them apart. That is the finding. The authors call it the **discovery-reliability gap**.
What Thinkingbox actually tests
The 507 tasks are policy-conditioned workflows, which is a precise way of saying the agent is not asked to write anything. It is asked to *do* something inside a system that has state, under rules that constrain what it is allowed to do.
The domains and their task counts:
- Retail and e-commerce - 98 tasks - Travel and hospitality - 104 tasks - Auto insurance - 100 tasks - Neobank support - 104 tasks - Consulting IT and HR support - 101 tasks
The sandbox runs multi-turn interactions between a simulated user, the AI agent driving the tools, and isolated MCP-compatible tool sessions, capturing complete execution traces. Grading is done by **executable judges** against three things: the terminal state of the backend, the side effects along the way, and the dialogue. Of the 507 cases, 477 are graded on backend state only; 30 also carry a response requirement.
Two design choices in there matter more than the scores.
First, **the verdict is conjunctive**: a task passes only when every required condition holds. No partial credit for getting the refund amount right while cancelling the wrong line item. Production is conjunctive too. Nobody has ever been thanked for a mostly correct chargeback.
Second, **extra effects fail the task**. The checks accept valid trajectories and reject wrong, missing, *or extra* effects. An agent that solves your problem and also quietly updates a second record has not solved your problem.
pass@1, pass@20, pass^20 in plain English
Three questions, three different tests:
- **pass@1** - given one attempt, does it work? This is the demo, and the number every vendor quotes. - **pass@20** - given twenty attempts, does at least one work? This is research capability, and it is what you are really measuring when a human is in the loop, retrying, nudging, rewording. - **pass^20** - do all twenty attempts work? This is the only one of the three that describes an agent left alone.
The reason this distinction has suddenly become urgent is that AI agents have started running without a human watching. Assistants now continue working after the user closes the chat, and the flagship models are sold on computer use and multi-step professional tasks rather than on text. When a person is present, pass@20 is close to the lived experience: they will retry, and one good path is enough. When the agent runs a nightly queue of four hundred refunds, pass^k is the lived experience, and there is nobody to notice the nineteen that went a different way.
The arithmetic that says these trials are not coin flips
Here is where the published numbers get genuinely interesting, and this part is simple arithmetic anyone can redo.
If the twenty attempts at a task were independent draws with a fixed success probability, then a model with 66.50% per attempt would pass all twenty at 0.665^20 - about **0.03%**. The observed figure is **47.53%**, more than a thousand times higher.
Run the same check on Qwen3.8-27B from the other end. At 51.70% per independent attempt, the chance of at least one success in twenty would be 99.99995%. The observed pass@20 is **89.35%**.
Both directions break the same assumption, and they break it hard. These trials are not coin flips. What the numbers describe is a world where **difficulty lives in the task, not in the roll** - which means a model's task list splits into three buckets.
For Qwen3.8-27B, where both figures are published, the split is exact:
- **Always** - 7.50% of tasks succeed on all twenty attempts. - **Sometimes** - 81.85% are found at least once but not every time (89.35 minus 7.50). - **Never** - 10.65% are not solved once in twenty attempts (100 minus 89.35).
That middle bucket is four fifths of the benchmark, and it is the bucket nobody has a plan for. A never-task is honest: it fails loudly, you route it to a human, you move on. An always-task is a feature you can ship. A sometimes-task looks exactly like an always-task in the demo and exactly like a never-task on the night it matters.
This also quietly explains why retry logic has been disappointing teams. Retries pay off enormously when failure is random. When failure is a property of the task, a retry mostly buys you the same wrong answer with a fresh timestamp - and, in a stateful system, a second set of side effects.
The finding that should worry operators more than the scores
Buried under the leaderboard is the line that reframes the whole thing: many failed trials show **clean termination and valid state-changing actions**. The authors are blunt about what follows - response and tool-call-level signals are not clear proxies for end-to-end task completion.
Read that as an operations sentence. The agent finished. It did not throw. Every tool call it made was well-formed and did something real. Your traces are green, your error rate is excellent, your dashboards are calm - and the record in the backend is wrong.
Almost all agent monitoring in production today watches exactly the signals this paper says are not proxies: exceptions, tool-call success, latency, token spend, whether the run terminated. None of that separates a correct terminal state from a plausible one. The only thing that does is an executable check on the state itself, written by someone who knows what the state should be.
That is expensive, and it is the actual cost of running agents unattended. It is also, notably, the same discipline the benchmark had to build before it could measure anything at all.
What this looks like when you apply it to yourself
We run a forecasting system and publish its record, so the honest move is to turn this lens on our own work rather than only on someone else's table.
Our ledger holds **858 sealed forecasts**, of which **565 have resolved**. Every one was hashed with SHA-256 and anchored to the Bitcoin blockchain before it was published, which fixes the one thing a track record must fix: the text cannot be edited after the outcome is known.
The design target for our core band is 50% - across many forecasts, price should land inside it about half the time. More is a padded band, less is an overconfident one, and both are failures. Measured on resolved 24-hour forecasts at the time of writing:
- Majors, volatility-conditioned band: **62.0% core** on n=121 - Majors, the older unconditional band: **88.2% core** on n=51 - Wider universe, conditioned: **54.2% core** on n=59
The 88.2% is the padded band we published about in July and then fixed; leaving it in the table is the point of keeping a ledger. And the number we like least:
- **30-day horizon: 3.8% core on n=130.**
Not a rounding problem, not variance. At thirty days our narrow band is wrong essentially always, while the wide 80% band around it holds at 87.7%. The method knows roughly how far price can travel and has almost no idea where the middle is. That is our sometimes-task bucket, and we are not going to quietly retire the horizon to make the average look better.
There is also a third category in our ledger, added this week, which is the direct analogue of the benchmark's conjunctive grading: **9 forecasts marked void**. A sub-cent asset was rounded to a price of zero by our own serializer, and a forecast of zero is not a forecast that can be scored. It would have counted as a guaranteed miss - a number in our favour, in the sense that it made us look worse - and we still refuse to score it, because a ledger that counts nonsense in either direction stops being a measurement.
How to measure your own agent's pass^k this week
You do not need 507 tasks. You need one workflow that matters and an afternoon.
1. **Pick a task with state.** Something that writes: a refund, a ticket, a booking change, a record update. Read-only tasks hide this failure mode completely. 2. **Write the check before the run.** Describe the correct terminal state - every field that must change, and, just as important, everything that must *not* change. Make it a script, not a judgement call. 3. **Run it k times from a clean state.** Twenty is the paper's number; ten will already tell you something. Reset the backend between runs. If you cannot reset it, you do not have a test environment, you have production with optimism. 4. **Grade conjunctively.** One failed condition fails the run. Count extra side effects as failures. 5. **Report three numbers:** how many runs passed (pass@1 rate), whether any run passed (pass@k), and whether every run passed (pass^k).
The gap between the first and the third is your real deployment risk, expressed in one number. Most teams discover it is wider than their roadmap assumed.
What to ask a vendor, now that the vocabulary exists
- What is your pass^k, at what k, on tasks that change state? - How is a pass decided - by the model's own report, by tool-call success, or by an executable check on the final state? - Are extra side effects counted as failures? - What fraction of tasks are in the sometimes bucket, and what happens to them in production? - When the agent fails, how does the system know? Specifically: what signal fires when the run terminates cleanly and the state is wrong?
A vendor who can answer the last one has thought about this properly. A vendor who answers with an accuracy percentage and a demo video has answered a different question.
The part worth sitting with
The temptation with a result like this is to read it as a verdict on the technology: agents are not ready, come back next year. That is not what the table says. A model completing two thirds of unfamiliar stateful business workflows on the first try is a remarkable thing, and would have been a headline on its own eighteen months ago.
What the table says is that the industry has been reporting the easier of two available numbers, and the easier number describes a world with a person in it. Every demo is pass@1. Every conference talk is pass@1. Every screenshot is, definitionally, pass@1 - it is the run that worked, or it would not be a screenshot.
The move from assistants to agents is exactly the move from a world where a human absorbs the variance to one where nobody does. Pass^k is what that costs, and it is not a small correction to the story - for one model in this paper it is the difference between 89% and 7.5%.
We think the interesting work of the next year is not making the first number bigger. It is making the second number knowable: executable checks on real state, records that cannot be edited after the fact, and published failures alongside published wins. That is unglamorous, it is most of the actual engineering, and it is the only thing that converts a demo into something a business can lean on.
One success isn't reliability. The paper put it in the title, and the field is going to spend a while catching up to it.
*Educational content - not financial advice.*