No. Accuracy measures whether an agent can finish a task once, and agent reliability measures whether it finishes the task correctly every time. Most agent leaderboards report single-run success, often called pass@1. It tells you whether an agent can do a task. Agent reliability asks whether it will do the task correctly every time, under real conditions.
The two numbers drift apart because a single successful run can hide inconsistency. An agent can pass a task once and fail it the next time, and a leaderboard that scores one attempt never sees the second run. τ-bench shows the gap directly: gpt-4o succeeded on under half of tasks on a single attempt, and in the retail domain fewer than 25% of tasks succeeded on all eight attempts (Yao et al.).
Newer models haven't closed that gap. A Princeton study of 15 models found that recent capability gains yielded only small improvements in reliability (Rabanser et al., ICML 2026). On outcome consistency, where 1.0 means the same outcome on every run, models scored 0.58–0.84 on GAIA and 0.52–0.90 on τ-bench.
pass^k is the probability that an AI agent succeeds on all k attempts at the same task. It was introduced in τ-bench and is the metric closest to what users feel. pass@k only needs one success in k attempts, so it flatters the agent as k grows. pass^k goes the other way and exposes inconsistency.
| Metric | Question | Effect of more attempts |
|---|---|---|
| pass@k | Can it succeed at least once? | Goes up |
| pass^k | Does it succeed every time? | Goes down |
A customer who uses your agent eight times will, on the τ-bench evidence, hit at least one failure on most tasks.
Errors compound because agents chain steps, and every step has to succeed for the task to succeed. If each step succeeds independently with probability p, an n-step task succeeds with probability pn.
Most people picture AI as a question and an answer, maybe with a web search in between. That's one or two steps. An agent doing real work takes many more. Here is a customer-support agent handling a single refund request:
| Step | What the agent does |
|---|---|
| 1 | Reads the customer's message and works out what they want |
| 2 | Looks up the customer in the CRM |
| 3 | Finds the order in the order system |
| 4 | Checks the refund policy for that product |
| 5 | Confirms the payment cleared |
| 6 | Calculates the refund amount |
| 7 | Issues the refund through the payment API |
| 8 | Updates the order status |
| 9 | Emails the customer a confirmation |
| 10 | Logs the case for the support team |
Each step is a chance to pick the wrong tool, misread a result or hit a slow API. If every step works 95% of the time, the whole refund goes through correctly about 60% of the time.
Real systems are rarely that kind. Suppose the payment API in step 7 and the email service in step 9 hit rate limits or slow responses, and the agent has no timeouts or retry handling. If those two steps drop to 80%, the refund succeeds about 42% of the time. Rate limiting was the most damaging fault in ReliabilityBench's tests of production-like conditions (Gupta, 2026). Missing safeguards also turn failures into damage: with no timeout the run stalls, and with no idempotency key a retry can refund the customer twice.
| Per-step reliability | 5 steps | 10 steps | 20 steps |
|---|---|---|---|
| 90% | 59% | 35% | 12% |
| 95% | 77% | 60% | 36% |
| 99% | 95% | 90% | 82% |
| 99.5% | 98% | 95% | 90% |
At 95% per step, success drops below 50% at 14 steps. To reach 95% over 10 steps, each step needs about 99.5%. The ARE Manifesto calls this the march of nines: each extra nine takes as much work as the last.
So shorten chains where you can, add verification between steps, and fix the step-level causes like vague tools, missing timeouts and unbounded retries. The full list is in Best Practices for Building Reliable Agents.
Six metrics measure agent reliability across consistency, robustness, predictability, safety and cost.
| Metric | Dimension | How to measure |
|---|---|---|
| pass^k | Consistency | Run each task 5–8 times; count tasks where every run succeeds. |
| Rewording success | Robustness | Rephrase tasks without changing meaning and compare success. |
| Fault-injection success | Robustness | Inject timeouts, rate limits, partial responses and schema drift. |
| Calibration | Predictability | Compare the agent's stated confidence or "done" signal to the actual outcome. |
| Error severity | Safety | Classify failures by blast radius: cosmetic, recoverable or irreversible. |
| Cost per completed task | Efficiency | Total tokens and tool spend divided by successful completions. |
The four dimensions come from the Princeton framework. The fault-injection approach comes from ReliabilityBench, which found success fell from 96.9% to 88.1% under modest rewording, with rate limiting as the most damaging fault.
A silent failure is when an AI agent reports success after taking the wrong action, with no error signal. It is the most dangerous failure because nothing flags it. Princeton found confidence calibration was generally poor and hadn't meaningfully improved with newer models. If you only measure what agents say they did, you will overestimate reliability. Verify end state instead — database rows, API responses and files.
You improve a low pass^k score by fixing the causes in code that make runs inconsistent. Low pass^k usually traces back to a handful of causes you can find in code, such as placeholder tool descriptions, network calls with no timeout, missing max-turns limits and unhandled tool errors. Trustabl's open-source Agent Reliability Analyzer flags these deterministically, so you can fix them before you run a single eval. To stop them appearing in the first place, Trustabl Forge gives coding agents like Claude Code and Cursor a skill built from the same rules, so the code they write already avoids these patterns. See Reliable Agents Start Before Runtime.
pass@k counts a task as solved if any one of k attempts succeeds, so it rises as k grows. pass^k requires every attempt to succeed, so it falls as k grows and exposes inconsistency.
Five to eight runs per task is a practical starting point. τ-bench reported results up to pass^8, where retail-domain success fell below 25%.
Related Blogs
Agent Reliability Resources