An AI Agent That Works in the Demo Will Still Fail on the Second Try. Here's Why.

The number vendors show you is the wrong number

A vendor pitching an AI agent will usually show you a demo that works, then quote a success rate from a benchmark: "92% task completion." Both numbers describe a single attempt. Neither tells you what happens when a customer runs the same request twice, or when your team puts the same agent in front of 500 similar tickets a week.

That distinction is the actual reliability question, and it's the one most buyers never ask.

What pass^k actually measures

The most useful benchmark for agentic tool use, τ-bench (tau-bench), scores models on something sharper than single-run accuracy: pass^k, the probability that k independent runs of the same task all succeed. Reported results on τ-bench's retail and airline domains show a consistent pattern across models: a strong pass^1 score on the first attempt, followed by a steep drop once you require the same outcome across repeated trials. One widely cited figure has a leading model scoring roughly 60% on a single retail-domain attempt, falling to under 40% once it has to succeed four times in a row on the identical task. Airline-domain tasks, which involve more state (bookings, seat changes, refund policies) typically score lower across the board.

The takeaway isn't the exact percentages — those shift with every model release. It's the shape of the curve. An agent that looks reliable in one run is not the same thing as an agent that is reliable, and the gap between those two is where production incidents live.

Why agents degrade on repeat, not just on hard cases

A simple script fails the same way every time you run it on the same input — that's what makes scripts debuggable. An LLM-driven agent doesn't have that property by default. The same prompt and the same tools can produce a different sequence of tool calls on different runs, because the model is sampling a plan, not executing a fixed one.

That would be a minor issue if each tool call were independent. It isn't. Agentic tasks are long-horizon: each tool call changes the state the next decision is made from. A wrong lookup in step two doesn't just cost you step two — the model reasons forward from the wrong state in steps three through seven, and because it's a language model, it's good at constructing a plausible-sounding justification for continuing down the wrong path rather than flagging that something is off. Errors compound instead of staying contained, which is a different failure shape than the single-call accuracy numbers in a model card suggest.

The failure that should worry you more: the silent one

The failure mode that actually causes production incidents isn't the agent that visibly errors out — that one gets noticed and retried. It's the agent that completes the task, logs a 200, and leaves the real-world state wrong: a refund issued to the wrong order, a support ticket closed with an answer that sounds right but references the wrong account, a scheduling change applied to the wrong row. The tool call succeeded. The business outcome didn't. Nothing in a standard API log distinguishes those two cases, which is exactly why they're expensive — they surface days later as a customer complaint, not as an error in your monitoring dashboard.

A useful way to break this down, consistent with how current agent-evaluation work categorizes it: failures cluster into tool selection (calling the wrong tool for the situation), argument extraction (calling the right tool with a malformed or hallucinated parameter), result utilization (misreading what the tool actually returned), and error recovery (not noticing, or not correctly handling, that a prior step failed). Most production incidents trace back to the last category — the agent had enough information to know something was wrong and proceeded anyway.

What actually reduces this, in practice

None of the fixes are exotic, and none of them are "use a smarter model":

• Hard caps, not best-effort limits. A fixed ceiling on tool calls and turns per task, enforced by code outside the model, not by instructing the model to "be efficient." When the cap is hit, return a controlled partial result or hand off to a human — don't let the loop keep guessing.
• Validate at every tool boundary. Check the model's arguments against a schema before the tool executes, and check the tool's result against expected shape before the model is allowed to act on it. This catches the argument-extraction and result-utilization failures before they propagate.
• Narrow retries, then stop. One or two retries on a transient failure, then a graceful fallback — not an indefinite loop that burns tool calls and money while making the state worse.
• Trace every decision, not just the final output. If you can't replay what the agent actually did step by step, you can't find which layer failed, and you'll keep fixing the wrong thing.
• Default to a workflow, not an agent, for well-defined tasks. If the sequence of steps is known in advance, a fixed pipeline with an LLM call at the steps that need judgment is more reliable and more debuggable than giving a model full autonomy over the whole task. Reserve open-ended agent loops for genuinely open-ended problems.

What this means if you're buying, not building

Ask any vendor pitching an agent to show you the same task, same inputs, run five times — not a single polished demo. If they can't produce that, you're being shown a demo's reliability, not the agent's. And ask what happens when a tool call fails midway through a multi-step task: a canned answer like "the model handles it" means no one has actually built the guardrail yet.

Takeaway: a single successful run tells you almost nothing about whether an agent is production-ready. The number that matters is what happens the second, third, and fourth time — and whether a bad step gets caught before it becomes a wrong answer that looks exactly like a right one.