Running AI Agents in Production — Idempotency, Budgets, and Confidently Wrong Answers

• Introduction — 8,042 Duplicate Executions Out of 6,780 Traces
• Retryable Tool Calls — Idempotency Is Not Something the Framework Hands You
• Budgets and Step Limits — Two Different Failures
• Observability for Non-Deterministic Control Flow
• Per-Tool Permission Boundaries
• Confidently Wrong Answers and the Escalation Path
• What to Alert On
• Closing — Retryability Is a Property of the Tool Layer, Not the Prompt

Introduction — 8,042 Duplicate Executions Out of 6,780 Traces

A few days ago a post titled Ask GN: To those of you running agents with several MCP tools attached went up on GeekNews. The author analyzed 6,780 benchmark traces and found 8,042 duplicate tool executions, and after filtering out harmless repetitions such as task-completion declarations, 4,249 remained. Counting only the ones that can be confirmed by entity ID, there are 159 cases in which a state-changing tool actually created a duplicate resource — a document, a spreadsheet, and so on. On top of that there are 199 carrying a timeout signature of the form "Timed out while waiting for response...10.0 seconds", plus 74 errors that look like resource conflicts.

The limits the author states about his own data matter. This data is entirely benchmark traces, not a real production service. And as he points out, most public datasets were never designed to detect duplicates in the first place — in an accuracy benchmark, a duplicate is simply tallied as a wrong answer. So these numbers are not an estimate of the production incidence rate; they are closer to an existence proof — "this failure mode exists, and nobody is counting it."

A comment proposed using session-scoped idempotency keys, and the author offers an exact rebuttal — you cannot inject a key of your own into a third-party API such as Notion or GitHub. That exchange is where this post begins.

Retryable Tool Calls — Idempotency Is Not Something the Framework Hands You

The structure of the agent loop creates the problem. Roughly, the loop goes like this — call the model, execute any tool calls in the response, append the results to the conversation, call the model again. Retries then happen independently at three layers.

The HTTP layer. The SDK automatically retries 429s, 5xxs, and connection errors. Most clients ship with two retries enabled by default. Timeouts are retried too, so in the worst case the actual wait is the timeout multiplied by the retry count plus one.

The tool layer. The tool implementation retries internally.

The orchestrator layer. If the run as a whole fails, it is restarted from the beginning.

The three layers know nothing about each other. The most dangerous combination is the third. If you were calling three tools in sequence, the second timed out, and the orchestrator retries the entire run, the first tool has already sent an email, and the second may in fact have written to the DB and merely failed to return a response. After the retry you are left with two emails and two records.

What makes timeouts especially nasty is that the absence of a response does not mean failure. The fact that 199 timeout signatures were tallied separately in the analysis above points at exactly this problem.

The skeleton of the fix is an old one — attach a deterministic key to every tool call that has a side effect, and keep the execution ledger on your side.

import hashlib, json

def idempotency_key(run_id: str, step_index: int, tool: str, args: dict) -> str:
    # 1) Fields that differ between retries MUST be excluded.
    #    Timestamps, request UUIDs, retry counters, anything computed "now".
    volatile = {"request_id", "timestamp", "now", "attempt", "trace_id"}
    stable = {k: v for k, v in args.items() if k not in volatile}

    # 2) Pin the serialization. Without sort_keys the same args yield a different key.
    material = json.dumps(
        {"run": run_id, "step": step_index, "tool": tool, "args": stable},
        sort_keys=True, separators=(",", ":"), ensure_ascii=False,
    )
    return hashlib.sha256(material.encode()).hexdigest()

Whether to put step_index into the key is a design judgment. Including it means only "the same step of the same run" gets deduplicated, so it does not block the legitimate behavior of a model deliberately calling the same tool twice. Leaving it out blocks more aggressively but also swallows legitimate repeat calls. I default to including it, and use a run-wide key only for tools that cannot be undone.

Back to the third-party API problem: when you cannot inject a key, three things are available.

First, your own execution ledger. Before calling the tool, look the ledger up by key, and if there is already a successful record, return the stored response as-is. A window of around 24 hours works well. The ledger has to record "call started" and "call finished" separately in order to handle the timeout case.

ledger[key] = {
  state:        started | succeeded | failed,
  started_at:   ...,
  external_id:  external ID of the created resource (on success),
  response:     cached response,
}

# state == started but stale = we do not know the outcome.
# Retrying here duplicates; not retrying loses. On to the next item.

Second, read-before-write. Most APIs will not take an idempotency key, but they will take a search. Before creating a document, look for one with the same title and parent, and return that if it exists. It is not perfect (a race remains), but it catches most cases like the 159 in the data above.

Third, planting a natural key yourself. Put the run key somewhere in the resource you create — a title suffix, a description field, a custom property, a label. It is messy, but it makes read-before-write precise.

And if you declare a side-effect class at the moment you first define a tool, every policy afterward hangs off that class. The table further down covers this.

One more thing. When you return tool results to the model, you must return failed calls as results too. Swallow the error silently and the model sees a tool call with no response and repeats the same call. Most APIs let you flag a tool result as an error, so use that. The same goes for parallel tool calls — if one response contains several tool calls, the results have to come back bundled together in one turn as well. Split them up and the model learns to stop making parallel calls (Anthropic's tool use documentation states this behavior explicitly).

Budgets and Step Limits — Two Different Failures

Agents break in two directions. They run forever, or they give up too early.

A step limit is a defense against infinite loops. It is a ceiling on loop iterations, and it is a hard cut. The mistake people frequently make here is treating a limit-triggered termination the same as a normal one. Hitting the limit means the task did not finish, but you cannot tell the difference from the return value alone. Return the termination reason explicitly, and do not tally a limit overrun as a success.

A budget is a different animal. It is a ceiling on tokens, cost, or wall-clock time. And there is a useful distinction that has emerged here over the past few years — the budget the model knows about versus the limit it does not.

Something like a response token cap is a hard cut the model does not know about. Hit it and the sentence is cut off mid-word. A task budget, on the other hand, is a value you tell the model, so it paces itself against the remaining budget and tries to wrap up. The Anthropic API's task budget works this way (the migration guide explains it) and its minimum is 20,000 tokens. The difference between the two matters a great deal in practice — a hard cut produces a truncated artifact, while a budget you disclosed produces a summarized one.

I recommend setting all three together.

• Step limit: infinite-loop defense. On overrun, state the reason and tally it as a failure.
• Cost budget: accumulated per run. Warn the model as it approaches the ceiling, and stop when it is exceeded.
• Wall-clock budget: especially on synchronous paths where a user is waiting. This has to be a deadline for the whole run, not a timeout on a single tool.

I want to emphasize that last item. An HTTP client's timeout setting is usually a per-chunk read timeout, so it keeps resetting as long as bytes keep trickling in. That is, it is not a ceiling on total elapsed time. You have to measure the deadline with a monotonic clock outside the run loop and cut it off yourself.

Observability for Non-Deterministic Control Flow

In an ordinary service the code defines the control flow, so you can read the code and know which path executed. With an agent the model picks a different path every time. Run the same input twice and the order of tool calls may differ. So you have to be able to reconstruct "what happened" after the fact, and this is not something you can bolt on later.

One run is one trace, and each step gets a child span. At minimum, this much has to be recorded.

# Per run
run.id                    same across retries. key material for the execution ledger.
run.trigger               user | schedule | webhook | retry
run.terminal_reason       completed | step_limit | budget | error | escalated

# Per step (one turn of the loop)
step.index                iteration number
step.stop_reason          why the model stopped (tool call / end / token limit / refusal)
step.model_id             needed to see whether the model changed mid-run
step.input_tokens         separated from cache reads/writes
step.output_tokens

# Per tool call
tool.name
tool.side_effect          none | read | write | irreversible
tool.idempotency_key
tool.attempt              1, 2, 3 ...
tool.outcome              ok | error | deduped | denied | timeout
tool.latency_ms
tool.external_id          if a resource was created

# Accumulated
budget.cost_spent
budget.wall_clock_ms
escalation.reason         if handed to a human, why

There are a few design judgments baked in.

The reason tool.outcome carries a separate deduped value is that the number of times deduplication fired is itself a metric. A sudden rise in it means the model has shifted toward calling the same tool repeatedly, or that retries are running away somewhere. It keeps the problem from becoming invisible just because it is being quietly handled.

The reason tool.side_effect goes on the span is post-hoc analysis. You have to be able to ask "how many times did an irreversible tool execute within one run."

Whether to record prompts and tool input/output bodies is a separate decision. It is essential for debugging, but personal data and secrets get stored verbatim. In practice a compromise works well — record metadata only by default, and store bodies with a short retention period only for failed runs. And those bodies are exactly your replay fixtures — store the tool responses too and you can deterministically reproduce the same situation after fixing your code. One production incident turns into one regression test.

The standardization picture is worth knowing as well. OpenTelemetry's GenAI semantic conventions are, as of mid-2026, still in development and have no 1.0. The v1.42.0 release on June 12, 2026 split the related attributes and spans out of the main repository into a dedicated one, but that is a separation, not a stabilization. Which is to say attribute names can still change. Follow the conventions, but it is safer to put a layer in between so your dashboards and alerts do not hang directly off attribute names.

Per-Tool Permission Boundaries

The accident starts with the sentence "I attached 20 tools to the agent." You have to sort tools into classes and apply a different policy to each class.

There are a few principles here.

Permissions attach to tools, not to agents. Decide that "this agent is an admin" and the permission surface quietly widens every time a tool is added. It is better to issue a separate least-privilege credential per tool.

Credentials must not enter the model's context. Configurations that put an API key in the system prompt or in tool arguments are still common, but that value stays in the conversation history forever and rides along into logs and summaries. Inject keys outside the call boundary, and let the model see only a placeholder.

Approval has to be per tool call. Session-scoped approvals like "allow mail sending for this session" are convenient, but they cannot distinguish the one message you approved from the ten you did not. An approval request has to contain exactly what will actually go out — recipient, subject, body. Approving on the basis of a summarized description is not approval, it is ceremony.

A setup that searches with a service account is a privilege escalation. This is especially true for agents that attach to an internal knowledge base or file system. Propagate the asker's identity all the way down to the tool call, and enforce authorization at the point closest to the resource.

Confidently Wrong Answers and the Escalation Path

The hardest failure mode to handle is not an error. It is the agent reporting that it completed the task when in fact it was wrong, or did only half of it, or did something nobody asked for. You can alert on errors, but you cannot alert on this — because every signal points at success.

Three things are needed together.

First, separate claims from evidence. At the prompt level, an instruction like "before reporting completion, check each claim against this run's tool results and report only what you can produce evidence for" actually works. Have it state explicitly which items are unverified. A useful observability metric falls out of this — a run that reported completion without ever calling a read tool is almost always suspicious.

Second, verify the termination condition in code. Put in an externally checkable criterion rather than the agent's self-report. Did the tests pass, did the ticket status change, does the file exist. If a task does not admit this, it is probably not a candidate for autonomous execution in the first place.

Third, make escalation one kind of normal termination. Many implementations treat "hand off to a human" as a failure path, which pressures the model not to hand off. Escalation has to be as legitimate a terminal state as success, and the following has to travel with it.

escalation = {
  reason:        "insufficient permission" | "contradictory information" | "budget exceeded" | "low confidence" | "approval required by policy",
  what_was_done: list of side effects already completed (including external resource IDs),
  what_remains:  work left over,
  blocking_fact: the specific point where it got stuck,
  run_id:        so the work can be picked up,
}

what_was_done is the crux. The first thing a human taking over needs to know is "what has already been executed." Without it the human starts over from the beginning and the side effects fire twice. This is where the earlier advice to keep an execution ledger pays off again.

Write the escalation conditions explicitly. Let it settle trivial judgments itself (variable names, picking between two equivalent approaches) and merely record them, and require it to ask about scope changes or irreversible actions. Without that distinction the agent goes to one of two extremes — asking nothing, or asking everything.

What to Alert On

It is common to have plenty of dashboards and no alerts. Splitting what should actually page you from what can wait for the weekly review gives you this.

Page immediately

• Executions of irreversible-class tools exceeding a threshold. Set it on an absolute count. Not a rate.
• An approval-required tool executed without approval. This is a value that has to be zero, so a single occurrence is an alert.
• A spike in the deduped rate. It is a leading indicator of runaway retries or a loop anomaly.
• Backlog time in the escalation queue. If nobody is looking, an escalation is simply a loss.
• A jump in the upper percentiles of cost per run. Look at p95 and p99, not the mean. Runaway runs are few and get buried in the mean.

Watch as trends

• The distribution of termination reasons. If the share of step_limit and budget is growing, the tasks got harder or the prompt regressed.
• The distribution of tool calls per run. A lengthening tail means the loop is wandering.
• The share of runs that reported completion without a read tool.
• The pass rate when you run regression tests off the replay fixtures.

There are also things you should not alert on. An individual tool call failure is not an alert target. It is normal behavior for an agent to see a tool fail and look for another path. Alert not on the failure itself but on runs that failed to recover after one.

Closing — Retryability Is a Property of the Tool Layer, Not the Prompt

What you confirm over and over in agent operations is that most of the problem sits not on the model side but in the layer beneath it. A model calling the same tool twice is not itself a bug. Two resources existing after it called twice is the bug.

• Attach a deterministic key and an execution ledger to every tool with a side effect. When a third party will not take an idempotency key, substitute read-before-write and a planted natural key. The framework will not do this for you.
• There are two kinds of limits. The hard cut the model does not know about produces a truncated artifact, and the budget you disclosed to the model produces a summarized one. Set both, but keep their roles distinct.
• Observability cannot be bolted on later. Record run-level traces, tool-level spans, and replay fixtures for failed runs from the start. The number of times deduplication fired is a metric too.
• Attach permissions to tools rather than agents, and take approval per call while showing the actual content.
• Make escalation a normal termination. If the list of side effects already executed does not travel with the handoff, the human starts over from scratch and fires the same side effects one more time.

And finally, I want to re-emphasize the caveat the author of that GeekNews post attached himself. Those numbers came from benchmark traces, and how often this failure happens in production is unknown because nobody is counting. Duplicate execution gets buried as a plain wrong answer in accuracy benchmarks, and in production it quietly disappears when a user sees two documents and deletes one. A failure nobody counts looks like a failure that does not exist.