Autonomous Agent Governance Failures: What Actually Breaks in Production

I run 23 autonomous agents in production. I have watched every category of governance failure described below happen to me or to teams I advise. This is the pattern catalog, the mechanisms behind each failure, and what the fixes look like when you stop treating the LLM as the control plane.

The four failure modes that cover 90% of incidents

After tracking incidents across my own fleet and ~40 teams I've talked to running agents in anger, the taxonomy collapses to four things:

Failure mode	What it looks like	Typical blast radius
Unscoped tool access	Agent given broad API keys, database creds, or shell	Data exfil, destructive writes, lateral movement
Non-deterministic policy	Guardrails enforced by another LLM or a system prompt	Jailbreak bypass, silent policy drift
Missing execution provenance	Logs exist but aren't tamper-evident or complete	Cannot reconstruct incidents, cannot prove compliance
No circuit breakers	No hard caps on spend, loops, or recursive agent calls	Runaway cost, infinite planning loops, thundering herd

None of these are theoretical. Each maps to incidents that have been publicly reported or I've personally debugged at 2am.

Unscoped tool access: the $47 incident and worse

My own opening story: an early research agent had an OpenAI key with no monthly cap and a loop bug that kept re-summarizing its own summaries. $47 in API fees before I noticed. That's the cheap version.

The expensive version is the Replit incident pattern — an agent with write access to production databases executing a destructive operation because a tool description said "clean up test data." Or a LangChain agent with an unscoped HTTP tool being prompt-injected via a scraped webpage into POSTing credentials to an attacker-controlled endpoint. Simon Willison has documented the mechanics of this class repeatedly; it is not a model problem, it is a permissions problem.

The fix is boring and known: every tool call gets a capability scoped to the minimum required action, evaluated at the orchestrator, with the agent holding no long-lived credentials. If you have ever written an IAM policy, you know how to do this. Most agent frameworks make it harder than it should be because they default to "give the agent the whole API."

Non-deterministic policy enforcement

This is the failure mode I see most often in teams that think they've solved governance. The pattern:

• Policy is written in a system prompt ("never send emails to external domains")
• Or policy is enforced by a second LLM ("LLM-as-judge checks each action")
• Or policy is a Guardrails-style regex/classifier on model output

All three share one flaw: the enforcement mechanism lives inside the same probabilistic trust boundary as the thing being governed. A sufficiently clever input can move both at once. The DPD chatbot swearing-at-customers incident, the Chevrolet dealership agreeing to sell a Tahoe for $1, the countless leaked system prompts — these are all LLMs asked to enforce rules on themselves.

Deterministic policy lives outside the model. A concrete shape:

# Policy evaluated by OPA or equivalent, not by an LLM
package agent.tools.email

default allow = false

allow {
  input.tool == "send_email"
  input.recipient_domain == "walkosystems.com"
  input.daily_count < 50
  input.agent_identity.signed_by == "ed25519:a3f2..."
}

The agent proposes an action. A separate evaluator with its own code path says yes or no. The LLM cannot talk the evaluator out of it, because the evaluator does not read natural language.

Missing execution provenance

When an agent does something wrong — and they will — the first question is: what exactly happened, in what order, triggered by what input, producing what output? Most teams answer this with application logs written to stdout and shipped to Datadog. That's insufficient for three reasons:

1. Completeness. If the logging is in user code, a crash or exception path can skip it. You learn about the failure from the blast radius, not the logs.
2. Tamper-evidence. A compromised agent can rewrite its own history. Without signed, append-only records, you cannot rule this out after the fact.
3. Reconstruction. Reconstructing a non-deterministic run requires capturing the full prompt, tool I/O, model version, temperature, and RNG seed. Application logs almost never capture all of this.

The working pattern is an execution kernel that signs every step with an ed25519 key, writes to an append-only log, and makes the signing key inaccessible to the agent. This is how you answer a regulator, a customer, or your own postmortem. This is what Sift does for our fleet; it's also what you can build yourself if you want to.

No circuit breakers

The final failure class is the simplest to fix and the one teams skip most often. Every autonomous agent needs hard numerical bounds at the orchestrator layer:

• Max tokens per run, per hour, per day
• Max tool calls per run
• Max recursive agent invocations
• Max spend, denominated in dollars, checked before each model call
• Wall-clock timeout on the whole run

Not soft limits in a prompt. Hard limits in code that raise and halt. The AutoGPT-era incidents of agents spawning agents spawning agents were all missing this. So was my $47 loop.

Why the usual answers fall short

The current market responses to agent governance mostly address one layer:

• Bedrock Guardrails, NeMo Guardrails, Lakera Guard — content filtering on inputs and outputs. Useful for prompt injection detection, insufficient for action-layer governance.
• LangSmith, LangFuse, Arize — observability. Tells you what happened, doesn't prevent it or make the record tamper-evident.
• Anthropic's and OpenAI's built-in safety training — reduces the probability of bad model outputs. Does nothing about unscoped tool access or missing audit trails.

These are complements, not substitutes. The layer missing from most stacks is a deterministic kernel between the agent and its tools that evaluates policy, enforces budgets, and signs the execution record.

What a working governance stack looks like

The minimum viable version, in order of what to build first:

1. Capability-scoped tool access. Short-lived tokens, per-call authorization, no long-lived credentials held by agents.
2. Deterministic policy evaluation. OPA, Cedar, or equivalent — policy as code, evaluated outside the model.
3. Hard circuit breakers. Dollar limits, loop limits, timeouts. Fail closed.
4. Signed execution logs. ed25519 or similar, append-only, keys inaccessible to the agent process.
5. Human-in-the-loop for defined action classes. Writes above a threshold, new domains, new tool combinations.

This is what I built Sift to handle, because gluing these five layers together by hand across 23 agents was itself a source of failures. But the pattern matters more than any specific implementation. If you take one thing from this: the model is not the control plane. Treat it like an untrusted client making proposals, and put the governance where you can reason about it deterministically.