agent design · failure modes · reliability

What breaks in autonomous agent loops

2026-08-03 · 7 min read

In short

The failure modes in autonomous agent products are consistent enough to be predicted, and almost none of them are model failures. They are accounting and accountability failures: work marked done that never happened, charges for attempts that failed, actions taken in a user's name without a decision, credentials concentrated so one breach is every customer's breach, and oversight ratios that make human review arithmetically impossible. Each has a specific design response, and each response is cheap if chosen before launch and expensive afterwards.

The interesting thing about autonomous agent failures is how few of them are model failures.

Read enough public complaint logs about "AI employee" products and the same handful of problems appear regardless of vendor, model or vertical. They are not about hallucination rates. They are about accounting, accountability and blast radius — decisions made in the architecture long before a token is generated.

A note on sourcing. The patterns below come from public reviews and published governance analyses of autonomous agent platforms in mid-2026. I have deliberately not named specific companies, and have not reproduced specific figures from those complaint logs, because I cannot independently verify individual claims and a post about honest reporting is a bad place to relay numbers on trust. The patterns are what generalise anyway. Everything I say about our own implementation is checkable in our code.

1. Phantom completions

The pattern: tasks marked done, with nothing behind them. Businesses "launched" that are a landing page over an empty space. A dashboard reporting high throughput while the actual delivered artifacts are a fraction of it.

The mechanism is mundane. Writing status = 'done' is one statement. Verifying that a real, reachable artifact came out the other end requires the artifact to have an identity, a location and a reference from the task that claims to have made it. The first is free and the second is work, so under deadline pressure the status field becomes the record of truth, and the record of truth becomes a story about what should have happened.

The design response: completion is defined by a resolvable reference, not a status value. When an action completes it persists a typed reference to what it produced, and the log renders a deep link to that artifact. If the link cannot resolve, the line cannot claim success. This makes the log a receipt rather than a narrative — and it is a genuinely uncomfortable constraint, because it means a partial success has to render as partial.

2. Charging for failure

The pattern: failed, duplicated or plainly wrong tasks consume credits anyway. Refunds owed under the vendor's own stated policy, not processed.

Charging at the attempt is the natural implementation, because the attempt is where the cost is actually incurred. The consequence is that unreliability is billed to the user, and the vendor's incentive to fix reliability is weakened by the fact that failures are revenue.

The design response: charge as late as possible, then make it idempotent. For agent work the charge is at approval, so nothing is held or spent at preparation time and a rejected draft is free. For manually triggered work — where the user is waiting on a result — the charge happens up front but the whole operation is wrapped so that any throw refunds it automatically, and a refund that itself fails is logged loudly for reconciliation rather than swallowed. Both paths spend through one Postgres function that will not bill the same attempt twice.

That last part matters more than it sounds. Agent systems retry constantly — network flakes, timeouts, double-clicked buttons — and a charge that is not idempotent turns every retry into a billing incident.

3. Action without a decision

The pattern: outreach sent in a user's name that they never saw, sometimes to consequential recipients, sometimes containing wrong names or stale prices.

This is the failure that cannot be walked back. A bad draft is a bad draft; a bad sent email is a relationship and a permanent record.

The design response: no code path from the agent loop to an outbound action. The loop terminates in a draft. Exactly one route executes anything, and it requires an explicit decision. Where we integrate with email, the most we do is open a pre-filled compose window in the user's own client — the human still presses send, because the alternative is us holding a send button. The longer argument for that boundary is here.

4. Credential concentration

The pattern: the platform provisions and holds the payment account, the repository, the email, the database. Convenient at signup, and it means two things: one breach at the platform is a breach of every customer simultaneously, and a lapsed subscription can strand assets the customer thought were theirs.

The design response: delegated access to accounts the user already owns, tokens encrypted at rest, no infrastructure provisioned on their behalf that we could hold. The blast radius of a compromise is bounded by what was delegated, and cancelling takes nothing with it. The related test is portability: if every artifact is text the user can export, there is nothing to be locked into.

5. The oversight ratio

The pattern, and the sharpest argument in the published governance critiques: one operator cannot supervise a thousand autonomous instances. Assume a very low incident rate — one percent. At a thousand instances that is ten situations a day requiring human judgement, arriving continuously, at a company that does not have ten people.

There is no version of centralised oversight that survives this arithmetic. The supervision degrades to sampling, and sampling of a long tail finds nothing.

The design response: do not centralise oversight. The person with the context, the stake and the judgement is the user, and they only have one instance to supervise — their own. This is the argument for the approval boundary that has nothing to do with trust or marketing: it is the only oversight model whose ratio does not get worse as you grow.

6. Accountability without review

The pattern: a user is legally responsible for actions taken in their name by software they never reviewed. Terms of service assign the liability; the product architecture makes review impossible.

You cannot fix this with a disclaimer, because the disclaimer is the problem — it allocates responsibility to the party with the least information. The only real fix is to make review possible and required before consequence, which is the same boundary again.

7. Output volume mistaken for outcome

The pattern: months of activity, extensive logs, little or nothing in revenue. The machinery runs; the market does not answer. And because the reporting measures throughput, the dashboard looks healthy the whole time.

This one is the easiest to build accidentally, because output is trivially measurable and outcome is not. An agent that reports "12 posts published, 340 emails sent" is telling you about its own effort.

The design response: ask whether it landed, and let the answer change behaviour. After work ships, the next report asks a one-tap question — replies, some engagement, nothing — and a run of nothing biases future decisions away from that approach and says so out loud: three posts, no bites, I want to try a different angle. An agent that can notice the market is not answering is worth considerably more than one that celebrates volume.

The pattern behind the pattern

Every item on this list is a decision about who bears the cost of the agent being wrong.

Charging at the attempt puts it on the user. Phantom completions put it on the user. Action without review puts it on the user, including legally. Credential concentration puts it on every user at once. Volume reporting hides it from everyone.

The alternative is more expensive to build and it means eating the cost of your own failures — which is the point, because it is the only arrangement where your incentive to be reliable is real. None of it requires a better model. All of it is much cheaper to decide before launch than after a complaint log exists.


Our version of this is an agent that prepares marketing work and waits: see what it produces from a URL, free and without an account.

Sources

  • Public complaint logs and review platforms, mid-2026

    Recurring patterns across user reviews of autonomous 'AI employee' platforms. Individual platforms are deliberately not named — see the note on sourcing in the post.

  • Independent AI governance reviews, 2026

    The oversight-ratio, liability and data-isolation arguments are drawn from published governance analyses of autonomous agent platforms.

  • Venturehand's own implementation

    Every claim about our design maps to shipped code and is described in docs/implementation-log/agent-loop.md.

Questions

What is a phantom completion in an AI agent?

A task the agent records as done when nothing usable was produced. It happens because recording that a step ran is trivial and verifying what it produced is not, so a status field gets written on the happy path regardless of the artifact. The fix is to make completion depend on a resolvable reference to a real output rather than on a status value.

Why do agent platforms charge for failed work?

Almost always because the charge sits at the attempt, where the cost is actually incurred, rather than at the delivered result. It is the easier implementation and it quietly transfers the cost of unreliability to the user. Moving the charge to the point of accepted output makes a failure cost the vendor instead, which is the correct incentive but a real cost line.

What is the oversight-ratio problem with autonomous agents?

One operator cannot meaningfully supervise thousands of autonomous instances. Even a very low incident rate produces more situations needing human judgement per day than any small team can handle, so supervision degrades to sampling. The structural answer is to distribute oversight to the person with the context and the stake — the user — rather than to centralise it.

Why is holding a customer's credentials a design risk?

Concentration. If a platform provisions and holds the accounts — payments, repositories, email, DNS — then one breach at the platform is a breach of every customer, and a lapsed subscription can strand assets the customer believed were theirs. Delegated access to accounts the user already owns has a strictly smaller blast radius and no hostage problem.

Are these problems solved by better models?

No. Almost none of them are model failures. A model with a lower error rate still cannot make a charge idempotent, cannot decide who is liable for an unreviewed action, and cannot fix an oversight ratio. They are architecture and accounting problems, and they are decided long before inference.

More writing