The boundary is the product

You’ve probably seen the headline this week: an AI agent deleted our production database. It bankrupted its operator. There’s a whole run of these right now — agents gone wrong, the hit pieces that follow.

I build an AI chief-of-staff for a living. So instead of piling on, I want to say plainly why the agents I build can’t do the thing in that headline — and what I think the real lesson is.

What mine can’t do

The agent I build drafts outbound actions and waits for a human yes. That’s the first rule and it’s not negotiable. It never posts on its own. It never spends past a cap. And it never acts on the untrusted text it reads — the emails, the pages, the documents it processes are input, not instructions.

Those three constraints aren’t a settings panel I bolted on at the end. They’re the shape of the thing. Draft, never dispatch. Approval before anything leaves. A cap it can’t exceed.

Why “don’t use agents” is the wrong lesson

It’s tempting to read this week’s stories as a warning against agents entirely. I read them differently. In each case the failure wasn’t that the agent was too dumb. It was that it was allowed to act, alone, all the way to a destructive outcome, with no stop in the path.

That’s not an intelligence problem. It’s a boundary problem.

The interesting engineering in an agent isn’t making it more capable. Capability is cheap now. The hard, valuable part is deciding exactly what it may do unsupervised, and where it must pause and hand the decision back to a person. Get that wrong and more capability just means it breaks things faster.

So when I design mine, the boundary isn’t a constraint on the product. The boundary is the product. The list of what it will never do on its own — post, spend past a cap, act on untrusted text — is the actual value. Everything else is a helpful assistant looking for a leash.

How I decided to build mine

This is how I made the call, in my own voice, for my own work: if an agent can’t be trusted to stop, it doesn’t ship. It drafts, it waits, it stays inside a cap, and it treats the text it reads as something to reason about — never as orders.

That’s less exciting than a demo of an agent doing everything by itself. It’s also the reason mine can’t end up as this week’s headline.

The agents aren’t the risk. Autonomy without a stop is. Build the boundary first.