My first agent did the wrong thing, confidently
The first autonomous agent I put in front of a real workflow worked beautifully in the demo and then, the first week in production, cheerfully took an action nobody asked for. It didn't crash. It didn't error. It reasoned its way to a decision that was locally sensible and globally wrong, and it did it with the calm confidence of something that has no idea it can be wrong.
That week taught me the thing that reframed how I build with these models: the failure mode isn't that the agent breaks. It's that it succeeds at the wrong objective. A crash you catch in seconds. A confident wrong action you catch after it's already happened. If you're lucky enough to be watching.
Capable is not the same as trustworthy
Early on I was optimizing for capability: give it more tools, more autonomy, more room to figure things out. And it did figure things out. Which is exactly the problem. The more latitude you hand an agent, the larger the space of surprising-but-technically-valid actions it can take. Capability and blast radius grow together. I'd been treating them as separate dials when they're really one.
So I flipped the question. Instead of "what can this agent do," I started asking "what's the worst thing this agent can do before a human sees it". And designing backward from that answer. That single reframe is the difference between a demo and something you'll let near a real system.
The three questions I now ask before any agent ships
What can it touch? Not what it's supposed to touch. What it can, if it improvises. Every tool you hand an agent is a verb it can conjugate in ways you didn't script. The tool surface is the blast radius.
What's reversible? I sort every action into reversible and not. Reversible actions can run on the agent's own judgment. Irreversible ones. Anything that sends, deletes, spends, or changes production state. Go behind a gate a human clears. Not because the model is dumb, but because "usually right" is a terrifying property for an irreversible action.
Who finds out, and when? An agent that acts silently is one you learn to distrust after the first surprise. One that narrates what it's about to do, and waits at the edges that matter, is one you can actually build on.
None of this is about the model being weak. The best model in the world still shouldn't have an unsupervised path to your production database. That's not a statement about AI. It's the same principle we've always applied to any powerful automated system. The model just makes the principle urgent, because it's the first automation that improvises.