An LLM in the incident loop. Without letting it touch prod
The dream everyone pitches is the self-healing system: AI detects the incident, diagnoses it, fixes it, and you read about it in the morning. After two years of building toward exactly this, here's where I actually landed. And it's more useful than the dream, because it's real. The model belongs in the incident loop. It does not belong with its hands on production.
The valuable thing a model does during an incident isn't acting. It's compressing. At 2am, the bottleneck is rarely knowing which command to run. It's the frantic gathering: which dashboard, which log, which of forty possible causes, what did the last person who saw this do. A model that has read your runbooks, your past incidents, and the live telemetry can collapse twenty minutes of context-gathering into thirty seconds of "here's what this looks like, here's what to check, here's the three most likely causes ranked." That's enormous. And it requires the model to touch nothing.
The shape that works
Read-only, always, on the diagnosis side. The model can query metrics, read logs, retrieve runbooks, correlate a trace. All read operations, none of which can make the incident worse. It grounds every claim in real, current data pulled at that moment, not in what it remembers being true. A confident guess is worthless in an incident; a cited observation from the actual system is gold, and the difference is entirely whether you grounded it.
The human stays on the trigger for anything that changes state. The model proposes the fix. Often correctly, often faster than I'd have gotten there. Then a person reads the proposal, understands why, and executes. This isn't distrust theater. It's that the cost of a wrong action during an incident is catastrophic and asymmetric: a good suggestion saves minutes, a bad auto-remediation turns one outage into two. You do not give unsupervised write access to production to something that is usually right. "Usually" is doing violent work in that sentence.
Everything it does is logged and reviewable. Every query, every retrieval, every suggestion. Partly for the post-mortem, partly because that's how you build justified trust over time. You watch it be right across a hundred incidents before you'd ever consider widening its latitude, and even then you widen it one careful notch at a time.
Why "less autonomous" turned out to be the better product
The fully-autonomous version is a great demo and a bad system, because it optimizes for the wrong metric. It optimizes for removing the human, when the actual goal is resolving the incident correctly and fast. A model that makes a great engineer twice as fast at diagnosis, while leaving the irreversible decisions with the human who's accountable for them, resolves incidents better than a model that acts alone and occasionally makes a confident, catastrophic mistake. The measure isn't how little the human does. It's how well the incident ends.
Two years ago I wanted to build the system that fixes itself. What I built instead is the system that makes the person fixing it dramatically better. Grounded, fast, and safe by construction. It's less flashy. It's the one I'd actually put in front of a production incident. And after everything, that's the whole lesson: the point was never to replace the judgment. It was to arm it.