The RAG that never phones home

Here's the workload the Spark was really for: retrieval-augmented generation over data that is too sensitive to ever upload. A model that can answer questions grounded in internal, confidential material. Without a single byte of that material, or the questions asked about it, ever crossing a network boundary.

Everything I've written about RAG still applies. Retrieval is the hard part, not generation. But running the whole pipeline locally adds a property you cannot buy from a hosted API at any price: the data never leaves. Not the documents, not the embeddings, not the queries, not the answers. For a whole class of regulated work, that's the difference between "we can't use AI here" and "we can."

The whole pipeline, on one box

The embedding model that turns your documents into vectors: local. The vector store those embeddings live in: local. The retrieval step that finds the relevant passages: local. The model that reasons over them to produce an answer: local. When every stage runs on hardware you control, there's no step where sensitive content is handed to a third party. Because there's no third party. The unified memory makes this practical: the embedding model, the working set, and the generation model can coexist without a frantic memory dance.

Why this matters more than it sounds

The usual objection to RAG in a sensitive environment isn't about quality. It's about exposure. "We'd love to let the team ask questions against our internal runbooks and incident history, but we can't send that to an external model." Local inference dissolves that objection entirely. The knowledge that was locked away because it was too sensitive to expose becomes queryable, by a model, safely, because the model comes to the data instead of the data going to the model.

And it composes with everything else I've been building. A read-only triage agent that grounds its reasoning in confidential internal context, runs on local silicon, and gates every real-world action behind a human. That's an agent you can point at genuinely sensitive operations without the exposure that would normally make it a non-starter. Safe by construction, private by construction, grounded in real data. That's the whole stack coming together.