Tuning local models on the Spark: what actually moved the needle

Getting a model to run locally is an afternoon. Getting it to run well for a real job is where the weeks go. A month into living with the Spark, here's what actually changed the numbers. And what turned out to be a distraction.

Quantization is the biggest lever, and it's not close

The single highest-impact decision is how aggressively you quantize. Dropping from full precision to a good 4-bit quant is the difference between a model that barely fits and thrashes, and one that sits comfortably in the unified memory pool with room to breathe. Often with quality loss that's genuinely hard to notice on the tasks I care about. The instinct is to hoard precision "just in case." The reality is that a slightly-quantized model you can actually run at speed beats a pristine one that's swapping. Fit the model to the memory, not the other way around.

Context length is a memory budget, so spend it on purpose

Every token of context you allow is memory you're committing up front. It's tempting to crank the context window to the maximum because bigger sounds better, but on a fixed memory budget that's a direct trade against model size and headroom. The move that worked was matching context to the actual job: a triage assistant that reasons over a focused slice of telemetry doesn't need a novel's worth of window, and giving it one just steals memory from the model itself. Right-size the context to the task and you get a bigger, better model in the same box.

Throughput is about keeping the model resident

The thing that wrecks local performance isn't slow compute. It's the model falling out of memory and having to reload. On a unified-memory box the win is keeping your working model hot: load it once, keep it resident, and let the requests come to a warm model instead of paying the load cost over and over. Once I stopped treating each request as a fresh start and started treating the loaded model as a long-lived service, the effective throughput changed character entirely.

What turned out to be a distraction

Chasing the newest, biggest model every week. The gains from picking a solid model and tuning how you run it. Quantization, context, keeping it resident, a decent prompt. Dwarfed the gains from swapping to whatever dropped on the leaderboard that Tuesday. The model was rarely the bottleneck. How I ran it was. That's a lesson I keep relearning in every corner of this work: the engineering around the model is where the performance actually lives.