RAG is a retrieval problem wearing a language-model costume
The first retrieval-augmented system I built underperformed, and I spent a week blaming the model. Better prompts, bigger model, more careful instructions. Small gains, nothing that fixed it. Then I actually read what the retrieval step was handing the model, and the problem was obvious: the model was answering perfectly. It was just answering from the wrong three paragraphs.
That's the thing nobody tells you when RAG is pitched as "just give the model your documents." The generation is rarely the bottleneck. Modern models are very good at synthesizing an answer from context. The hard, unglamorous, determines-everything part is getting the right context in front of them. And that's an information-retrieval problem that predates language models by decades.
Where the quality actually lives
I stopped tuning the model and started treating retrieval as the product. That shift is where every real gain came from:
Chunking is a design decision, not a default. How you split documents determines what can ever be retrieved together. Split too small and you sever the context a passage needs to make sense; too large and you drown the signal in noise and blow your token budget. There's no universal right answer. It depends on the shape of your documents and the shape of your questions. But there's definitely a wrong answer, which is "whatever the tutorial used."
Embeddings encode similarity, not relevance. And those aren't the same thing. Two chunks can be semantically close and one of them still be useless for the actual question. Pure vector search gets you into the neighborhood; it doesn't get you the right house. The systems that worked combined vector search with keyword/lexical matching and a re-ranking pass, because the failure modes of each cover for the other.
The most valuable thing you can log is what got retrieved. When an answer is wrong, "what did the model see" is the entire diagnosis. Nine times out of ten the model reasoned correctly over bad inputs. If you're not logging the retrieved context, you're debugging blind. Staring at the answer instead of the cause.
The reframe that fixed it
Once I started treating the retriever as the thing under test. Measuring whether the right passage made it into context, before the model ever ran. The whole system got better fast. Grounding an answer in real data is worth enormous amounts: it's the difference between a model that confidently makes things up and one that confidently cites what's actually true. But grounding is only as good as what you retrieve. Garbage context, confident garbage answer. The model faithfully amplifies whatever you feed it, which is exactly why the feeding is the job.