How We Cut RAG Latency in Half for Enterprise-Scale Conversational Systems
Using model racing to accelerate query rewriting and keep agents real-time.
Get full access
RAG improves accuracy by grounding responses in large knowledge bases, but every millisecond it adds is a pause the caller hears. Query rewriting alone accounted for more than 80% of our RAG latency.
This piece explains how we removed that bottleneck:
- Why query rewriting is necessary for multi-turn conversations, and why it was slowing retrieval down
- How model racing sends each query to several models at once and takes the first valid response
- The role of self-hosted open models and a one-second fallback in keeping latency predictable
The result: median RAG latency down from 326ms to 155ms, with p75 and p95 improving alongside. For technical leaders evaluating how retrieval affects agent response time at scale.
