Agentic RAG creates an unavoidable trade-off: more reasoning can improve evidence quality, but it can also add latency.
The engineering problem is not to choose speed or accuracy once. It is to decide which path a query deserves.
Trade-off table
| Decision | Latency impact | Accuracy impact |
|---|---|---|
| Direct retrieval | Lower | Weaker for complex questions |
| Decomposition | Higher | Stronger for multi-part questions |
| Hybrid retrieval | Medium | Stronger for mixed semantic and lexical needs |
| Caching | Lower when safe | Neutral or risky if context is wrong |
| Clarification | Higher interaction cost | Safer when evidence is missing |
Engineering rule
Low latency is a first-class constraint, but not permission to skip grounding.
What a good system should do
A good Agentic RAG runtime should avoid expensive work when the query is simple. It should spend more effort when the query is complex. It should reuse intermediate work when context allows it. And it should stop when evidence is missing.
That is the practical target: fast enough to be usable, grounded enough to be trusted, and explicit enough to debug.