Replacing RAG routing and reranking with TypeSafe Jev cuts 9 seconds of time to first token

Yashovardhan Singh @Yasho_27_Originally posted 2026.10.02

Open original on X

I made the RAG pipeline very robust to increase accuracy, but that left me with a problem of very high latency. Yesterday I worked on optimizing it and cut down latency as much as possible. It can be reduced even further, and I am still exploring how! I cut 9 FULL SECONDS of pre-stream silence from my ChithhiLM (from 16s down to 7s TTFT) without losing a single drop of accuracy. Here’s the complete breakdown: 1/ The Problem End-to-end query turnaround was ~24s. Users waited 16 seconds in total silence before the first word started streaming. Why? Compounding sequential LLM round-trips over HTTPS. I was asking heavy models to write JSON for internal intermediate decisions, and calling them one after another. 2/ Fix 1: Parallelizing Independent LLM & Retrieval Calls In the old flow, Adaptive Routing ran first sequentially ➔ then Query Translation ➔ then Channel Routing ➔ then Memory Fetch. We refactored the entire pre-search phase into a concurrent Promise.all() fan-out: • Strategy Routing (TypeSafe Jev) • Query Translation & 15-word HyDE (gpt-4.1-nano) • Channel Selection (TypeSafe Jev) • Intent-Gated Memory Fetch All 4 operations now execute in parallel! Instead of 4 serial network round-trips taking ~14 seconds, the entire pre-search phase resolves concurrently in ~2.7 seconds. 3/ Fix 2: TypeSafe AI System One (jev-1.13.0) Instead of asking a generative LLM to guess an integer score (0–10) in a JSON string: I migrated Strategy Routing, Channel Selection, and Reranking to TypeSafe mathematical primitives (choice, score, noul). • Reranking: 4,500ms ➔ 684ms (85% drop!) • Context Grading: 3,500ms ➔ 0ms on the happy path (fast-path skip), or ~400ms when full evaluation runs • Zero JSON parsing failures. 4/ Fix 3: The 150-Word HyDE Tax I was asking gpt-4.1-mini to write a 150-word hypothetical essay on EVERY query. Generating that essay took ~7.8s, and embedding that huge text slowed down Qdrant to ~6.8s. Solution: • Switched to gpt-4.1-nano • Replaced the 150-word essay with a 15-word focused search key. Translation dropped to 3.0s (saved 4.8s), search dropped to 3.5s! 5/ Fix 4: Gated Episodic Memory Why query Neo4j graphs & embed Qdrant episodic memory if the user is asking "What does page 4 say?" Added a 50ms TypeSafe intent check. Non-personal queries bypass graph & vector memory completely. Memory retrieval is now a true ~550ms. 6/ Fix 5: DB Compound Indexes • MongoDB: Added { userId: 1, sourceId: 1, level: 1 } for 100% index-covered BM25 fetches, plus parent expansion indexes (expansion dropped from 186ms to 71ms). • Neo4j: Added uniqueness constraints on (u:User {id}). • Qdrant: Added payload indexes for multi-tenant isolation. 7/ The Scorecard: ⚡ Concurrent Pre-Search: ~14.0s ➔ ~2.7s ⚡ Translation: 7.8s ➔ 3.0s ⚡ Search (Vector + BM25): 6.8s ➔ 3.5s ⚡ Rerank (TypeSafe): 4.5s ➔ 0.68s ⚡ Pre-Stream Silence (TTFT): 16.0s ➔ 7.2s ⚡ Total Completion: ~24s ➔ ~15.8s Retrieval feels significantly faster now. It’s still not close to the absolute sub-second UX out there, but shaving 9 seconds of dead silence makes a massive difference. Before, a few complex queries were even taking 60+ seconds, now everything is predictable and bounded. Will keep optimizing and share updates soon!