Chithhi LM uses Jev for semantic routing to cut latency
Chithhi LM uses Jev for millisecond semantic classification to route queries to BM25, vector search, or multi-hop hybrid RAG, cutting time to first token by 43%.
Over the last 3 days, I completely re-engineered Chithhi LM. I tackled the entire stack: from semantic routing and BM25 math to Bun, BullMQ, Gzip tuning, and 60fps streaming. The numbers: ⚡ TTFT: 7s ➔ 4s (-43%) ⚡ Total Retrieval & Response: 15–17s ➔ 5–15s ⚡ Server CPU load: -80% Here’s the full technical breakdown 🧵👇 1/ Adaptive Semantic Routing with @typesafe_ai (Jev) Blindly generating dense embeddings for every user query kills latency. We added Jev for instant single-digit ms semantic classification: • Keywords/error codes ➔ BM25 • Concepts ➔ Dense Qdrant Vector • Multi-hop ➔ Parallel Hybrid RAG 2/ Parallel Hybrid Search + CRAG • Fired Qdrant vector search and BM25 concurrently using Promise.all (zero waterfall delay). • Added Corrective RAG (CRAG) with an LLM context-grader that filters out irrelevant chunks before passing them to the generator, slashing hallucinations. 3/ Inverted Index BM25 (Zero Linear Scans) In-memory BM25 usually scans every chunk in O(N). Rebuilt it with an Inverted Index + Float64Array doc lengths: ✅ Scores ONLY candidate docs with matching terms ✅ 0% accuracy drop (formula math is identical) ✅ Eliminates thousands of Map allocations 4/ MMR Diversity Re-ranker Optimization MMR prevents duplicate context in prompts. Previously, string tokenization inside the MMR while-loop was O(N × K). • Hoisted chunk tokenization into precomputed Sets (O(N)) • Swapped regexes for Set intersection over the smaller set: O(min(|A|, |B|)) Jaccard scoring. 5/ Bun Runtime & Decoupled Workers • Migrated core Express API to @bunjavascript for instant cold starts & maximum I/O. • Decoupled heavy PDF/DOCX parsing into a standalone @BullMQ worker (Node 22 + Redis). • Docker Compose stack with Traefik TLS + Caddy reverse proxy. 6/ Worker Ingestion Optimizations Heavy file uploads were eating CPU cycles: • LangChain TextSplitters promoted to module singletons (zero regex recompilation per doc). • Hoisted parent chunk metadata extraction outside the inner child-chunk loop, eliminating repeated Mongoose .toObject() clones. 7/ HTTP Transport & Gzip Tuning Compression was silently eating backend CPU. • Tuned compression to Level 1 (fastest gzip, ~4x lower CPU than Level 6). • Added 1024-byte threshold to skip tiny payloads. • Enforced strict bypass filter on SSE streaming routes for 0ms token buffering. 8/ Database & Memory Layer • Mongoose queries converted to .lean() with field projections (no Mongoose document wrapping overhead). • In-memory BM25 TTL caching per tenant/source selection. • Added Neo4j Graph Memory + rolling chat summaries for multi-turn conversational context. 9/ 60fps SSE Token Batching on the Client High-frequency LLM tokens streaming over SSE were causing React layout thrashing. • Wrapped token updates in requestAnimationFrame (RAF) in Zustand. • UI updates sync with the monitor's 60fps refresh rate (~16ms). • Zero dropped tokens, silky-smooth rendering. Check out the full architecture flow diagram attached! GitHub: http://github.com/Yasho321/NotebookLM-Clone