Real-time voice retrieval adds Jev-compatible Clef judge on a 400 ms budget

Mohamed Gaber @mhgaberOriginally posted 2026.10.26

Open original on X

Ever since Jev dropped, the timeline’s been full of decision-model talk. Most of it is APIs and benchmarks. We put one in the hardest place we could find: the hot path of a live voice call, on a 400 ms budget. The goal wasn’t to fix something broken. It was to take something that already works well and push it further. Baseline Knowledge-base retrieval in http://Agent.sa runs the industry default: BM25 + embeddings, RRF, and a freshness multiplier. No model calls in the hot path. It’s fast, and the right answer is already #1 89.3% of the time. A lot of teams would stop there. But a voice call has no results page. One answer gets spoken. So we went after the last mile. The ceiling Similarity-based retrieval has a known limit: It measures “which entry is closest?”, not “which entry contains the answer?” “What’s in the Gold plan” and “How much is the Gold plan” are almost the same words. Only one has the price. RRF ranks by position, not score. The top 10 often sit within 13% of each other, so a tie-breaker like freshness gets more weight than it deserves. There’s no calibrated “I don’t know”. The best heuristic (“if the top score is low, there’s no answer”) only reaches an AUROC of 0.74. The fix Let retrieval do what it’s good at. Put a judge on top. Retrieval returns the top 8 candidates. The judge reads each one and rules on it. Our judge is Cloudflare’s open-source Clef (Apache 2.0, Jev-API compatible). Because the weights are open, it runs on our own GPUs. The customer’s knowledge base never leaves our infrastructure. The question We ask a Score with four levels: 0 → different subject 1 → same topic, but doesn’t answer (the price, the condition, the date) 2 → partially answers 3 → directly answers Level 1 is the whole story. It gives the near-miss a name: right neighborhood, wrong house. Tip: describe each level in a full sentence, not just a number. The judgments get noticeably sharper. Reordering The reorder is deliberately boring and deterministic: 2 and 3 first → then 1 (and anything not judged in time) in original search order → then 0 at the bottom. Demote, never remove. Freshness is now just a tie-breaker. Answerable is tri-state: true / false / null. The false is what lets the agent say “let me check and get back to you” instead of reading out the closest thing it found. Hot-path rules we don’t bend One 400 ms deadline for all 8 judgments together. Late ones get dropped. No retries. If nothing comes back, we fall back to plain search order. A slow judge never stalls a call. Allowlist, not denylist. Only 4 fields go to the judge. Anything added later stays private until someone explicitly decides otherwise. off / shadow / live per agent. Shadow judges after the response is already sent and logs what it would have changed — production latency and accuracy with zero risk. A status on every query: disabled / shadow / applied / partial / timeout / error. A degraded judge should be visible, never silent. Eval (we don’t ship on vibes) Questions that sound like real callers: Egyptian Arabic, formal Arabic, English, mixed, broken voice-to-text, plus questions the KB genuinely can’t answer. Labels from a different model so the judge isn’t marking its own homework. Ablation: the same pipeline without the judge matched the baseline. The gain is all the judge. Paired + McNemar: the odds the gap is chance are about one in a million. Results Top-1: 89.3% → 97.5% (11 questions improved for every 1 that got worse) English questions on Arabic content: 81.7% → 98.3% Questions about a specific detail: 83.3% → 95% “I don’t know”: AUROC 0.74 → 0.99 Because the probabilities are calibrated, the threshold becomes a business decision, not a guess. At 0.5 it catches 96.4% of unanswerable questions while wrongly flagging only 3.3%. If a wrong “I don’t know” costs more, 0.2 drops that to 1.9%. The best part “I don’t know” became a backlog for the knowledge base. Every unanswered question goes to a background worker that: masks personal numbers re-judges anything that missed the deadline classifies it (off-topic, needs live data, unclear, or genuinely missing knowledge) looks deeper than the top 8 groups questions that ask for the same thing The trap The same question in Arabic and English scored 0.22–0.43 similarity. Two different questions on the same topic scored 0.72. Any similarity threshold works exactly backwards. So embeddings only build a shortlist. The decision model decides “is this the same question?”. It correctly separated Velents’ WhatsApp number from Velents Pay’s. The final knowledge-base suggestion is never allowed to invent a fact — not in the prompt, in the code. A knowledge base that makes up its own prices is worse than one with gaps. All of it runs on a dedicated thread with a bounded queue, not the framework’s background tasks (those share a pool with retrieval, and nobody wants gap analysis delaying a call). Takeaway Retrieval is great at finding candidates. A decision model is great at choosing between near-misses. Different jobs. Generation is a commodity. The decision inside the hot path is the real engineering.