CMU paper: JEV-as-a-Judge routes reviews by confidence
CMU proposes using JEV's calibrated probabilities as an LLM judge that accepts confident verdicts and escalates uncertain ones to GPT-6, saving 59% compute while being more accurate.
Holy sh*t, researchers at CMU just solved the cost-latency tradeoff of LLM-as-a-Judge, and the architecture is remarkably elegant Paper: "JEV-as-a-Judge: Accept When Confident, Escalate When Unsure" (2609.26550). The fundamental insight: LLM evaluation divides into two regimes: 1. Text-grounded verdicts (where the answer can be read off the prompt/context). 2. Derived reasoning (math proofs, multi-file code logic, deep symbolic deduction). Large generative reasoning judges (GPT-6 Astra, Claude Opus 5.5) are overkill for Regime 1, while lightweight reward models fail on Regime 2. JEV uses probability calibration to detect the exact boundary between the two. How JEV-as-a-Judge works under the hood: 1. Non-Generative Decision Output: Instead of autoregressively generating 500 reasoning tokens, JEV emits calibrated softmax probabilities over discrete decision labels. Median latency drops to 0.15s at 0.36% the cost of GPT-6. 2. Fixed Threshold Routing: With a frozen confidence threshold (tau): → Sharp branches (p >= tau): confident verdict accepted without human or frontier model intervention. → Split branches (p < tau): hard ambiguous cases escalated to GPT-6 Astra. 3. Empirical Validation (16 Generative & Reward Models + Blinded Human Adjudication): → 1,610 Held-out Pairs: The JEV -> GPT-6 cascade scored 0.9 points higher than GPT-6 alone, while reducing total compute spend by 59%. → Live Generalization: In live tests across 2 fresh production workloads, the cascade matched frontier model ground truth with zero accuracy loss. → Limitations Identified: Confidence calibration degrades on style-adversarial inputs and reference-free creative prose; authors provide a local threshold validation recipe to tune tau. Why this matters for agent engineering: This is the exact reason modern agent trees put JEV at the fork layer (which file, which tool, retry or stop). The fast model absorbs 70%+ of the traffic in 150ms, and the frontier thinker only touches the genuine splits. Read on micro. Reason on high. ↳ http://alphaxiv.org/abs/2609.26550
This post quotes another post: x.com/28qortex/status/2103949670435492020