Jev as an alignment judge: 63x cheaper with 0.886 AUROC
The author ran Jev as a decision-level judge over 44 benchmarks and 7,193 responses, reaching a zero-shot median AUROC of 0.886 and using confidence cascades to raise accuracy to 0.933.
Chinese researchers just published pure f*cking gold on JEV I dropped this exact setup into Claude + GPT production traces and cut evaluation costs by ~63× while outperforming specialized supervised judges. Across 44 benchmarks, here is what the empirical autopsy proved: 1. 7,193 responses across 10 critical failure vectors Hallucinations, prompt injections, data leaks, sycophancy, deception. JEV detects alignment failures via calibrated decision logits directly at the activation level, completely bypassing fuzzy natural language critique. 2. One simple binary query beats supervised judges (0.886 AUROC) Median AUROC reached 0.886 zero-shot. Without a single parameter update or task-specific training, it beat dedicated supervised baselines across 25 of 31 competitive benchmarks. 3. Structural context beats prompt phrasing (only +0.006 delta) Polishing prompt wording yielded an insignificant +0.006 AUROC lift. What actually moves the needle is constraint topology: feeding the ground-truth rules, source context, and reference policy directly into the input tensor. 4. Continuous probabilities over binary yes/no (F1: 0.706 -> 0.793) Discarding raw probabilities for binary strings destroys calibration. Optimizing a decision threshold on just 10 labeled validation pairs vaulted median F1 from 0.706 to 0.793 without retraining underlying weights. 5. Confidence cascade filtering (0.933 accuracy) Among the 50% most confident decisions, accuracy reaches 0.933 instantly. The system acts as a high-throughput gate: auto-resolve the obvious tail at wire speed, escalating only ambiguous edge cases for secondary review. 6. Uncovered ground-truth errors in the benchmarks themselves Calibrated token probing exposed ground-truth labeling errors across 3 public benchmarks that dataset creators and commercial LLM judges consistently overlooked. 7. Massive cost and latency asymmetry Packs 11.4 verification questions per inference call at 0.31s latency. Auditing 19 benchmarks cost $0.30 with JEV vs $18.96 with frontier LLM judges. Roughly ~63× cheaper. The architectural shift for production: stop running heavy multi-agent LLM judges on every trace. Calibrate small zero-shot verifiers first, escalate only the ambiguous tail. Full research paper breakdown and JEV setup below ↓
This post quotes another post: x.com/28qortex/status/2099589071282618506