CMU tests Jev as a decision-only judge with confidence routing
Jev takes the evaluation state and allowed outputs and returns a typed decision with label probabilities; at a 0.9 confidence threshold, escalating only 34% of judgments kept 99.6% of GPT-6's accuracy at 47% of its cost.
JEV as a judge! Researchers at Carnegie Mellon University just tested how far JEV can go as a decision-only judge, and where a stronger LLM still needs to take over. The setup is simple. JEV takes the evaluation state, the judging instructions, and the allowed outputs, then returns a typed decision with probabilities for each label instead of generating a long rationale first. The paper compares it against generative LLM judges and reward models across preference, factuality, and answer-adjudication tasks. On ordinary preference, evidence-grounded factuality, and final-answer adjudication, JEV stayed within roughly 3 percentage points of the strongest judge tested. The cost difference was much larger. On the measured panel, JEV came in at about $0.04 per 1,000 judgments with roughly 0.15s median latency. The more useful result is what happens when you use JEV's confidence as a routing signal. When JEV is confident, you keep the decision. When it is unsure, you escalate that case to a stronger LLM judge. With a 0.9 confidence threshold, the pooled cascade escalated about 34% of the judgments and retained 99.6% of GPT-6's accuracy at 47% of its fee. That gives you a much more practical evaluation pipeline: Input → JEV → Confidence check → Accept or escalate → Stronger LLM The paper also shows where this stops working well. JEV falls further behind on difficult reasoning, coding, and style-adversarial cases where an incorrect answer is written convincingly. So the confidence score is useful for routing, but it should not be treated as proof that the verdict is correct. I like this because it turns LLM-as-a-judge from a single-model choice into a systems problem. You do not need the strongest judge for every evaluation. You need a cheap first pass that knows when it should hand the difficult cases over to something stronger. I've shared the link to the paper in the comments.
This post quotes another post: x.com/sumanth_077/status/2101639788961112279