CMU 把 Jev 当决策型 judge 并做置信度路由

Sumanth @Sumanth_077原推发布于 2026.09.27

在 X 上打开原推

JEV as a judge! Researchers at Carnegie Mellon University just tested how far JEV can go as a decision-only judge, and where a stronger LLM still needs to take over. The setup is simple. JEV takes the evaluation state, the judging instructions, and the allowed outputs, then returns a typed decision with probabilities for each label instead of generating a long rationale first. The paper compares it against generative LLM judges and reward models across preference, factuality, and answer-adjudication tasks. On ordinary preference, evidence-grounded factuality, and final-answer adjudication, JEV stayed within roughly 3 percentage points of the strongest judge tested. The cost difference was much larger. On the measured panel, JEV came in at about $0.04 per 1,000 judgments with roughly 0.15s median latency. The more useful result is what happens when you use JEV's confidence as a routing signal. When JEV is confident, you keep the decision. When it is unsure, you escalate that case to a stronger LLM judge. With a 0.9 confidence threshold, the pooled cascade escalated about 34% of the judgments and retained 99.6% of GPT-6's accuracy at 47% of its fee. That gives you a much more practical evaluation pipeline: Input → JEV → Confidence check → Accept or escalate → Stronger LLM The paper also shows where this stops working well. JEV falls further behind on difficult reasoning, coding, and style-adversarial cases where an incorrect answer is written convincingly. So the confidence score is useful for routing, but it should not be treated as proof that the verdict is correct. I like this because it turns LLM-as-a-judge from a single-model choice into a systems problem. You do not need the strongest judge for every evaluation. You need a cheap first pass that knows when it should hand the difficult cases over to something stronger. I've shared the link to the paper in the comments.

这条是引用着另一条写的：x.com/sumanth_077/status/2101639788961112279