CMU paper tests Jev as an LLM judge
On focused judgments like preferences, factuality, and answer checks, Jev stays within three percentage points of the strongest judge at 0.36% of the cost, and low-confidence cases can be escalated to a stronger model.
Brilliant paper from CMU. If you use Jev/LLM-as-judge, this one is for you. (bookmark it) We use LLMs to generate answers, then use another LLM to judge them. But many evaluations only need a bounded decision. - Was the answer grounded? - Did it follow the instruction? - Which response was better? This Carnegie Mellon University paper tests whether Jev can make these decisions cheaply, then identifies the cases that need a stronger judge. The results reveal Jev’s real edge. → Jev works well on focused judgments On ordinary response preferences, evidence-grounded factuality, and final-answer checks, Jev remained within three percentage points of the strongest judge tested. → The cost difference is substantial On the paper’s matched workload, Jev cost only 0.36 percent of the strongest comparator’s fee. This makes it practical to evaluate more agent responses, across more criteria, without scaling the evaluation bill at the same rate. → Confidence becomes a routing mechanism Jev returns a decision and label probabilities. When confidence is high, accept its decision. When confidence is low, escalate the case to a stronger LLM. A frozen version of this cascade retained about 99 percent of GPT-6’s accuracy while using roughly 57 percent of its fee. → Jev still has a clear boundary It fell behind when evaluation required checking complex derivations or resisting an elaborately written wrong answer. It also struggled with factuality judgments when no reference evidence was provided. In those cases, confidence did not reliably expose the errors. So the paper is not arguing that Jev should replace every LLM judge. It presents a more useful architecture. Code handles exact rules. Jev handles focused semantic judgments. Stronger models or humans handle uncertain and reasoning-heavy cases. That is also the idea behind the workflow I built. The agent generates an answer. Jev judges bounded claims against the supplied evidence. Uncertain cases move to review. Opik records the traces, scores, and experiment results. The paper explains why this architecture works. My article shows how to build it. The full implementation breakdown is quoted below.
This post quotes another post: x.com/akshay_pachaar/status/2102087107410002345