Using Jev decision models as LLM judges for evals
The author recommends System One decision models like Jev as LLM judges and links two Jev-as-a-Judge guides plus a paper.
Learn to build evals, folks. Highly recommended reading on using decision models for evals. From our internal evals, System One models like Jev are a great fit for LLM Judges. If you want to go beyond Hamel's great write-up, I've also put together a few resources on applying decision models to LLM judges. (bookmark it) 1) Jev-as-a-Judge for Agent Evaluations - https://academy.dair.ai/dashboard/resources/jev-as-a-judge 2) Jev-as-a-Judge: Evaluating Agents with a Decision Model - https://academy.dair.ai/labs/jev-as-a-judge-for-agent-evals Bonus paper I would recommend, with a few other ideas that combine System One and System Two models: Jev-as-a-Judge paper - https://arxiv.org/abs/2609.26550
This post quotes another post: x.com/hamelhusain/status/2112588257104216294