Developer with Jev access plans to replace LLM-as-a-judge with thresholded probabilities
The author plans to use Jev for binary evals by feeding it state plus questions and thresholding the returned probabilities, and as a control plane in agent harnesses for routing, retries and review decisions.
got access to Jev!! heard a lot of good things about it, especially as a classifier. really curious to try it for evals. lots of llm-as-a-judge for binary decisions are going to get replaces. give Jev some state + a bunch of questions and get back probabilities you can actually threshold on. also feels pretty useful inside agent harnesses as a control plane for making all those fuzzy decisions like which model or tool to route to, whether to continue or retry, whether an output looks good enough, whether something needs human review, etc. basically letting the llm handle generation, while Jev handles a lot of the judgment calls around it.