Hamel tests Jev and other decision models for evals

Antaripa Saha @doesdatmaksenseOriginally posted 2026.10.21

Open original on X

can we trust decision models for evals? @HamelHusain and I tested top decision models like Jev, d1-ultra, clef, pplx-decider, openai decisions, etc. on our dataset we explored failure detection, probability calibration, threshold optimization, and the practical tradeoffs of using them for evals. give it a read: https://hamel.dev/blog/posts/decision-model-evals/