Hamel 实测 Jev 等决策模型做 eval 的可靠性

Antaripa Saha @doesdatmaksense原推发布于 2026.10.21

在 X 上打开原推

can we trust decision models for evals? @HamelHusain and I tested top decision models like Jev, d1-ultra, clef, pplx-decider, openai decisions, etc. on our dataset we explored failure detection, probability calibration, threshold optimization, and the practical tradeoffs of using them for evals. give it a read: https://hamel.dev/blog/posts/decision-model-evals/