jeval calibrates Jev confidence thresholds and adds CI regression checks
The author open-sourced jeval, which checks what a Jev confidence of 0.8 actually hits on your logs, computes the right threshold from misjudgment cost, and alerts in CI when a model update makes things worse.
Thinking about using Jev? Jev doesn't write text. It just makes judgments — "this ticket goes to billing" — and hands you a number for how sure it is. Fast and cheap. But as an engineer, the thing that actually matters isn't model performance. It's whether this fits into your service properly. Because eventually you have to write this line: if confidence > 0.8: auto_handle() Where did 0.8 come from? Usually copied from a docs example. Nobody ever measured what 0.8 actually means on your data. Too low and things break. Too high and you've automated nothing. And you can't tell which way you're wrong. That's why I'm building jeval. Feed it your logs and it tells you "0.8 was actually 77%." Give it your cost of a wrong call, and it computes where the line should be. When a model update quietly makes things worse, your CI catches it. Not Jev-only. Works with any classifier that returns confidence. GitHub: https://github.com/rlaope/jeval #jev #ai #opensource