Testing Jev's calibration: miscalibrated on 7 of 8 datasets
Across 16,500 predictions on 8 real datasets covering loan defaults, bankruptcies, and medical diagnoses, Jev failed calibration on 7 of 8 and systematically overstated the probability of bad outcomes by roughly two times.
@fakegingerbitch asked and I checked. Here is what I quickly found: TypeSafe sells its AI model Jev on one promise: when it says "70% likely", it's right about 70% of the time. I tested that on 8 real datasets — loan defaults, bankruptcies, medical diagnoses, marketing responses. 16,500 predictions. It fails on 7 of 8. Jev is 10× more miscalibrated than CatBoost, a standard model anyone can download for free. On average it's no more useful than guessing the overall rate for every single case. And it fails in one direction: it thinks bad things are about twice as likely as they really are. Loan defaults, bankruptcies, disease — all overstated, on every dataset. German loan data: applicants flagged "critical account" repay 83% of the time. Jev says 26%. The fix takes 120 rows of data and a textbook method from 2017. It removes three quarters of the error. A model that was actually calibrated would gain nothing from that. Calibration is something you measure. Not something you announce.
This post quotes another post: x.com/fakegingerbitch/status/2101118764561453436