Skepticism over Jev's 'hallucination-free' claim and its benchmark
The author credits Jev's speed and cost but notes it is only a classifier limited to preset labels that can still pick wrong, and that the RLCD method and benchmark are unpublished.
Jev is definitely interesting. But there are still a few things I’d want to see before buying the bigger claims around it. TypeSafe loudly calls Jev “hallucination-free.” That’s mostly marketing. A model that isn’t generating free text simply cannot be compared with one that can. Jev is only allowed to choose from answers you define in advance. It can’t invent an invalid output - but it can still confidently choose the wrong one. TypeSafe’s RLCD method hasn’t been published, and neither have the calibration curves. That’s what I’d really like to see. The speed/cost numbers are impressive too, but notice the 193.6× faster / 444.6× cheaper headline is the best case - not the average. The early benchmark is interesting: Across 711 cases, Jev scored 67.8% - roughly in line with GPT-5.6 Terra at 67.9%, and below Sol and Opus 5 at 74.1% and 73.1%. At ~$0.0004 per case and ~0.4s, that’s impressive. But the benchmark itself still isn’t public. It covers just four TypeSafe-designed workflows, and I’d especially like to see how the reference answers were established - ideally against human judgment, not just model agreement. Strip away the branding and Jev currently looks like something simpler: A large pretrained transformer, post-trained into an extremely fast probabilistic classifier over caller-defined labels. That could be very useful. Whether it’s a new class of model is a much bigger claim.