JevBench creator: benchmarks beat vibes for model selection
The JevBench creator rejects choosing System One models by vibe and trust, and describes how version changes, public and private test pools, and benchmaxxing detection prevent score gaming.
I'm the creator of JevBench. I admire @CompleteSkeptic and what he has given the world with Jev. But to claim that "vibes and trust" are the best basis for deciding which System One model to choose sounds like a bad joke - or like a way of coping with the fact that there's no moat in the field of model development. Yes, benchmarks aren't perfect, but they're the best objective, scientific proxy we have for how good a model is. With JevBench, I go to great lengths to ensure that the scores can't be "benchmaxxed." (as an aside: my benchmark site http://benchmarkheaven.com has the worlds first benchmaxxing score, so it's a topic I deeply care about.) For those who are interested, here's the explanation how JevBench ensures its results can't be gamed: Every version surprises. From one release to the next we change the metrics and the composition of the test set. Model builders can't train on "our style" of questions, because the style keeps moving. Public set + large private set. The public items let anyone see how carefully and how diversely our questions are built. But most of the score comes from a much larger private pool that model creators never see, and every release brings a fresh sealed set. Even models we call through an API have nothing to train on. A built-in benchmaxxing detector. We compare each model's score on the public items with its score on the sealed ones. If a model does statistically clearly better on the questions it could have seen, it gets a substantial penalty. As far as I know, almost no other benchmark does this, and it just works. There is no single "best" model. A System One model is only as valuable as its mix of intelligence, speed and cost. Early versions just ranked a simple equal-weight composite. But people need different trade-offs, so on the site you can now set how much each axis matters and get your own ranking. And I introduced lots of advanced analysis views to allow looking at the data in detail. For users: choose for your task, not the leaderboard. Our radar charts show how each model does across question types and topics, and on easy vs. hard questions. If your questions are simple, a tiny model on a CPU may be perfect. If you need maximum intelligence, Jev may be the right pick, or if speed and cost isn't so important, maybe even GPT-6 Luna. And if you act on confidence thresholds (auto-approve above 0.9), calibration matters most: open models like Plumb-4B and JevK5 v0.3 score 63.1 on calibration vs. Jev's 54.7, at about half the price per decision. Regarding evals on your own data: This is the one point I fully agree with Diego - in the end, evaluating on your exact use case beats any leaderboard, and so does fine-tuning on exactly those tasks. If you want an evaluation built on your own dataset, reach out. I offer it as a service, and I'm working in stealth on something in exactly this direction. Happy to talk with anyone interested. http://benchmarkheaven.com/jev-models