开发者实测 Jev 概率校准：8 数据集 7 个失败

Han @HanchungLee原推发布于 2026.09.21

在 X 上打开原推

Jev looks interesting, but its differentiator, probability caliberation, is underwhelming. Structured outputs, latency, and cost can be replicated w/ off-the-shelf models. And I find it's biggest Aside from the 'System One' branding, Jev has 4 main promises: caliberated probabilities, structured outputs, low latency, and low cost. Cost and latency. Smaller LLM have similar cost and latency profile vs Jev. @ekzhang1 built a Jev-compatible API w/ Qwen-3.6-35b-a3b and SGLang doing 64 decisions/sec. https://x.com/ekzhang1/status/2100651678110515383. @mmastrac built it w/ DiffusionGemma and VLLM with similar latency vs Jev on DGX Spark. https://x.com/mmastrac/status/2100626204554592508?s=20. So, fast decision APIs are reproducible with existing models. Structured outputs. This is a capability trained into all autoregresive language models since 2023. Jev provides a similar abstractions to @deliprao 's AutoRubrics https://x.com/deliprao/status/2101732243848732895?s=20. Perhaps I am not applied or product minded enough, or i've been thinking too much about evaluations and alignments, the most useful application for Jev off the top of my head is as an LLM or search rubrics / preference model with its calibrated probabilities. But I am skeptical about its broad calibration claims, which it calls RLCD (calibrated decisions). What does it calibrate to? Calibrations is highly dependent on the known probability of the distribution q. @predict_addict ran 16.5k predictions across 8 datasets and found caliberation failures on 7/8. https://x.com/predict_addict/status/2101404670488875042. My own experiments (coin toss, 2 dice, 3 UCI datasets) shows that Jev has high expected caliberation errors (ECE). This is especially pronounced once we venture out of simulation based experiments. There is real product value here. @pdrmnvd runs poker arena https://x.com/pdrmnvd/status/2101470195428827493?s=20. @ShengyaoZhuang runs reranking with it https://x.com/ShengyaoZhuang/status/2101212268440723895?s=20. @BrainsAndTennis hooked Jev up to play Warcraft 3 https://x.com/BrainsAndTennis/status/2100998807903482364?s=20. And look at that damn Mountain King randomly jerking around. I suspect most of those capabilities come from Jev's base model, not from its RLCD. A durable advantage is consistently more reliable and calibrated probabilities, which translates directly into better decision paired with thresholds. It does not exist yet on Jev, and most capable teams should build from internal.

这条是引用着另一条写的：x.com/brainsandtennis/status/2100998807903482364