开源复刻 Jev：151M 参数单次前向决策模型

Hemant @heman10x原推发布于 2026.09.18

在 X 上打开原推

TypeSafe AI came out of stealth with Jev, and access is behind a waitlist. I built an open source version Verdict (Open-jev) you can run right now in a browser tab: And its a real post trained model..(link in comments) It is a post trained 151M parameters model. ModernBERT-base with a GLiClass head. Instead of token-by-token autoregressive decoding or JSON string parsing, it takes an input text and candidate options, then returns calibrated probabilities and an explicit "insufficient evidence" outcome in a single forward pass. Some of The numbers: Calibration: 0.83% adaptive expected calibration error on 1,000 held-out cases. I trained with a multi-class Brier score term, then tried standard post-hoc temperature scaling. Temperature scaling degraded the metrics, so the model ships with temperature 1.0. Order invariance: reversing the option list flips 3.0% of predictions across 1,000 cases. For reference, another open Jev reproduction measured a frozen Qwen3.5-4B at 27.8% flips on the same perturbation, though on their own 36-case fixture rather than mine. Full bidirectional attention evaluates the candidate list simultaneously instead of scanning left to right. Shuffled-context control: Pairing questions with the wrong document drops accuracy from 95.0% to 22.4% and drives abstention from 21.8% to 72.7%, confirming the model actively conditions on the context. Latency: 89ms median in-browser via WebGPU, and 35.6ms via ONNX on CPU at 5 candidates. The negative results: I ran 12 use cases on public data across both checkpoints and three label framings three passed the promotion gate. Passage re-ranking hit 11.9%, jailbreak screening 34.9%, and phishing detection 57.5%. On 337 cases from TypeSafe's public evaluation set, Jev reaches 90.8%, a 26B diffusion model reaches 88.4%, and Verdict reaches 48.1%.which is expected since its just 151M param model but working on trying it on bigger scale as well. A 151M encoder is calibrated, order-invariant, and fast, but it lacks broad world knowledge. In a tiered architecture where a cheap model gates an expensive one, knowing exact failure boundaries is essential. 48 of the 50 classification errors fall below a 0.95 confidence threshold, making the gate predictable. For the next steps i will be testing the same harness on a larger backbone a bigger model with more params to see whether order invariance holds at scale or depends strictly on bidirectional attention. Demo and repository links are in the first in the first reply. Will be trying this on a bigger variant soon, let me know your thoughts what i can do better would love to know your thoughts as well @CompleteSkeptic @typesafeai

这条是引用着另一条写的：「ChatGPT 联合发明人发布新模型 Jev」