Jev 的校准概率与「该不该交给人」测试

mashrur haider @Mhr1036原推发布于 2026.09.19

在 X 上打开原推

An AI model can have lower average accuracy and still be more useful for automation, if it reliably identifies which decisions to hand off. That is the most important claim to test in Jev, TypeSafe’s new “System One Model.” Jev makes narrow judgments: choose an option, assign a score, estimate whether a statement is true. TypeSafe reports response times of 70–500 milliseconds and charges $0.042 per million input tokens, with no output charge. Its output is designed to become the next step in a program. You provide context, questions, and permitted answers. Jev evaluates multiple questions in parallel and returns structured values and probabilities. Each question is evaluated independently; your code combines the results and enforces the rules. For example, a support system could separately ask whether a customer wants a refund, whether required information is missing, and whether the request needs review. Code then applies the policy. The architectural trade-off is explicit: Jev gives up free-form text generation and the sequential decoding that comes with it. The training objective matters too. TypeSafe calls its approach Reinforcement Learning for Calibrated Decisions. The goal is for probabilities to match observed outcomes: events assigned an 80% probability should occur roughly 80% of the time across many predictions. That could make uncertainty something software can act on. A system could handle routine cases cheaply, request additional evidence when needed, and send difficult cases to a stronger model or a person. Developers are already exploring parts of this design. • Supercov uses Jev to assess code quality. It asks questions about source files and calculates scores in code, giving coding agents weaknesses to investigate. The project reports roughly a cent per megabyte of source for scoring. • Fast Jev Compaction uses it to decide what a coding agent should retain. The plugin evaluates earlier tool calls and results, then keeps, truncates, or removes them. Retained content stays verbatim. • TypeSafe built Doom and Wikipedia navigation bots. The Doom demo makes roughly ten queries per second from structured game state. The Wikipedia bot chooses among available links to reach a target page. These are early implementations. What interests me is the role the model plays. In the coding tools, Jev helps determine what another model reads and what work deserves attention. Those decisions can influence the cost and quality of everything that follows. A context filter might save thousands of tokens. It might also discard the one error message needed to solve the problem. A quality scorer might direct an agent toward a useful refactor. It might also send it into hours of unnecessary changes. My interpretation: the value of a cheap decision model depends heavily on the expensive work its decisions cause or prevent. There is an important research distinction here. Calibration alone is insufficient. Imagine a dataset where 80% of requests qualify for approval. A model that assigns every request an 80% approval probability can be calibrated while providing no way to distinguish the requests that should be rejected. For automation, you also need confidence to separate cases the model handles well from cases it handles poorly. Research on selective prediction makes this measurable: as the model declines more cases, how does the error rate change among the cases it accepts? SelectiveNet explicitly trained prediction and rejection together. That is a more useful test of Jev’s automation claim than average accuracy alone. The early evidence is mixed. An independent pilot found Jev more accurate than GLiNER2.5 on news classification and banking intent, but worse calibrated on emotion classification. The sample contained only 100 examples per task.