231 条真实 agent 决策回放:Jev 式决策层延迟与准确率实测
用 Open-Jev-27B 在单张 B300 上重放 231 条决策,路由/分类 15ms 几乎全对,日期计算 15 条只对 2 条,慢决策吃掉 54% 时间且静默出错。
I built a terminal that shows a Jev-style decision layer working through 231 real agent decisions, and most of what it shows is time spent on a handful of them. Every decision is a light in the galaxy, and the dust around it is the compute it burned. No API here: the model is Open-Jev-27B, an open Jev clone, and every latency is measured on one B300, replayed as it happened. > ROUTING - pick the agent or the tool: 24 of 24 right, ~17 ms each > CLASSIFY - intent and extraction: 46 of 48 right, ~15 ms > JUDGE - pass or fail someone else's answer: 15 of 17 right > DATES - temporal and numeric checks: 2 of 15 right > LONG POLICY - read a rulebook, pick the ruling: 11 of 19 right, ~141 ms > MULTI-HOP - chain facts across the state: 16 of 18 right, ~125 ms 37 of the 231 decisions eat 54% of the clock. Long policies and multi-hops. Only 27 of those 37 come back right, against 170 of the other 194. The cheap forks are the safe ones. Routing, tool selection, ordinal, adequacy, fact: 60 of 60 right at ~15 ms each. The slow ones fail quietly. Decision #121 denied a claim the rulebook says to pay, took 198 ms, and returned the wrong ruling in the same clean format as a right one. Date math is where it really breaks: 2 of 15. I wouldn't hand that to a decision layer without a check behind it. Give the narrow forks to the decision model. Send the rulebooks and the date math up. how to run that decision layer on your own machine, no API, is in the article ↓
这条是引用着另一条写的:x.com/asteri_eth/status/2104901781725598103