Replaying 231 real agent decisions: latency and accuracy of a Jev-style decision layer
Replaying 231 decisions with Open-Jev-27B on a single B300, routing and classification were nearly all correct at about 15 ms, date checks got 2 of 15 right, and the slowest decisions took 54% of the time while failing silently.
I built a terminal that shows a Jev-style decision layer working through 231 real agent decisions, and most of what it shows is time spent on a handful of them. Every decision is a light in the galaxy, and the dust around it is the compute it burned. No API here: the model is Open-Jev-27B, an open Jev clone, and every latency is measured on one B300, replayed as it happened. > ROUTING - pick the agent or the tool: 24 of 24 right, ~17 ms each > CLASSIFY - intent and extraction: 46 of 48 right, ~15 ms > JUDGE - pass or fail someone else's answer: 15 of 17 right > DATES - temporal and numeric checks: 2 of 15 right > LONG POLICY - read a rulebook, pick the ruling: 11 of 19 right, ~141 ms > MULTI-HOP - chain facts across the state: 16 of 18 right, ~125 ms 37 of the 231 decisions eat 54% of the clock. Long policies and multi-hops. Only 27 of those 37 come back right, against 170 of the other 194. The cheap forks are the safe ones. Routing, tool selection, ordinal, adequacy, fact: 60 of 60 right at ~15 ms each. The slow ones fail quietly. Decision #121 denied a claim the rulebook says to pay, took 198 ms, and returned the wrong ruling in the same clean format as a right one. Date math is where it really breaks: 2 of 15. I wouldn't hand that to a decision layer without a check behind it. Give the narrow forks to the decision model. Send the rulebooks and the date math up. how to run that decision layer on your own machine, no API, is in the article ↓
This post quotes another post: x.com/asteri_eth/status/2104901781725598103