A sober breakdown of Jev's pricing demo: it saves money, not accuracy
The author notes that 10,000 yes-or-no calls at $0.42 is a pricing calculation rather than an accuracy benchmark, and warns that a 52% vs 46% win label discards the uncertainty that should block an automatic release.
10,000 yes-or-no calls at 3 cents each is $300. expensive way to fill in checkboxes... ryven's worked example puts that count at $0.42 with jev, assuming roughly 1,000 input tokens per decision. that's a pricing calculation, not a benchmark showing identical accuracy. the 32-second terminal visualizes the split across 16 modules: take the request, route the work, run the workers, verify, gate, record. the counters and dollar amounts are demo telemetry, not a measured engineering run. writing a patch needs generation. choosing the next worker from a fixed list is a different job. counting changed files or checking whether tests passed shouldn't reach either model. code already has those answers. the awkward part is what happens when “ship” wins by almost nothing. in the article's example, it beats “return” 52% to 46%. reading only the winning label throws away the uncertainty that should stop an automatic release. a typed answer can still be wrong. the gate needs tested thresholds, hard checks and a review path. accepting the output format is not the same as accepting the change. the useful audit is counting how often your agent writes something, and how often it merely chooses something. you've probably been billing both as the same job.
This post quotes another post: x.com/imryven/status/2104255840819491030