CTO retests Jev's speed and cost on 1,934 production requests

Jev is fast and cheap, but why? Our CTO spent two days testing it on a $3,500 desktop at home, not on Trioma’s GPUs. He started with 34 handwritten requests, then ran 1,934 real production requests. What we found changes how you should look at Jev. The numbers are below👇 Everyone is talking about Jev, the new "System One" model from TypeSafe AI. The claims are loud: up to 400× faster than GPT, up to 10,000× cheaper, a decision in 70-500 milliseconds, errors that are "mathematically impossible", input at four cents per million tokens, output free. Some people say it labelled ten thousand emails in twenty minutes for a dollar. Others say that on real pipelines the effect is "not zero, but very weak". When something sounds like that, someone should check. So our CTO and co-founder, Dmitriy, did: on a desktop box in his home office, over two days. This is his review, published as he wrote it, including the parts where he loses. TL;DR - The speed and the "impossible errors" come from the form of the question (one token, a decoder grammar), not from the model. Any model you already run can do it. - With zero labels, Jev and a plain 27B open-weight model tie on accuracy: 0.770 vs 0.769. - Chain-of-thought reasoning doesn't help on these tasks. It costs 75× more for the same answers. - What Jev really sells is calibrated confidence with no labels: it routes 4× more traffic automatically than raw model probabilities do. - 50-100 labelled rows from your own logs close that gap. - On batched questions Jev is 2.6× faster than the box. On cost, the desktop box loses to Jev; a $2,000 gaming card would win (projected). - Classification is the one task where local hardware catches up completely, and it's also the task that ships the most data out. Privacy is the real argument, not speed. The rig Deliberately nothing special: an ASUS Ascent GX10, a $3,500 desktop box the size of a thick book, 128 GB of unified memory, running an off-the-shelf 27B open-weight model in BF16 on vLLM. It writes prose at 4.5 tokens per second: slow enough to make coffee while it finishes a paragraph. If the trick works here, it works anywhere. Credit first Jev calls itself a System One model, after Kahneman: fast, intuitive answers. The insight is right. Most of what we ask language models to do isn't writing. It's choosing: which pipeline, which tool, approve or escalate, toxic or fine. Those questions don't need an essay. They need an answer. TypeSafe offers three primitives for this: Choice (pick one of a set), Score (rate on a scale) and Noul (how true is this, from 0 to 1), asked in batches against one shared state, with probabilities attached. The trick behind the speed When a model picks one option out of five, the useful output is one token. A letter. Everything else in {"choice":"D"} is packaging, and decode time scales with output tokens. Ask for one token and read the probabilities straight from the logits: you get a decision plus probabilities from any model, with no training. "Type errors are mathematically impossible" isn't magic either. It's the decoder. If the output grammar only allows A-E, a sixth answer can't exist. In vLLM that's one line, guided_choice. And TypeSafe's documented limit of 255 options looks a lot like one byte per class. About the famous 70 milliseconds That's server time. The road to the server isn't in the ad. Measured from Singapore, a TLS handshake to http://api.typesafe.ai took 367 milliseconds: one AWS region in Oregon, no edge termination. The same handshake to a Cloudflare-fronted API took 19. Where a model lives matters as much as how fast it thinks. (Part two measures the real end-to-end figure: about 460 ms. Anycast saves the handshake; it doesn't save the ocean.) Test one: 34 requests Five routes (text, new image, edit image, video, audio), eight requests deliberately ambiguous. Each asked two ways. One token: 493 ms, 85.3% accuracy JSON object: 1,381 ms, 85.3% Pretty JSON with schema: 3,735 ms, 85.3% Identical answers on all 34, mistakes included. Six of seven tokens were paid for nothing. Why a desktop box can play this game: physics 27 billion parameters × 2 bytes = 54 GB of weights. The chip reads memory at 273 GB/s. 273 ÷ 54 ≈ 5 tokens per second; measured, 4.5. Decoding is a memory-read problem, not a math problem. So eight decisions in parallel read the weights once: 308 ms per decision alone, 110-155 ms in a batch of eight (throughput, not latency; it drifted between runs, and he says so). And putting the context first and the question last means a second question to the same context drops from 3,025 to 341 ms. The cache does the rest. "But surely it needs to think?" Chain-of-thought on, 16 cases. With reasoning: 12 of 16 correct, 36.3 seconds, about 172 tokens each. One token: 12 of 16 correct, half a second. Reasoning didn't remove the errors. It rearranged them, at 75× the price. On a System One task the answer is already in the first pass. Where he lost Raw LLM probabilities are overconfident. In the "95-99.9% sure" bucket the model was right 40% of the time. If you route on a threshold, that's unusable. This is exactly what TypeSafe fixes with training, and a prompt can't. Round two: 2,000 real requests, and Jev itself Then he caught himself cheating: 34 prompts he wrote himself, none in English, median 42 characters, against real traffic that is twice as long, mostly English and messier. So round two: about 2,000 anonymised real requests, and a real head-to-head with Jev via OpenRouter. The run cost 3.5 cents and took seven minutes. First, what does "nothing fancy" score? - Always guess the most common class: 0.207 - Keyword rules: 0.490 - Naive Bayes on word counts, a 1960s algorithm, with - labels: 0.761 - Zero-shot 27B model, no labels: 0.703 A zero-shot LLM is worth about 650 labelled examples. That's a number you can put in a budget. Head-to-head, same prompts, same context, same rules, zero labels: Jev 0.770, his model with five yes/no probes 0.769. A statistical tie (p = 0.89). Detailed class descriptions helped Jev by 6.5 points and hurt his model by 4; there is no universal prompt. With about 1,500 labels, his stack (probes plus a measured confusion matrix) reaches 0.792 against Jev's 0.770 (p = 0.0016). And a fun one: Naive Bayes and the 27B model, on equal data, are a dead heat (p = 0.67). Combined, they reach 0.834. They are wrong in different places. Where Jev is really better Confidence you can trust with zero labels. Share of traffic you can route automatically at 5% error or less: Raw probes: 10.5% Jev: 42.8% Probes plus confusion matrix (needs labels): 59.0% That's the real product: not accuracy, which is ordinary, but calibrated uncertainty out of the box. It's honest value, and he says so. How many labels to close the gap yourself? About 50 gets you level on average; 100-200 puts you comfortably above with far less variance. And those aren't hand labels: they're rows from your own logs saying which pipeline actually ran. Their real form: batches Eight questions in one call take the time of one, cost a quarter of eight separate calls, and gave the same answers in 39 of 40 cases. The Score primitive returns a full distribution plus a confidence; on an unclear case it said 0.07. It knows when it doesn't know. End-to-end from Singapore, real calls: one question, box 364 ms vs Jev 441; eight questions, box 942 vs Jev 361. Their time is flat: all answers come out of one pass. His grows: each question is a separate call. On their form they win by 2.6×. Hardware, not intelligence Decode floor equals weight bytes divided by memory bandwidth, and that transfers across GPUs. Measured on the GX10: 118 ms per question. Projected from bandwidth: about 8 ms on an RTX 4090 at INT4, about 6.7 ms on an H200. Strip the network out of Jev's 361 ms and you get roughly 7.6 ms per question: datacenter class. No miracle. They stand on better hardware. The box lost on hardware, not on method. Economics, counted to the end - Jev: $0.000022 per decision one at a time, $0.0000041 in batches of eight - The GX10, amortised over three years plus power: $0.000006 So the desktop box is more expensive than Jev batched. He didn't hide it. A $2,000 gaming card, projected from bandwidth (INT4 accuracy unmeasured), comes to about 17× cheaper than Jev. Datacenter cards beat Jev by only 1.2-1.5×: their price roughly tracks their hardware. The most expensive lesson in the write-up: don't buy a box with "AI" written on it. Buy memory bandwidth. 128 GB at 273 GB/s is the wrong shape for this job; a gaming card has four times the bandwidth for half the price. GB/s per dollar is the one number that sets the price of a decision. What nobody prices in Someone labelled ten thousand Gmail messages with Jev in twenty minutes for about a dollar. That's also ten thousand emails on someone else's server. Classification needs the full text (you can't ask "is this spam?" without showing the email), and it's bulk by nature (one labelling of an archive is the whole archive). The cheapest task quietly becomes the biggest data export you'll ever do. And classification is the one LLM task where a local box catches up completely. For writing you may need a frontier model you can't host; that trade-off is real. For a one-token decision there is none. The task where you most want data to stay home is the one where staying home is free. The recipe, if you build it yourself - One call, one token: max_tokens=1, logprobs on. Read the probabilities from the logits. - Five yes/no probes ("is this pipeline X?") instead of one five-way choice. - Forbid impossible classes: no attachment means it can't be "edit". Two lines, plus five points, no labels needed. - No labels? Stop here, at 0.769. - A hundred labels or more? Add the confusion-matrix decoder. That's what turns raw probabilities into a threshold you can rely on. - Don't route everything. Escalate the unsure half. Almost none of this is new: one-token classification is GPT-3 era; guided decoding and error-correcting decoders are decades old. What's new is the numbers. Nobody publishes negative results, so everyone relearns them.

This post quotes another post: x.com/princedoesai/status/2102719300587143489