Pipecat voice pipeline test: Jev beats GPT-5.6 Luna on command accuracy and latency
In the speech-to-structured-command step of a Pipecat pipeline, Jev reached 92.6% command accuracy at 296ms median latency, versus 81.3% and 1008ms for GPT-5.6 Luna.
.@jonptaylor recorded a detailed walkthough of Jev vs GPT-5.6 Luna as the "operator" element of a Pipecat speech interface pipeline. GPT-5.6 Luna: - 81.3% command accuracy - 1,008 ms median latency Jev - 92.6% command accuracy - 296 ms median latency A few notes here ... 1) We expected to see a big latency benefit. But the higher accuracy is maybe more interesting. Jev (with a bunch of code wrapped around it) is much better at turning messy transcriptions from a noisy speech environment into correct command structs. The very hand-wavy explanation here is something like: LLM tool calling is an amazingly flexible, general capability but is only one thing that LLMs are trained to do. Jev is trained to do a narrower thing: to produce probabilities over a fixed set of options. If we can express our tool calling requirements as fixed set of choices, it's perhaps not surprising that the model trained specifically and only to evaluate choices does a better job at that task. Some keywords to talk to your friendly neighborhood LLM about this topic, if you're interested in a deeper dive, are: discriminative vs generative models, calibrated probabilities, constrained decoding, and closed-set vs open-set prediction. Note that this is GPT-5.6 Luna without reasoning enabled. With reasoning, today's LLMs are going to do much better on evals like this. Reasoning lets models build a scratch pad of options, which is massively helpful in classification tasks. But we generally can't use models in reasoning/thinking mode for applications like voice agents or voice user interfaces, because the latency is too high. 2) An application-specific eval lets us make informed choices about whether we're getting measurable benefits from different models and different architectures. It's gotten a lot easier to build pretty good, very specific evals as part of agent development. Fable and Astra know how to do this and mostly get things right without a lot of guidance. 3) Using Jev feels more like programming than using an LLM does. Mostly, with LLMs, we're doing "context engineering." Which is just a fancy way of saying we're trying to feed the LLM a big pile of natural language and hoping we're aiming for the right part of the latent space. In most of our sophisticated agents, there's a lot of code to do this context engineering: constant summarization, switching prompts for different parts of the conversation, etc. But the end result is always just really long prompt strings! With Jev, we're sending the model natural language, but in addition we're doing very explicit mapping between application state and Jev's input data structures.