Breaking down why Jev is faster than LLMs: parallel forked inference contexts read logits
The author speculates that Jev's speed comes from an inference trick: forking multiple-choice questions in parallel over a shared prefill and reading the logits of option tokens to get a choice plus confidence, decoding once for N questions.
How is Jev so much faster than an LLM? It's not about how the model is trained - it's the inference technique. In fact, you can modify an inference engine to provide a performant Jev-like API with any open-weight LLM. Say you are trying to ask N multiple choice questions in parallel. In standard text generation, you decode tokens one at a time to generate a JSON representing those properties, and then parse the JSON. In a Jev-like system, what you can do is: - Label choices for each property 1,2,3,... - Prefill the shared information (output shape and other context) - Fork this inference context for each property, re-using that cached prefill - Add an individual question to each parallel context - Look at the logits (token probabilities) for each one, constrained to the valid choice labels - Map the labels back to the choice values. Include the softmax of the logit itself to get a combined choice+confidence score This means you decode one time in parallel across all questions, instead of one at a time - so the efficiency gains roughly scale with the number of questions per request (until compute is saturated). So what makes Jev's model unique? It's because they train a small model to perform well in this specific inference setting - improving both cost and capability compared to a non-specialized model. Ultimately though, if you have a use case Jev would be good for, you could do it with open-weight models too! (Some speculation made here about how Jev's engine might work, but this is one way you could get similar performance)
This post quotes another post: “ChatGPT co-inventor releases Jev, a model built for decisions rather than text”