Visual explainer shows how Jev outputs valid JSON in one decoder pass
A diagram explains how Jev uses a single Transformer decoder pass with cached keys and values to pick each JSON field's token directly, replacing token-by-token generation.
For anyone curious how Jev works, I made a visual explanation using @claudeai :) This is based on the Qwen2.5-RLCD model which @harshagundal released on @huggingface The idea is to replace autoregressive LLM generation by a single Transformer decoder (of a pre-trained LLM), which processes the context + JSON schema only once. The keys and values of those tokens are cached. Next, for each field of the JSON schema, we: 1. pass its field suffix tokens through the Transformer decoder again (reusing the KV-cache) 2. obtain a final hidden state, which we pass through the language modeling head 3. we obtain scores, also called logits, for all tokens in the vocab of the LLM 4. we only look at the scores of the tokens we care about for the given field, and pass those through a softmax to obtain probabilities which sum to 1 5. we take the token with the highest probability. The benefits of this are that: 1. it's fast (we don't need to generate the JSON schema token by token) 2. it's 100% valid JSON (we don't need to rely on the model to generate a valid schema)
This post quotes another post: x.com/nielsrogge/status/2100114968460820986