Jev API hands-on: classification and confidence only, no new tokens
After an hour-long livestream testing the Typesafe Jev API, the author says it cannot generate new tokens, requires a programmatically built candidate list, and only outputs Choice, Noul, or Score results.
Spent an hour on YT livestream today with the Jev model via the typesafe API. Thanks to typesafe team for helping out with credits. The main use-case for this LM is to predict context aware decisions when the decision space of the problem is known apriori. The biggest downside is that you CANNOT generate new tokens - it only works as a classifier + confidence-predictor. You cannot use it even for extraction or form-filling stuff. You always have to pass in a list of candidate answers as well. IOW, you must programmatically create a candidate list BEFORE calling the API - and the model will output the probabilities for each entry given the context. You can send few shot examples and instructions too... dspy-style... but the output schema is fixed into 3 types of things: Choice (enums), Noul (boolean+probability), and Score (ordered value) We also went into masked LM and parallel constrained decoding. We discussed theories about how the model functions - and how it happens so quickly. Random thought: I feel like the next version of this model is probably going to be running small autoregressive/diffusion heads over the output token embeddings. This way prefill can still remain parallel, but arbitrary token output is also possible.
This post quotes another post: “Hands-on with the Jev API: declare output shape like DSPy”