Distilling a Jev-style decision model into two small self-hosted models
Using 1,000 human-graded messages, a small LLM classifier and an even smaller logistic regression model matched the Open-Jev 27B teacher model, letting users own the model and where data is sent.
Jev from @typesafeai has obviously been on everyone's minds this past week. 'System One' models, decision models, smart classifiers, whatever you want to call them... they're very useful. We even built them into Kitaru as a whole new 'judge evaluator' package. But what if you want to own the model and where the data gets sent? I distilled an open Jev-style model into two small students, inspired by an article by @sjgoedecke (linked below). We used 1000 human-graded messages (h/t the Banking77 dataset on @huggingface) to train a small LLM-driven classifier and an even smaller logistic regression model that performed equally to the Open-Jev 27B parameter teacher model (thanks @Zefan_Cai!).