A 17M-parameter model can be finetuned to beat Jev 80% of the time
The author tested finetuning a 17M-parameter Ettin model and found it beat Jev in 80% of cases, with synthetic data costs, 3090/CPU/phone-GPU training times, and criteria for when finetuning is worth it.
80% of the time a 17 million parameter model can be finetuned to beat Jev. 40% of the time a 17 million parameter model can be finetuned, on human label, to beat Kimi-k3. 20% of the time a 17 million parameter model can be trained on Kimi-k3 synthetic data and beat Jev. An aditional 50% (so 70%), that distilled model in ball park of Jev. Sometimes, general models (like kimi and Jev), or still better because you training data does not cover well you test (prod?) data. It can take from 8 seconds to ~7 mins getting the synthetic data needed and cost between 0.77$ to ~30$. It takes between 45 seconds to 3 minutes to finetune the 17m (Ettin) model on a 3090 gpu. its 10 mins to 40 mins on cpu, and about 20 mins to 3 hrs on a samsun s21 gpu (XD just i did it!). If you have more then 500 000 inputs to classify, you should consider finetuning on synthetic data it will be both faster and cheaper then using alternatives, if you have about 150 labels per categories (human label) it will also likely be better!