Architecture discussion for training an open-source Jev from scratch
A developer plans to build an open-source Jev for a YouTube project, discussing a bidirectional attention transformer, masked-LM training and fixed-schema structured output, and comparing three output-head designs.
Chat told me to build open source jev for the next YT project That would be fun. I am thinking of using a bidirectional attention transformer and borrow ideas from masked LM training. Training data + prompt format + loss fn are the main unknowns. I will generate synthetic datasets in the <PROMPT> <CHOICES> <JSON OUTPUT> format, and train only on the json values instead of the full JSON string (i.e. the schema is fixed & guaranteed to follow the structured out). In other words, a fill-in-the-blank model with verifiable outputs. So one can even do RLVR training. I am not sure what the best architecture for the output-head is - these all seem doable: (a) Train it over a list of special tokens such as "CHOICE_1", "CHOICE_2" etc. The choices get passed int he prompt, and we train using standard cross-entropy loss. OR, (b) it could do some kind of encoder-decoder architecture where it matches the embedding of the actual choices against the masked token embedding. Like a contrastive loss OR, (c) it actually generates probabilities over the entire vocab and uses constrained decoding to figure out the best choice from the available list of choices. This technically requires an autoregressive LM to generate the tokens using llguidance/outlines stuff - so I am pretty sure that this is not what JEV does Lets see, currently leaning towards (b)