从零训练开源 Jev 的架构设计讨论
开发者计划为下个 YouTube 项目自建开源 Jev，讨论用双向注意力 transformer、掩码 LM 训练和固定 schema 的结构化输出，并比较三种输出头方案。
Chat told me to build open source jev for the next YT project That would be fun. I am thinking of using a bidirectional attention transformer and borrow ideas from masked LM training. Training data + prompt format + loss fn are the main unknowns. I will generate synthetic datasets in the <PROMPT> <CHOICES> <JSON OUTPUT> format, and train only on the json values instead of the full JSON string (i.e. the schema is fixed & guaranteed to follow the structured out). In other words, a fill-in-the-blank model with verifiable outputs. So one can even do RLVR training. I am not sure what the best architecture for the output-head is - these all seem doable: (a) Train it over a list of special tokens such as "CHOICE_1", "CHOICE_2" etc. The choices get passed int he prompt, and we train using standard cross-entropy loss. OR, (b) it could do some kind of encoder-decoder architecture where it matches the embedding of the actual choices against the masked token embedding. Like a contrastive loss OR, (c) it actually generates probabilities over the entire vocab and uses constrained decoding to figure out the best choice from the available list of choices. This technically requires an autoregressive LM to generate the tokens using llguidance/outlines stuff - so I am pretty sure that this is not what JEV does Lets see, currently leaning towards (b)