Analysis of the architectural gap between CLM and Jev on decision tasks
The author argues CLM's two-tower design encodes the question and each option separately, leaving it below chance on binary tasks and not comparable to Jev on the Decision Index.
You guys might've seen this launch for CLM as its going viral. I want to congratulate their team for this generous release and insightful writeup. However I'd like to address some "on par with Jev" claims. Here are the mathematics and architectural reasons behind their model performance falling behind jev on HuggingFace's Decision Index: The state and each option are encoded separately and never see each other. That works when options mean something by themselves, like tools, moves or documents to rank. It breaks when an option's meaning depends on the question. For "Yes" and "No" on a sarcasm question, the "Yes" vector is the same whatever was asked. That's why it's below chance on binary tasks: BPoMP scores 37, iSarcasm 14.8, and ESCI 4.1 macro-F1. It also can't reason. GSM8K is 27 and GPQA is 21, because a single embedding can't do multi-step inference. How it compares to the "on par with Jev" claim. That holds on the four tasks they chose: T-Rex, BFCL v4, WikiRacing and Super Mario. All four are action-selection problems where two-tower models shine. On the broad Decision Index it isn't comparable to Jev or to us. Their larger multimodal CLM-35B, due next month, will likely help, but the architecture limit remains.
This post quotes another post: x.com/jackyk02/status/2102905335925424285