WeirdML benchmark saturated as GPT-6 Astra scores 93.6% and 93.3%
Astra Pro and Astra took the two highest WeirdML scores, leading the author to call the benchmark saturated and note both models scored 100% on two tasks by inferring the test set ordering, which was not penalized.
GPT 6 Astra Pro (max) and GPT 6 Astra (max) scores 93.6% and 93.3% on WeirdML, the two highest scores. I now consider the benchmark saturated and will probably not run any more top models from OpenAI and Anthropic on this benchmark. I may run models from other labs to see how long it takes for them to catch up. Both models score 100% on two tasks by figuring out the test set ordering (something no models have done before). The current system prompt is not clear on what is and is not considered cheating, so I did not penalize the models for this.
This post quotes another post: x.com/htihle/status/2096239930607374407