Benchmarks should standardize on cost per task, not token price, author argues
The author lists input and output prices for Gemini 4 Argon, Opus 5.5 and GPT-6 Astra, calls Argon notably efficient, and says its multimodal capabilities are worth waiting for.
for benchmark scores we should standardize around making it really easy to find details like: - $/Task - Harness - Reasoning level - Infra used (for latency) @ValsAI makes nice reports covering some of these details per individual bench they use, some details vary per bench (like infra limits) the thing ppl actually care about is “how much money do I pay to have this unit of work done” you can’t tell that from Token pricing because the harness and model efficiency wildly affect total cost/speed see: gemini 4 argon: $2 / $10 opus 5.5: $4 / $20 gpt-6 astra: $10 / $50 different models also have varying efficient levels on different domains depending how they were trained Argon looks pretty sick for efficiency 👀 Google also always has great multimodal capabilities, excited to try there once it’s publicly released!
This post quotes another post: x.com/valsai/status/2105388446844072033