WindTunnel benchmark: Astra's code execution browser control costs 49x more
GPT-6 Astra with code execution solved all 49 tasks on the WebMCP benchmark, but was about 49x more expensive and 2.9x slower than GPT-5.6 Luna with WebMCP.
Today we added GPT-6 Astra to WindTunnel, our open WebMCP benchmark. The results are quite surprising: 1/ Both GPT-5.6 Luna with WebMCP and GPT-6 Astra with code execution solve 49/49 tasks, perfect score. (code execution is Astra's recommended method). 2/ WebMCP with GPT-5.6 Luna is ~49x cheaper than GPT-6 Astra using code execution. 3/ WebMCP is also 2.9x faster and uses 4.2x fewer tokens. Full benchmark here: https://webmcp.com/benchmark A few words on how Astra operates the browser, and what these results mean: WindTunnel compare different ways an agent with a browser can operate the same website: WebMCP, screenshot-based computer use, DOM/accessibility, and code execution. For Astra, the recommended method is code execution: you send Astra the page context, it returns a code script that runs the page, you execute it and then send a screenshot of the result back. This lets Astra perform multiple actions in one step instead of going screenshot -> click -> screenshot loop for every action. Code execution allows the agent to operate the browser *really* fast (all of these cool demos you’ve been seeing). And it worked very well: Solved 49/49 tasks (a lot of the “pure” computer use methods get poor results in the benchmark). But compared with GPT-5.6 Luna using WebMCP, it was still: ~49x more expensive 2.9x slower 4.2x heavier on tokens Which means even fancy code execution doesn't beat good old APIs with a good description. In theory Astra should shine on long runs - one script can replace hundreds of clicks - so it may still win where actions are numerous and no tool covers them, like bulk edits on a site without WebMCP. But on sites that do expose tools, we saw the opposite: on our longest tasks, Astra's code execution cost over 100× more than Luna on WebMCP. The benchmark is open-source, feel free to run it yourself and update us with any feedback. If you find WebMCP interesting, we’ll be holding a free WebMCP workshop next week for both developers and enthusiasts. Feel free to sign up and find more information here: https://luma.com/d9epljk6