Astra's 72.6% OSWorld score breaks down to 4% on ten-step tasks
An article traces desktop agents from MIT's 1979 Put-That-There to Astra's 72.6% OSWorld score, noting that ten chained steps succeed only 4% of the time at high cost.
OpenAI opened yesterday's launch film with a clip from 1979. A man sits in a leather chair, points at a wall, and says create a large green circle there. The wall does it. Then he moves a cruise ship across the Caribbean by pointing at it. That is Put-That-There, built at MIT by Richard Bolt, Chris Schmandt and Eric Hulteen. The film attached here is the 1981 cut and it runs ten minutes. In those ten minutes the machine says where twenty seven times. It never guesses. Forty seven years after that demo, OpenAI cuts from the clip to a man in a loft talking to a wall. Greg Brockman tells reporters welcome to the AGI era. The number under the announcement is 72.6 percent. That is GPT-6 Astra on OSWorld, a benchmark of ordinary desktop work, at about forty minutes a task. The model it replaces scored 65.7 at seventy five minutes. Every headline yesterday carried that number. Nobody chained it. Ten steps at 72.6 percent each finishes 4.07 percent of the time. Ten steps is not an exotic pipeline. Pull the invoice, match it to the order, fix the address, refile it, tell the customer. That is a Tuesday. It completes four times in a hundred. You are billed for the other ninety six. A forty minute run is a few hundred screenshots. At ten dollars a million tokens in and fifty out, call it six dollars sixty a run. One finished ten step job costs about five hundred and sixty eight dollars. Sixty six of that is the run that worked. The ladder is the part that never makes an announcement. METR publishes how long a task a model can finish, and it publishes that length at fifty percent success. Toby Ord fitted the decay curve. Reliability does not cost length linearly. It costs it exponentially. Fifty percent reliable: the length in the chart. Eighty percent: about a third of it. Ninety percent: about a seventh. Ninety nine percent: about one seventieth. Payroll does not run at fifty percent. Refunds, filings, shipping labels, reconciliations. Those run at ninety nine and up. The horizon you can actually buy is one seventieth of the one being advertised. The progress is real and I am not arguing with it. METR clocks the doubling at roughly seven months, and the eighty percent curve doubles just as fast. But a factor of seventy is a bit over six doublings. Six doublings at seven months each is three and a half years. That is the distance between the demo and the invoice, at the industry's own measured rate. The 1981 machine asked where twenty seven times because it was not allowed to be wrong. The 2026 one does not ask. It bills you for the difference.
This post quotes another post: x.com/andreysuperior/status/2094744606533455996