Stanford scan of 36,710 repos finds 6,290 "successes" that never ran the agent
The poster cites a Stanford study of 36,710 repositories to argue many agent workflows only report green, and says it is why his GPT-6 Astra setup is built around verifiable evidence of completed work.
STANFORD STUDENTS SCANNED 36,710 REPOS AND FOUND A WORKFLOW THAT SUCCEEDED 6,290 TIMES WITHOUT EVER RUNNING ITS AGENT Every run was green The search that was supposed to trigger the agent matched nothing. Again and again. The workflow kept reporting success That finding is why I’m building my GPT-6 Astra setup around evidence of completed work The researchers confirmed autonomous agent loops in 217 repositories. Then they searched for the building blocks everyone says these systems need > TRIGGER: what starts the work > STOP CONDITION: what counts as finished > VERIFIER: what can reject the result > STATE: what survives into the next run > BUDGET: how far the system can go before it pauses The scan found no matching stop conditions, no verifier subagents referenced by loop artifacts, and no budget files Two candidate state files turned out to be unrelated to loops That’s what showed up in committed files. Some systems keep their state elsewhere, and a PR review can finish without carrying anything into another run But 6,290 successful runs with zero agent executions makes the measurement problem pretty clear The top row shows the progression from a prompt to a recurring loop. Underneath are the five blocks that make the process inspectable For my Astra build, I want each run to leave something I can actually check: what it worked on what it produced whether verification passed what stopped the run what still needs me Then I can open the terminal in the morning and inspect the work instead of trusting a green status Full paper and breakdown below
This post quotes another post: x.com/n01ennn/status/2096962591125905888