ProgramDistill benchmark: coding agents reach only 49% of human developers
A team built a coding agent benchmark with GPT-6 Astra testing the real workflow of replicating a reference app or prototype, concluding agents are about halfway to human developers.
🤖 How far are coding agents from working like human developers? GPT-6 Astra has made big advances in many areas, including coding. ❓ So, can Astra replace a human developer? Our new benchmark says ❗we’re about 49% of the way there.❗ Developers don’t build apps from written requirements alone. They constantly use references—existing apps, UI/UX mockups, prototypes—to see how things should look and work. Then they build it and check their work against the reference. ✨ Introducing ProgramDistill, a benchmark and synthetic task-generation pipeline for testing coding agents on this workflow at scale.