Open Kernel Challenge: human-guided agent fleets beat Astra-guided by 2x
The author spent ~$5k running agent fleets on H200s for GPU kernels and found human-guided fleets progress over 2x faster than Astra-guided ones, while Gemini Argon kept stopping for security reasons.
Update: I’ve been making progress on all the other kernels using fleets of agents, and discovered human-guided fleets are over 2x faster than Astra-guided fleets! We spent ~$5k testing fleets of agents running experiments on H200s. Interesting results from the Open Kernel Challenge: 1) Human-guided fleets now progress over 2x faster than Astra guided fleets, and Astra alone is now too slow to make progress. Early on, human guided Astra made most of the progress based on intuition alone. As the basic memory and GPGPU tricks are exhausted, most of the new kernels breakthroughs come from exhaustive testing on GPUs for optimizations and reorganizations of data/code. So early boosts came from human insight, but now it's mostly based on how much you can spend on GPU testing. 2) Astra sometimes makes obvious mistakes a human expert would catch, like not combining other kernels into the full decode kernels, and not submitting breakthroughs for incorrect rules. 3) The best design is having a human expert keep adding agents till they respond every 1-5 minutes, then guide them with simple quick answers. I now run 30. 4) A human-guided fleet running Gemini 3.8 flash outperforms Astra Ultra alone and the Astra-guided fleet. 5) We spent ~$4500 in Gemini 3.8 flash running the fleet, 30 agents in parallel, and ~$600 in subscriptions running Astra, so the fleet is very expensive to run. 6) Gemini Argon kept stopping for security reasons so 3.8 Flash did better. Most of the time was previously spent waiting for experiments, so I added agents till I didn't have to wait. I built a local queue and lock system for all my GPUs, so they constantly have 100% utilization running independent experiments from the fleet. Sometimes the fleet still goes off the rails, so humans need to re-prompt them. Astra can resolve some of this and verify the fleet, but when Astra goes off the rails you need a human expert. This explains the slowdown when Astra guides the fleet. This is strong evidence that human experts guiding fleets of agents is both important to reach out-of-distribution solutions and make progress faster(for now).
This post quotes another post: x.com/rohanarun/status/2113794964953948336