Databricks AIR opens to all customers: agent swarms self-provision GPU training endpoints
A Databricks researcher says his team now runs experiments via agent swarms, and AIR is available to all customers so agents can provision GPU endpoints from <1B to 1T+ and monitor them.
Over the course of the past year, my research team at @databricks has completely shifted from manually running every experiment to having a swarm of agents run experiments for us. As a result, the training workloads of my research team have skyrocketed and continue to rise. Fortunately, we have a very scalable GPU infra named AIR. Through AIR, our agents can easily provision GPU endpoints for training models of various sizes (<1B to 1T+) and monitor their status. The agent swarm + AIR give us an infinite amount of execution power, so our focus has shifted towards defining crystal-clear goals and reviewing the outcomes. Our models come out more quickly, the quality goes up, the speed increases, and new products also get launched more quickly… AIR is now available for all @databricks customers to use. We dogfood our own product first ;) Now you should be able to unleash your AI research team with infinite execution power as well.