Mine Odyssey: a Minecraft-map benchmark for spatial agent intelligence
A new open-source benchmark has models navigate community-made Minecraft maps by reading maps, finding paths, opening doors and climbing ladders; the authors say GPT-6 Astra and GPT-6.1 Sol reach the destination quickly with far fewer turns.
Introducing an interesting project- Mine Odyssey, a benchmark for measuring spatial agentic intelligence in the wild! TL;DR: We leverage a wide variety of Minecraft maps created by the amazing community (yes, games are great lol) and challenge models to navigate through them by reading maps, finding paths, opening doors, going up and down stairs, climbing ladders, and more. Thanks to the incredible Minecraft modding community, we can also introduce a wide range of variations for ablation studies, including weather conditions, different HUDs, first-person vs. third-person views, and reference route lines, to investigate how these factors affect model performance. When we first launched this project, we honestly didn't think any model would be able to solve these challenges smoothly. But now, GPT-6 Astra and GPT-6.1 Sol can make it all the way to the destination remarkably quickly, with significantly fewer turns! 😨 Meanwhile, other models, even when they manage to reach the destination, often take unnecessary detours or get lost along the way. Our benchmark is fully open-source and Harbor-compatible! Kudos to @SArt_Maxine @llleo0612 @junxian_he More details: https://mine-odyssey.github.io/