Robotics reading club: how language models will eat robotics
A CMU speaker argues that once frontier LLMs overflow with intelligence, hard robotics problems become engineering loops, citing Astra's in-context learning and RoboDojo's 22.48% success rate among 12 takes.
🤖 Language will eat robotics. Slowly, then suddenly. @saturdayrobotic & World Model Reading Club #33.1, hosts: @junfanzhu98, @aurorafeng_01, @shawnshenjx (@memories_ai), @BenZhouMemories, @intelchentwo. https://luma.com/y9omqzz0 🎤 @TongheZhang01 (@CMU_Robotics, ENPIRE co-author) opened with Wittgenstein: "The limits of my language mean the limits of my world." His thesis: once frontier LLMs overflow "absolute intelligence," robotics' hardest problems become engineering loops that coding agents can close. 👉🏻 Full Article: https://x.com/junfanzhu98/status/2109345200003059772?s=20 12 hot takes 👇 1️⃣ Dexterity is a reward problem. A coding agent is itself a great reward model. 2️⃣ Difficulty: Reasoning ≫ Dexterity ≫ Speed. LLMs already own the hardest one. 3️⃣ In-context learning comes free with GPT-6 Astra: no fine-tuning, no new data. 4️⃣ Hardware is code: YAML → build123d → STEP → assembled robot. CAD becomes a viewer. 5️⃣ Sim-to-real is optimization: fit the simulator to reality by gradient descent → an action-conditioned world model. 6️⃣ Contact is the last gap. Solve contact verification and real2sim is solved. 7️⃣ Verification is the currency. 10,000+ Mac minis as software sandboxes, 1,000 robots as hardware ones → robotics becomes recursive self-improvement. 8️⃣ Noise is a representation problem. Brute-force every modality into tokens: ViT patches, VQ-VAE codebooks. 9️⃣ Low-level control needs no chain-of-thought. Let the output format decide when to think. 🔟 If end-to-end is feasible, it beats System 1/System 2. 1️⃣1️⃣ Solve intelligence big, then distill. GPT-7-nano on better chips is next. 1️⃣2️⃣ More structure = less scalability. 📊 Reality check: - GPT-6 Astra tops RoboDojo… at 22.48% average success. Precision: 4%. - Naive one-shot demos hurt (22.9% → 12.9–17.9%); RoboICL's structured context took a real Franka from 14 → 79 progress with 3 demos. - ENPIRE's 99% is pass@8, with retries. - π0.5 + Astra: 48% vs 26% for Astra alone. - 250 Astra inferences ≈ 5 s of simulated quadruped walking. 🎯 My read: intelligence is no longer the bottleneck. Verification, contact and latency are. Which take is wrong? Fight me below 🔥 Join us any Saturday: http://luma.com/saturdayrobotic #Robotics #EmbodiedAI #LLM #SaturdayRobotics
This post quotes another post: x.com/junfanzhu98/status/2109345200003059772