Apple releases RSI-Inference Bench for inference engineering
Apple ML Research built a benchmark from real vLLM and SGLang tasks to test whether AI agents can optimize inference systems; the best agent reached only 26.7% pass@1.
Introducing RSI-Inference Bench from 🍎 Apple ML Research 🌐 http://rsi-inference.com 💻 https://github.com/apple-aiml-research/ml-rsi-inference-bench Can AI agents improve the systems that run AI? We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering: 🚀 Day-0 Model Support ⚡ Inference Optimization 🔧 Bug Fix → Unsaturated: best agent at 26.7% pass@1 → Comprehensive coverage of the inference stack → Verifiable rewards: binary and continuous Watch the video to find out more!