MiniMax M3.1 Flash EXL3 quants: 428B on one DGX Spark at 96.89% token agreement
Two SAGE-EXL3 releases fit the 428B model on one or two DGX Sparks with 96.89% and 98.44% top-1 agreement, though vLLM currently caps context at 32K.
🚨 @MiniMax_AI 𝗠𝟯.𝟭 𝗙𝗟𝗔𝗦𝗛 — 𝟰𝟮𝟴𝗕 𝗣𝗔𝗥𝗔𝗠𝗘𝗧𝗘𝗥𝗦. 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞. 𝟵𝟳% 𝗧𝗢𝗞𝗘𝗡 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧! 🧪🔥 I've done it AGAIN, and this time I'm releasing TWO new SAGE-EXL3 quantizations of the brand-new MiniMax M3.1 Flash! 428B total parameters. 23B active. Native 1M-token context capability. My goal was simple: 𝗦𝗤𝗨𝗘𝗘𝗭𝗘 𝗧𝗛𝗘 𝗠𝗢𝗗𝗘𝗟, 𝗡𝗢𝗧 𝗜𝗧𝗦 𝗜𝗡𝗧𝗘𝗟𝗟𝗜𝗚𝗘𝗡𝗖𝗘. And the fidelity measurements are some of my most exciting yet! 🤯 ━━━━━━━━━━━━━━━━━━━━ 🟢 𝗥𝗘𝗟𝗘𝗔𝗦𝗘 #𝟭 — 𝗢𝗡𝗘 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞 𝗠𝗶𝗻𝗶𝗠𝗮𝘅-𝗠𝟯.𝟭-𝗙𝗹𝗮𝘀𝗵-𝗘𝗫𝗟𝟯-𝗦𝗔𝗚𝗘-𝟮.𝟭𝟱𝗯𝗽𝘄 📦 108.64 GiB checkpoint 🧠 96.89% Top-1 Token Agreement 🎯 99.75% Top-5 Agreement 📉 Mean KL Divergence: 0.02528 𝗔𝗟𝗟 𝗢𝗙 𝗧𝗛𝗔𝗧 𝗜𝗡 𝗔 𝗦𝗜𝗡𝗚𝗟𝗘 𝟭𝟮𝟴𝗚𝗕 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞! And YES, I already have a working vLLM serving configuration! But there's a catch, and I'll explain why shortly. 👀 ━━━━━━━━━━━━━━━━━━━━ 🟣 𝗥𝗘𝗟𝗘𝗔𝗦𝗘 #𝟮 — 𝗧𝗪𝗢 𝗗𝗚𝗫 𝗦𝗣𝗔𝗥𝗞𝗦 𝗠𝗶𝗻𝗶𝗠𝗮𝘅-𝗠𝟯.𝟭-𝗙𝗹𝗮𝘀𝗵-𝗘𝗫𝗟𝟯-𝗦𝗔𝗚𝗘-𝟰.𝟮𝟲𝗯𝗽𝘄 📦 209.88 GiB checkpoint 🧠 98.44% Top-1 Token Agreement 🎯 99.94% Top-5 Agreement 📉 Mean KL Divergence: JUST 0.00592! 𝗡𝗜𝗡𝗘𝗧𝗬-𝗘𝗜𝗚𝗛𝗧 𝗣𝗢𝗜𝗡𝗧 𝗙𝗢𝗨𝗥 𝗙𝗢𝗨𝗥 𝗣𝗘𝗥𝗖𝗘𝗡𝗧 𝗧𝗢𝗞𝗘𝗡 𝗔𝗚𝗥𝗘𝗘𝗠𝗘𝗡𝗧! 🤯 That's exceptionally close to the unquantized reference in my held-out token-level evaluation. Both releases were measured against the original unquantized MiniMax checkpoint using full-vocabulary teacher-forced evaluation. These are quantization-fidelity measurements, not task pass rates or proof of mathematically lossless behavior. But THESE are the kinds of quality-per-bit results I'm chasing! ━━━━━━━━━━━━━━━━━━━━ ⚠️ 𝗧𝗛𝗘 𝗣𝗥𝗢𝗕𝗟𝗘𝗠 𝗜𝗦𝗡'𝗧 𝗝𝗨𝗦𝗧 𝗧𝗛𝗘 𝗠𝗢𝗗𝗘𝗟 𝗔𝗡𝗬𝗠𝗢𝗥𝗘. It's the inference engine. My current ONE-SPARK vLLM recipe WORKS, but it's still a WORK IN PROGRESS. The measured configuration currently reaches 32K context using a quantized KV cache. MiniMax M3.1 Flash supports up to 1M native context, but vLLM's memory overhead on this hardware is consuming headroom I'd rather dedicate to the model and its context. I could shrink the weights further to compensate. 𝗕𝗨𝗧 𝗪𝗛𝗬 𝗦𝗔𝗖𝗥𝗜𝗙𝗜𝗖𝗘 𝗠𝗢𝗗𝗘𝗟 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬 𝗧𝗢 𝗔𝗖𝗖𝗢𝗠𝗠𝗢𝗗𝗔𝗧𝗘 𝗥𝗨𝗡𝗧𝗜𝗠𝗘 𝗢𝗩𝗘𝗥𝗛𝗘𝗔𝗗?! That's the wrong trade-off for what I'm trying to accomplish. I'm optimizing for the BEST possible balance between model quality, memory usage and practical inference speed. And that's where something else I've been building comes in... ⚡ 𝗥𝗘𝗫𝗟𝟯. Our upcoming public inference engine and dedicated recipe are aimed at removing these limitations and making the model's full native context practical on Spark. We're continuing to validate that work before release. 𝗦𝗧𝗔𝗬 𝗧𝗨𝗡𝗘𝗗. 𝗧𝗛𝗘 𝗣𝗨𝗕𝗟𝗜𝗖 𝗥𝗘𝗟𝗘𝗔𝗦𝗘 𝗜𝗦 𝗚𝗘𝗧𝗧𝗜𝗡𝗚 𝗖𝗟𝗢𝗦𝗘𝗥! 🚀 ━━━━━━━━━━━━━━━━━━━━ 🤝 𝗖𝗔𝗟𝗟𝗜𝗡𝗚 𝗔𝗟𝗟 𝗧𝗪𝗢-𝗦𝗣𝗔𝗥𝗞 𝗗𝗘𝗩𝗘𝗟𝗢𝗣𝗘𝗥𝗦! Here's where I'd love some help from the community! The 4.26 bpw model is already available on Hugging Face, but I haven't had the time to properly build and validate its TWO-SPARK serving recipe. If you have two DGX Sparks and experience with distributed inference, I'd LOVE to collaborate! Fork my GitHub recipe. Experiment. Submit a PR with your improvements and measurements. 𝗜 𝗪𝗔𝗡𝗧 𝗧𝗢 𝗪𝗘𝗟𝗖𝗢𝗠𝗘 𝗖𝗢𝗡𝗧𝗥𝗜𝗕𝗨𝗧𝗢𝗥𝗦 𝗔𝗡𝗗 𝗚𝗜𝗩𝗘 𝗧𝗛𝗘𝗠 𝗣𝗥𝗢𝗣𝗘𝗥 𝗖𝗥𝗘𝗗𝗜𝗧 𝗙𝗢𝗥 𝗧𝗛𝗘𝗜𝗥 𝗪𝗢𝗥𝗞! 💚 I want these releases to push the quality-per-bit frontier for EXL3, while making it easier for the community to reproduce and extend the inference results. ━━━━━━━━━━━━━━━━━━━━ 🔗 𝗢𝗡𝗘 𝗦𝗣𝗔𝗥𝗞 — 𝟮.𝟭𝟱𝗕𝗣𝗪 https://huggingface.co/vcruz305/MiniMax-M3.1-Flash-EXL3-SAGE-2.15bpw 🔗 𝗧𝗪𝗢 𝗦𝗣𝗔𝗥𝗞𝗦 — 𝟰.𝟮𝟲𝗕𝗣𝗪 https://huggingface.co/vcruz305/MiniMax-M3.1-Flash-EXL3-SAGE-4.26bpw 🛠️ 𝗥𝗘𝗖𝗜𝗣𝗘 & 𝗖𝗢𝗠𝗠𝗨𝗡𝗜𝗧𝗬 𝗣𝗥𝗦 https://github.com/vcruz305/MiniMax-M3.1-Flash-EXL3-Spark-recipe 𝗧𝗪𝗢 𝗤𝗨𝗔𝗡𝗧𝗦. 𝗧𝗪𝗢 𝗠𝗘𝗠𝗢𝗥𝗬 𝗧𝗔𝗥𝗚𝗘𝗧𝗦. 𝗢𝗡𝗘 𝗣𝗥𝗜𝗢𝗥𝗜𝗧𝗬: 𝗙𝗜𝗗𝗘𝗟𝗜𝗧𝗬. And the next chapter is getting very interesting. 🧪🔥
This post quotes another post: “MiniMax M3.1 Flash open weights: 428B total, 23B activated”