用 GPT-6 Astra 五小时给 vLLM 加 GGUF 支持
开发者借助 GPT-6 Astra 在 5 小时内为 vLLM 实现 IQ/Q2 到 Q8 的 GGUF 量化支持，batch size 8 下比 llama.cpp 快约 30%，但仅支持 Qwen3.5 架构、只在 Ampere GPU 上测过。
I'm working on adding GGUF support to vLLM through the Humming kernels. In just 5 hours with GPT-6 Astra, I managed to get full support for quantization formats from IQ/Q2 to Q8, matching llama.cpp's speed in my benchmarks and outperforming it by ~30% at batch size 8. Still significantly slower than native Humming with GPTQ layers, though. A year ago, this would have taken me days, if not weeks, of work. What a time to be alive! Of course, it's still messy and limited in scope: Qwen3.5 architecture only Ampere GPUs tested (Grace Hopper and Blackwell support is implemented but untested) But it already achieves my only goal: significantly speeding up my GGUF evaluations. I'll definitely do this more often going forward. I might publish the code, but with no promises regarding maintenance.