GPT-6 Astra used to add GGUF support to vLLM in 5 hours
A developer used GPT-6 Astra to add full IQ/Q2–Q8 GGUF quantization support to vLLM in five hours, beating llama.cpp by ~30% at batch size 8, though only Qwen3.5 on Ampere GPUs was tested.
I'm working on adding GGUF support to vLLM through the Humming kernels. In just 5 hours with GPT-6 Astra, I managed to get full support for quantization formats from IQ/Q2 to Q8, matching llama.cpp's speed in my benchmarks and outperforming it by ~30% at batch size 8. Still significantly slower than native Humming with GPTQ layers, though. A year ago, this would have taken me days, if not weeks, of work. What a time to be alive! Of course, it's still messy and limited in scope: Qwen3.5 architecture only Ampere GPUs tested (Grace Hopper and Blackwell support is implemented but untested) But it already achieves my only goal: significantly speeding up my GGUF evaluations. I'll definitely do this more often going forward. I might publish the code, but with no promises regarding maintenance.