DeepSeek V4.1-Flash on 12GB VRAM via GSQ-RCO quantization and Strata
Report of running DeepSeek V4.1-Flash with GSQ-RCO quantization and the Strata engine on 12GB VRAM plus 64GB RAM at 105–117 tok/s, with a caveat that prefill and decode speeds differ.
DeepSeek-V4.1-Flash is reportedly running on consumer hardware with just 12GB of GPU VRAM. The setup combines GSQ-RCO quantizations with the Strata inference engine, allowing a model that would otherwise exceed available GPU memory to run through a combination of GPU and system RAM. One reported configuration uses an RTX 5070 with 64GB of system RAM, reaching around 105 to 117 tokens per second under the tested workload. There’s an important detail, though: prompt processing and text generation are not the same thing. Prefill is much faster than decode. Prefill processes the input prompt, while decode generates the response one token at a time. A system can process prompts quickly but still take longer to generate each new token, so those speeds need to be evaluated separately. Strata uses layer-major chunked reading to manage how model weights are accessed during inference. Instead of requiring the entire model to fit in GPU memory, this approach helps move model data through the available memory resources in manageable chunks. System RAM also matters. When model weights must be offloaded from the GPU, having more RAM can provide more room for the workload and reduce memory constraints. But extra capacity alone doesn’t guarantee faster decoding. Memory bandwidth, storage speed, GPU performance and the inference implementation all affect the final result. So, can you use this setup as a practical replacement for a fully GPU-resident model? That depends on your workload and tolerance for slower generation. For experimentation, though, it’s an interesting way to explore large-model inference without buying a GPU with enormous VRAM. One caveat: the 105 to 117 tokens-per-second figure is a reported result, and its usefulness depends on the exact benchmark, prompt length and whether it measures prefill or decode. Those details matter when comparing setups.
This post quotes another post: x.com/oluwaphilemon1/status/2114543357246771711