Bonsai2 27B runs 262k context on a single RTX 5060 Ti 16GB
PrismML's Bonsai2 27B, a 1.75-bit compression of Qwen 3.8 27B, runs with a 262k context, MTP head and vision all loaded on one 16GB card via the author's llama.cpp fork, generating about 67 tok/s.
every single rtx 5060 ti 16gb vram owner, bonsai2 27b dense flies on your gpu with the full 262k context window, the mtp head and vision all loaded at once, and it runs at 67 tok/s. > 262k context + mtp head + vision on one card: 15.2 of 15.9 gb > 67 tok/s with the mtp head, 71 tok/s on code, 53 tok/s without it, the stock build does 42 tok/s > with the head at depth: 39.5 tok/s at 39k tokens of context, 21 tok/s at 119k > 4 agents at once: 127 tok/s total with the head, ~33 tok/s each, the stock build does 58 tok/s without it > prefill: ~1,060 tok/s, a 119k token prompt read in about 3 minutes > vision: read all ten stats off a dashboard screenshot, answered at 66 tok/s > power: ~155 of 180 w the whole run what you're watching is the first 8 minutes of it thinking through a build i gave it, at 10x. every box is live: speed in the server log, power and memory on the card, prefill and generation speed in the llama.cpp web ui. the model is bonsai 2 27b, @PrismML's 1.75 bit compression of qwen 3.8 27b, with qwen's mtp head grafted back on so it drafts its own next tokens, served by my fork of llama.cpp. v1.1 shipped today with two blackwell fixes, a bug in my kernel that turned output into garbage on rtx 50 cards, and one in llama.cpp's attention that ran long context at less than half speed. next on this card: computer use and browser use. all links below. if you have a single rtx 5060 ti 16gb you can run this tonight, and there's a line for every 30, 40 and 50 series card with 8gb or more.
This post quotes another post: x.com/sudoingx/status/2103922552838009174