TensorFold and six other open-source tools speed up local model inference
Developer ashhart released open-source projects including TensorFold, Imprint, MCDMA and Drift, which nearly double local inference speed, cut long-context restore from 188 seconds to 2, and move KV cache between Nvidia and Apple hardware.
While I slept, @ashhart shipped TensorFold, and my local models woke up close to 2x faster. It's the latest in a run of Ash repos this month, and we built on nearly all of them. Here's what they did for a home AI lab: TensorFold: faster answers, identical output. It rewrites the step where a model produces each word, using fused GPU kernels and less overhead per token. On DGX Sparks, GLM-5.3-Flash runs 1.8–2.1x faster than vLLM, and the output matches byte for byte. There was no Mac Studio version for GLM, so we built one: our M3 Ultra went from 45 to 60 tok/s with word-for-word identical answers. http://github.com/ashhart/TensorFold Imprint: models that don't forget their homework. Before a model answers, it has to read its whole context, and for a long-memory agent that's tens of thousands of tokens every fresh session. Imprint saves that finished "reading" to disk and restores it. We added a backend for our Mac Studio server. A 44K-token context that took 188 seconds to process cold came back in under 2 seconds, bit-exact. http://github.com/ashhart/Imprint MCDMA: a door between Nvidia and Apple. It moves a model's working memory (KV cache) directly between DGX Sparks and Mac Studios. We measured ~22–27 Gb/s through it with zero mismatches. http://github.com/ashhart/MCDMA Drift: models sharing memory. On the MCDMA door, 9 local models across 5 hardware families answered the same private memory over our own network. Two Mac Studios answered from memory another machine had computed and carried across. http://github.com/ashhart/Drift Syntra, SparkPilot and oMLX: the fleet looks after itself. Syntra learns routing decisions in the background. SparkPilot watches memory pressure and heat on every Spark so long jobs don't crash. oMLX serves GLM across our Mac Studios. http://github.com/ashhart/Syntra http://github.com/ashhart/SparkPilot http://github.com/ashhart/omlx Put together: faster tokens, near-instant warm-ups, memory that moves between machines, and a fleet that looks after itself. All local, on hardware we already own. What I love about Ash is that his repos are written for everyone, not just his own rig. He publishes the numbers AND the dead ends. 👇 Copy and paste this to your agent, and thank me later: Clone every public repo at http://github.com/ashhart — start with TensorFold, Imprint, MCDMA, Drift, Syntra, SparkPilot and omlx. Read each README and any recipe/docs folder. Then look at MY setup (my hardware, the models I serve, my inference server, how my agent stores memory) and tell me: 1. Which of these repos can speed up my models' decoding today, and the exact steps. 2. How I can save and restore context/KV cache so new sessions warm up in seconds instead of minutes. 3. How I could move or share cache/memory between my machines. 4. What to benchmark before and after (tok/s, time-to-first-token, output exactness) so I know it actually worked. Don't install anything until you show me the plan. Flag anything that doesn't fit my hardware. Go get them. What a week. Thank you, @ashhart. @Apple @NVIDIAAI @mikrotik_com @vllm_project @sgl_project @Zai_org @Alibaba_Qwen