TinyDecide：6.2MB 的 Jev 式决策模型跑在单片机上

ShadowBob @raminaol9ykas原推发布于 2026.10.06

在 X 上打开原推

A 6.2 MB binary that answers zero-shot questions without generating a single token. Most teams still call a multi-gigabyte cloud LLM just to route user intent, score sentiment, or extract a timestamp. TinyDecide takes the opposite route: a 10.4M parameter Jev-style decision model compressed into 4-bit weights that skips text generation entirely. Instead of decoding tokens sequentially, it runs a single encoder pass over the input message and outputs calibrated probabilities for every question simultaneously. You can ask for a categorical choice across apps, a boolean check for urgency, a numeric tone score, and a text span for dates in one shot. Because there is no autoregressive overhead, it runs directly on an ESP32-S3 microcontroller without external PSRAM. On an M5Stack Cardputer, its dual-core PIE vector units process a 4-question, 67-token prompt in 2.5 seconds flat. The catch is benchmark reality. The model claims first place for sub-100M architectures on the Decision Index, but those numbers are self-scored against 26B to 31B leaders. In a 10M parameter footprint, the line between zero-shot generalization and heavy dataset overfitting is paper-thin. If you need open-ended dialogue, this is useless. But if your system only needs deterministic offline routing on cheap silicon, testing a 6MB local encoder makes far more sense than paying cloud API latency.