用定制 Jev 式 PCD 模型给 agent 流程做前置分流省 token
作者设想企业先用定制 Jev 风格并行约束解码模型对工单、发票、理赔做分类，高置信度时跳过前沿模型，声称可省掉六成 LLM 开销。
Prediction: millionaires will be made using custom Jev style models (parallel constrained decoding) to make the agent systems companies already run more token efficient. Let me explain with a scenario: Imagine a company already has an agent workflow running where an llm reviews every item before it moves on: a support ticket gets triaged, an invoice gets approved or held, a claim gets flagged. Every one of those goes through a frontier model today, a few seconds and a few cents each, on the way to a decision that in most cases is obvious. Behind that flow sits years of humans (or agents) making the exact same call, with the outcome attached. Now imagine you first run each item through a custom PCD or similar model that costs a fraction of the llm and returns a classification of what to do at that step, with a mathematically accurate probability attached. When it's confident, the item skips the llm entirely. When it isn't, the llm handles it as normal. The model has seen years of your team making this exact decision, usually a constrained set of decisions, so it should be right most of the time. Say it comes back confident on 6 out of 10 items. That's more than half your llm spend potentially gone from that step, likely with comparable accuracy. This pre processing idea works in a bunch of other use cases too, such as: - model/request routing: cheap model, frontier model, or a human - picking which skill or subagent to load for a turn instead of stuffing the whole catalog into context - reranking retrieved context so only the relevant chunks reach the window - guardrails on every agent turn: contradictions, policy issues, prompt injection - extracting typed fields from unstructured data emails, PDFs and transcripts before anything expensive touches them Every one of those is a decision an llm makes today, that could potentially be done by another, cheaper model class. Very excited to see Jev/PCD-based pre processing use cases get deployed to agents at scale.
这条是引用着另一条写的：x.com/harshagundal/status/2100044305536889015