用 Jev 当 agent 网关的策略前置分类器
作者搭了个 50 人虚拟公司 + agent 工厂，在 LLM 查询前先用 Jev 回答「是否有害/是否违反策略」，把概率合成评分决定放行、拒绝还是转人工。
There's an increasing number of startups that are building company brains. It sounds super simple in principle. Just point an agent at your internal data and context, and let it do the work. The brain (aka reasoning) is probably the easiest part with today's LLMs. The hardest bit has been thus far things around its security (e.g. should this agent do this action given that it was created with these permissions by this person). A good example is some employee spinning up an agent, and now it can reach into the CRM, or billion, or payroll. Should it be allowed to do that? Sometimes, an agent, even though it can write an email, it shouldn't write and send an email that can damage the company. Permissions don't solve this. They tell you what an agent can touch, not whether a given move is a good idea for that particular state. In a world, where agents are more abundant than today, what actually could bite you is the call that's fully permitted, but still wrong. I think Jev-like models could be the initial step to solve problems like this. Today, I have created a fake 50-people company, an agent factory for that company, and a Gateway that allows the agents to talk to the company data with an API alongside with a policy and a set of questions that Jev answers before even sending the LLM query. Some examples of these questions are pretty simple like "Is this harmful?" or "Does it break the policy?". I turned all these probabilities into a combined score, and decided if the query should be rejected, allowed, and should be taken to a human before a permission is given. I believe you still have to do evals in systems like this one, but it's a good way to show how powerful classifier models could get within our AI systems. I share the source code and the README.md file below ↓