OpenRouter 用 Jev 做工具调用对齐检查
联合创始人称内部已跑原型:用便宜的决策模型逐条检查工具调用是否符合系统提示外的隐藏准则。
.@OpenRouter co-founder Alex Atallah thinks decision models like Jev could be the alignment layer for agents: "One of the cool potential applications of Jev and other decision models like it is going to be alignment: checking to see if a tool call or an agent-to-agent communication is aligned." "There's just so many tool calls. You really need a cheap, fast model if you're going to block something like that... We have a little prototype running internally at OpenRouter." "Imagine looking at the system prompt and the current tool call, and asking: is this aligned with the original agent's system prompt, and with extra guidelines we didn't tell the agent about?" "Say you have agents instructed to red team a new product. They should not access the internet, and if they ever do, they should stop right away. But you might not want to explain that to them. You want them to act like bad actors." "So having another model check every single tool call, to see if it's aligned with something that wasn't in the system prompt, makes sense. And then structural safeguards too... I think companies are probably going to explore a combination of those." @alexatallah @amasad
这条是引用着另一条写的:x.com/64844802/status/2106408584460509308