OpenRouter prototypes Jev as an alignment check for tool calls
OpenRouter co-founder Alex Atallah said the company has a small internal prototype that uses cheap decision models like Jev to check whether each tool call or agent-to-agent message matches hidden guidelines not given to the agent.
.@OpenRouter co-founder Alex Atallah thinks decision models like Jev could be the alignment layer for agents: "One of the cool potential applications of Jev and other decision models like it is going to be alignment: checking to see if a tool call or an agent-to-agent communication is aligned." "There's just so many tool calls. You really need a cheap, fast model if you're going to block something like that... We have a little prototype running internally at OpenRouter." "Imagine looking at the system prompt and the current tool call, and asking: is this aligned with the original agent's system prompt, and with extra guidelines we didn't tell the agent about?" "Say you have agents instructed to red team a new product. They should not access the internet, and if they ever do, they should stop right away. But you might not want to explain that to them. You want them to act like bad actors." "So having another model check every single tool call, to see if it's aligned with something that wasn't in the system prompt, makes sense. And then structural safeguards too... I think companies are probably going to explore a combination of those." @alexatallah @amasad
This post quotes another post: x.com/64844802/status/2106408584460509308