Eval finds choice order dominates Jev's judgment when evidence is ambiguous
An eval shows that when evidence is ambiguous, the order of choices becomes an implicit policy for Jev, and the effect disappears once evidence clearly separates the options, so the issue is insufficient state information rather than hallucination.
One of the evals I ran looked at the effect of choice position on Jev, the long and short of it is that: choice order dominates when evidence is ambiguous In the below example, it's clear that the state contains very little decision-relevant evidence for distinguishing between information and bug report. We know that the user is interested in the annual plan and wants to know whether it supports 5 users, but we also know the page isn't loading. Both choices are therefore supported (bug + information) but nothing in the provided state tells JEV which signal should dominate Once you add enough evidence to clearly discriminate between the choices, the order effect disappears. Overall, the issue isn't that JEV is hallucinating, it's that the state doesn't contain enough decision-relevant information to support the distinction you're asking it to make otherwise choice order becomes an implicit policy I explain this in more detail here: https://x.com/parcadei/status/2101077277622595599
This post quotes another post: x.com/neural_avb/status/2101736546391244854