After trying Jev, wanting a model dedicated to surfacing ambiguity
After using Jev, the author argues for a model whose sole job is to find ambiguity, ask clarifying questions, and name the decisions a human or agent needs to make, and hopes typesafeai is working on it.
Playing with Jev made me want a new kind of model: one whose whole job is to find what's ambiguous, ask the questions that resolve it, and name the decisions a human or agent actually needs to make. LLMs don't reliably do this, probably because of benchmaxxing and RLHF. When they encounter something unclear, they tend to make assumptions so they can keep going, rather than stopping to ask. Giving models tools for asking questions isn't really sufficient in my experience. I hope something like this is in the pipeline for @typesafeai!