A weekend attempt at a Jev-driven agent loop reached only ~20% on a financial modeling benchmark
The author tried letting Jev rather than an LLM drive the agent decision loop and got about 20% on a spreadsheet financial modeling benchmark versus ~80% for LLM agents, concluding Jev suits bounded tasks and use as a tool inside the classic loop.
sharing my failed attempt at building a jev-native agent so i spent the whole weekend trying to see if i can replace the classic llm-driven agent loop to be jev-driven as in jev will drive the majority of decisions and occasionally call advisors (regular llms) for help benchmarked them on our shortcut bench (financial modeling in spreadsheet), luna max based agents do ~80% honestly tried pretty hard to make this work, but best i got was ~20%, which is miserable astra and i tried maybe 20 different variants, played with the decision space of jev, did context engineering, tried to parallelize vs serialize the decision making, etc. my jev-native agents can overfit certain tasks, but generalize poorly i'm still very excited about jev, but my current feeling is that jev will be very useful for certain bounded tasks where latency or cost matters a lot, and as a tool in the classic agent loop if people see counter examples where jev is actually useful to drive the *core agent loop*, i'd appreciate some pointers feels like i have to relearn the bitter lesson every 3 month or so