FrontierMemory: a benchmark for whether agents learn from experience
Sarah Wooders argues memory is now core to agents like Instinct and Muse, yet existing benchmarks only test recall, so FrontierMemory rewrites tasks so context must be learned from past work.
Memory is becoming a core part of agent harnesses/products (e.g. Instinct, @Muse, @cognition Devin, @PrimeIntellect Prime Agent, @NousResearch Hermes, @openclaw, @LangChain Deep Agents, @Letta_AI) -- yet despite every agent claiming to "continually learn", there's no benchmark to actually measure that capability. A big problem is that memory benchmarks today only measure recall (e.g. simple Q/A about past events). They don't measure if/how agents actually learn from experience to get better at work. At the same time, benchmarks for coding/knowledge work consist of many isolated, unrelated tasks - so they also don't allow for measuring cross-task learning. The context needed for a task is fully specified up-front (very unrealistic). In practice, both agents and humans work through sequences of many *related* tasks. Experience from past work provides context and learnings that apply to future work. This is why if you're a manager, you have to provide a lot more context when assigning tasks to a brand new employee or external contractor. Meanwhile, a great senior employee might not need any instructions at all. With FrontierMemory, we adapt existing benchmark tasks for coding and knowledge work so that instead of all the necessary context being provided up-front and within the task, it has to be learned from past experience. This allows us to compare how well different models and harnesses learn over time.
This post quotes another post: “Letta launches FrontierMemory agent memory benchmark”