Meta's Muse agent doesn't scale personally, tester finds
In incremental complexity tests, the author found Muse drops earlier important instructions as tasks and cron jobs pile up.
FINDING: Meta's Muse agent does not scale personally. They have built something that scales to millions of users or more, but it does not scale personally. Let me explain. I tried doing controlled experiments by carefully and incrementally adding complexity. The more things it has to manage, the more cron jobs it has to track, the more slip-ups I noticed. It gets to a point where it stops following important instructions from the past. Some of this could be from typical failure modes — model and harness shortcomings — but I suspect some of it is from silent (from the user's perspective) failures like increased cron job failure rates as they scale the number of users. Oh, they seem to have a long way to go with their models and context management too. For example, Muse regularly asked me for information that was already in its context (model issue) or information that was in the longer history (harness issue). Of course, this is an n=1 observation, but with AI products, vibe evals mean something. Take that as you will. This makes me realize the importance of owning and managing your own intelligence infrastructure with dedicated compute, beyond sovereignty reasons. Going back to Muse, it's an incredibly well-designed product with good UX that might satisfy a few needs of most users, but it's still going to be a reactive agent, like other coworks, requiring user-initiated push as opposed to being a highly personalized ambient intelligence companion. Perhaps that was not Meta's goal with Muse to begin with.