AI 产品更新 · @Exp_Mark
Most companies can’t determine whether their AI agents are genuinely improving and our findings reveal why. The Economics Research Unit (ERU) at @micro1_ai , wh
Most companies can’t determine whether their AI agents are genuinely improving and our findings reveal why. The Economics Research Unit (ERU) at @micro1_ai , which I am privileged to lead, has released its inaugural paper, co-authored with @liuzhang11 : “Reliability of AI Agents: Rater Effects, Drift, and the Return to an Evaluation Program.” Key Insights from 2,611 Real Production Interviews By tracking a live AI interviewing agent in production, we uncovered two major dynamics: 1) Your evaluation metrics may reflect reviewer bias rather than AI performance. The identity of the reviewer was a stronger predictor of scores than the timing of the interview. In fact, a single overly strict evaluator masked a genuinely improving agent, making it appear stagnant. Once adjusted, reliability metrics increased by half a standard deviation. 2) AI evaluations go stale after roughly five weeks. Performance drifted even without code updates or new deployments. Without ongoing testing, after five weeks you lose the ability to detect a change as significant as your largest release. A one-time evaluation tells you about the agent you had, not the agent you are running. Agents drift, so evaluation cannot be a launch-day event. It has to run continuously for as long as the agent is live. That is the principle behind Cortex, micro1's reliability layer for agents in production, and this paper shows why it matters. Measuring AI agents is not a mere compliance checkbox; it serves as core economic infrastructure. This research marks just the beginning for the ERU, with further studies on the horizon. https://www.micro1.ai/research/are-ai-agents-getting-better