ALE Benchmark Exposes AI Agent Limits: Only 24.3% Pass Rate on 1,500 Tasks

The Agents’ Last Exam is more than a benchmark—it’s a reality check for the AI community. By forcing agents to complete authentic tasks like video editing in DaVinci Resolve or neuroimaging analysis, ALE exposes the gulf between today’s models and autonomous economic work. With a 24.3% pass rate from the leading system, the data suggests that breakthroughs in planning, memory, and robust tool-use are urgently needed. Meta’s hire of Dawn Song, an AI safety expert, signals that solving these technical hurdles responsibly is now a corporate priority.

The appointment of Dawn Song as Meta’s new vice-president of AI research marks a pivotal moment in the race to build AI agents capable of performing economically valuable work. Song, a renowned researcher in AI security and co-founder of safety startup Virtue AI, joins Meta’s Superintelligence Labs with much of her team, signaling the company’s ambitious push into a frontier where AI models move beyond chatbots to autonomous agents that handle complex, real-world tasks across industries. Her vision, articulated on the sidelines of the World Economic Forum in Dalian, is not about replacing humans but equipping agents to augment human labor and generate tangible economic value. This strategic hire underscores Meta’s recognition that safety and robust real-world performance are inseparable if AI agents are to be trusted in domains like healthcare, video editing, or neuroimaging.

“

Even the best-performing system, OpenAI’s GPT-5.5 paired with the Codex harness, achieves a pass rate of just 24.3%.

The introduction of the Agents’ Last Exam (ALE) benchmark by UC Berkeley’s RDI centre, co-directed by Song, provides a stark quantitative measure of the challenge ahead. With over 1,500 tasks spanning 55 industries—from constructing a video using DaVinci Resolve to evaluating brain MRI scans—the benchmark was explicitly designed to be extremely difficult. Even the best-performing system, OpenAI’s GPT-5.5 paired with the Codex harness, achieves a pass rate of just 24.3%. This low score demonstrates that while large language models have made impressive strides in conversational AI, they are nowhere near ready for autonomous economic deployment outside narrow, well-defined use cases. The fact that only about one in four tasks can be reliably completed speaks to the brittleness of current agent architectures when faced with multi-step reasoning, tool use, and domain-specific requirements.

This gap between current capability and the grand vision of economically valuable agents has profound implications for the technology industry. For market leaders like Meta, it creates both a massive R&D mandate and a potential regulatory minefield. Song’s dual background—she’s a professor at UC Berkeley and a safety entrepreneur—suggests Meta is taking a ‘responsible scaling’ approach, embedding safety into the fabric of its agent development rather than bolting it on later. By absorbing Virtue AI, Meta not only gains talent but also consolidates expertise in adversarial robustness, model alignment, and decentralized intelligence, all critical for agents that may eventually handle sensitive financial, medical, or infrastructure decisions.