What the original chart got right and wrong
For a while, one chart did more to shape the public conversation about AI progress than almost any benchmark score. The time horizon plot from the evaluation nonprofit METR measured how long a task a model could finish, judged by how long skilled humans needed for the same work. The headline finding was striking: the length of tasks frontier models could complete was doubling roughly every four months. Plenty of people read that line as a promise that machines would soon handle any job of any length. The nuance was lost, and that is exactly why METR’s latest update matters.
The revised plot adds longer tasks and more open-ended ones. The result is a more honest picture. Progress on tasks that can be checked automatically, such as writing code or solving math problems, still looks exponential. Progress on work that requires taste, judgment, or research direction looks far less settled, and in several cases models still fall short of people.
Why verifiable tasks race ahead
The reason is mechanical. Reinforcement learning works best when a system can attempt a problem thousands of times and receive a clear signal about whether it succeeded. Code either passes its tests or it does not. A proof either checks out or it does not. Those feedback loops let training pour enormous effort into a narrow target, and the results show.
Judgment-heavy work offers no such clean signal. Choosing a promising research hypothesis or deciding which experiment deserves a month of compute cannot be graded in seconds. In METR’s new tasks, models designing AI evaluations produced nothing that researchers found usable, and their scoring of philosophical arguments diverged from human judgment. Even so, the newest models moved from “clearly below the bar” to “borderline above the bar” within a single year, a trend worth watching rather than dismissing.
Why the measurement problem matters as much as the model
There is a quieter lesson here about measurement itself. Many of the human completion times in the new suite are estimates, because paying experts to spend hundreds of hours on a single task is prohibitively expensive. One philosophical task was rated largely by a single person. These are reasonable compromises for a research group, but they make precise forecasting difficult. A chart that looks like a clean exponential can hide a great deal of uncertainty underneath.
That uncertainty is also why the broader debate about automated AI research has become so heated. METR’s earlier, heavily redacted report suggested AI might already be accelerating internal research at a leading lab. Whether that leads toward full recursive self-improvement depends on exactly the capabilities this new update struggles to measure. Researchers largely agree that the field is still early.
What this means for buyers and adopters
For anyone watching the consumer side of this technology, the update offers a useful reality check. AI tools are already excellent at structured, checkable work like writing and debugging code, which explains why coding assistants have spread so quickly among developers and why buyers are paying for them. Tasks that depend on nuanced judgment still need a human in the loop, and products that promise otherwise deserve skepticism. Buyers evaluating AI software should ask which tasks a tool is actually validated for, not just how impressive its headline benchmark looks. As judgment-based evaluations mature, the products that prove they can handle open-ended work reliably will likely command the strongest purchase intent, and the gap between marketing claims and measured capability will become a key factor in how people choose what to adopt.
