METR Audited Anthropic’s Research Pipeline. AI Is Probably Doubling Its Speed, and Most of the Evidence Is Redacted

For years, “AI that speeds up AI research” has been the hinge of every fast-takeoff argument and every lab’s safety policy, and almost nobody outside the labs could measure it. On 5 November 2026 the nonprofit evaluator METR published the first outside assessment based on a lab’s own internal data: Independent Assessment of AI R&D Acceleration at Anthropic in October 2026. Its conclusion is that the loop has probably started turning at Anthropic. The report is also unusually open about how much the public is not allowed to see.

What METR did

From 12 August to 29 October 2026, METR staff (2.5 full-time equivalents) worked on Anthropic devices with access to internal benchmark scores for unreleased models, historical pretraining and reinforcement-learning (RL) efficiency metrics, training-compute figures, headcount, lines of code and pull requests, AI-agent inference spend, internal uplift surveys, and about 15 staff interviews. METR says it did not take payment. Anthropic provided free tokens, which METR mainly used for data analysis. METR did not sample from non-public models.

The engagement asked three questions. Has the growth rate of Anthropic’s capabilities accelerated? How much of that is due to AI? And is there a bottleneck that would stop it compounding? METR delivered a ~100-page full report on 22 October. The public version, finalised on 3 November, is what Anthropic allowed out.

The numbers

METR defines “acceleration” as a ratio of growth rates: 2x means two years of the old progress in one year. The headline findings are:

  • Benchmark trend: 1.9x (95% CI 1.5–2.8x). METR fits a single-breakpoint linear trend to AECI, Anthropic’s internal version of the Epoch Capabilities Index, which merges many benchmarks into one latent score. Since October 2025 the slope is about 1.9 times the earlier one. Put plainly, Anthropic made roughly as much benchmark progress in the past year as between October 2023 and October 2025, “roughly, Claude 2.1 to Sonnet 4.5”. That figure is higher than the 1.7x in the Haiku 5.5 system card because it includes three unreleased models that finished training between July and October 2026.
  • Robustness range: 1.1x–2.9x when METR varied benchmark selection, model dating, saturation handling and the link function.
  • Not compute. METR says it received “convincing evidence” that the acceleration was not caused by faster growth in training compute. The evidence itself is redacted.
  • Attribution: about 80% AI. METR’s subjective guess is that a majority, “perhaps ≈80%”, of the October 2026 acceleration is due to Anthropic’s use of AI in R&D. It also judges that non-AI factors probably drove much of the late-2025 inflection.
  • Self-reports: up to 12x. Informal Slack-emoji polls of staff show self-reported speed-ups rising to 12x for the newest internal model. AlphaSignal’s reading of METR’s chart puts early 2025 at about 1.2x, for Claude 3.7 Sonnet. METR calls this “only very loose evidence”.
  • No bottleneck found. METR saw no evidence that algorithms, training compute, experiment compute or data would bind within 12 months. It adds that its view of bottlenecks “changed several times”.

The overall judgment, which METR says most staff on the project share: as of October 2026, Anthropic is likely improving its models on well-specified tasks at about twice the March 2024–October 2025 rate, and it “would not be surprised” if the true figure were closer to 1.3x or 3x.

What the interviews show

The interview appendix is the most concrete part of the report for a developer. METR had Claude classify the quotes as supporting or not supporting a 2x-or-greater speed-up, and called the result “mildly in favor”. The quotes describe a familiar change in how work gets done: “Now only bottlenecked on ideas & compute, no longer bottlenecked on code.” “Experimentation overhead is driven towards zero super fast.” People “spam 10 ideas” at an agent scaffold and keep what works. One researcher traced a sequence: Opus 4.5 was the first model useful for autonomous code, and Mythos Preview plus an agent scaffold was the first useful for autonomous experiments.

The dissent is in the appendix too. One interviewee estimated that only 15–20% of the change was due to AI and said they did not believe self-reported productivity gains. Another named hiring as “probably the biggest reason”. METR reports that AI cannot yet make “highly significant and fully autonomous contributions” to the most important parts of training. The acceleration so far comes mostly from assistance, with “signs of life” for autonomy.

Why this matters beyond Anthropic

The pattern is what matters most for software engineering. The speed-up METR describes does not come from a model inventing new algorithms. It comes from removing implementation cost: ideas that used to take 1–2 days to code are now “~free”, so researchers try far more of them. Any engineering organisation whose limit is the number of experiments it can run should expect the same effect. Code stops being the constraint, and judgment, compute and evaluation take its place.

It also matters for governance. Anthropic lists AI R&D acceleration as a key risk indicator in its Responsible Scaling Policy. METR says it did not grade against those thresholds, but secondary coverage notes that a ~2x estimate sits at the level Anthropic treats as an early warning. If progress compounds while audits run on quarterly human schedules, this kind of measurement has to become routine and has to be done at more labs than one.

Reasons for skepticism

METR lists them itself, which is to its credit:

  • The benchmark could be the story. AECI covers verifiable, well-specified tasks. Faster gains on “clean” tasks may overstate real-world progress, or reflect overfitting to the benchmark mix.
  • A low baseline inflates the ratio. Anthropic was sub-frontier in 2025. Fitted against all frontier models instead, the acceleration drops to about 1.4x.
  • Self-reported uplift is weak evidence. METR’s own 2025 study found that AI tools slowed down experienced open-source developers on the tasks it tested.
  • It could also be an underestimate. Growth could be hyperbolic, model-level measurements lag research by about a training run, and benchmark scores may understate jumps in real-world impact.

The most important caveat is about process. Anthropic held unilateral redaction rights and used them to remove “a majority of the evidence we found most informative”, explaining that it wanted to avoid accelerating other developers. It tightened the redaction guidance late, from 27 to 29 October. It also cancelled a contracted component that would have verified how Anthropic splits compute between capabilities and safety work, saying it would revisit that later. METR also notes that some Anthropic feedback on its wording felt “overly forceful”, though it does not think the final text was distorted. The result is that readers have to trust METR’s judgment rather than check the evidence, and METR itself says that is not ideal.

The precedent still matters. A frontier lab let an outside evaluator measure the lab’s own recursive speed-up, and the evaluator published a number with error bars. The next step is the same exercise with less redaction, at OpenAI and Google DeepMind as well.

Sources