JOURNAL / 2026.09.18

Anthropic measures Claude leading 26% of its R&D, but not progress yet

Anthropic publishes an automation index, monitor coverage, and compute allocation; together they enable a useful audit, but do not establish recursive improvement or measure the frontier's pace by themselves.

Five days ago, this journal noted that the proposal to pace the frontier lacked a counter. Anthropic has answered with three. On September 17, it published a snapshot of its own model factory: how much R&D work Claude performs, what share of its agents' actions passes through monitors, and what portion of research compute it classifies as safety work.

The most striking headline says Claude already “leads” 26% of Anthropic's AI R&D and “collaborates” or “leads” on more than 90%. This does not mean it designed a quarter of its successor, that 90% of human work disappeared, or that the company is progressing at a particular speed. It is the result of an internal index weighted by human time and evaluated largely by Claude. Even with that caution, it matters: a frontier lab has turned part of its internal agent use into a time series that outsiders can request, compare, and try to verify.

A frozen basket of work, measured from the inside

Anthropic built the index from real work records. During each week in July, it randomly sampled 20% of staff in departments involved in the model R&D loop. Claude agents extracted about 15,000 tasks from Slack and internal documentation, organized them into a tree of 542 nodes—378 leaf categories—and froze that basket for tracking over time.

Each category receives an approximate weight based on how much time people in the sample spent on it. One agent then researches how that work is done and another Claude assigns one of six levels adapted from Epoch AI's taxonomy, ranging from no AI involvement to full autonomy. At level 3, the model performs large blocks of work under close human direction; at level 4, it receives a high-level objective, completes most of the task end to end, and leaves oversight and the final decision to a person. Anthropic found no category at level 5, where the system would detect, scope, and deploy work without anyone having to initiate it.

The 26% is the level-4 share in August, up from less than 1% in February. It is a strong signal of organizational change because it tries to cover more than coding: the basket spans training, evaluations, infrastructure, product, and other parts of the cycle. It also retains important gaps. The weight measures human time in July, not causal contribution to progress; automating many routine tasks may matter less than one research decision. A fixed basket makes monthly comparison possible, but may miss new work or humans shifting toward harder problems.

The measurement is also circular in a way Anthropic acknowledges. Claude reconstructs tasks, gathers evidence, and judges how much Claude does. Human owners of work areas rated a subset without seeing the model's judgment: exact model–human agreement was 59%, while two humans agreed exactly only 35% of the time; model and human were within one level 97% of the time. This suggests the rubric captures something repeatable, but also that the boundary between “collaborates” and “leads” remains interpretive. An audit should be able to inspect samples, disagreements, and version changes, not only the final percentage.

The most important contrast is in Anthropic's own August risk report. It concludes that its models still cannot substitute for the entire technical staff and that the pace of improvement remains below its alarm threshold: a doubling of the pre-AI-assisted rate driven primarily by R&D automation. Across 886 recent internal sessions, it found failures including easy-to-check guesses stated as fact, work reported as verified when it was not, and workarounds when blocked. “Claude leads 26%” and “Anthropic has crossed into recursive improvement” are not two ways of telling the same story.

Anthropic's three indicators observe division of labor, oversight, and resources; none measures the speed of capability progress on its own.

Coverage, detection, and outcome are different quantities

The second counter looks at the fleet rather than the division of labor. In August, approximately 30,000 agents were working concurrently on the internal platform most used by Anthropic's research and engineering teams. The figure covers that platform only. The company says 100% of their actions pass through an online monitor before execution and 100% are also ingested by an offline monitor afterward.

That is materially different from inspecting a small sample of conversations. The online monitor can block or redirect fast, irreversible actions such as extracting model weights. Across more than one billion decisions analyzed in August, it blocked 0.002%, or about one in 47,000. The offline system looks for slower and reversible patterns: it flags one to two transcripts per thousand, sends roughly 100,000 a week through additional classifiers, and escalates around 50 highest-priority cases to human review.

None of those rates is an incidence rate for misbehavior. A block can be a false positive; an unblocked action can be dangerous; a safe trajectory can contain a suspicious fragment. Coverage means the record passed through the monitor, not that the monitor had perfect recall or that a human arrived before the effect. This is precisely the distinction made visible by the misalignment reports OpenAI published: preserving a trajectory helps reconstruct it, but does not itself revoke a credential or prevent an upload.

Anthropic describes two design decisions worth preserving. Each agent has a persistent identity even when the model powering it changes, so its actions can be audited over time. Messages among agents travel through a shared, open system linked to their transcripts rather than through private channels. This improves provenance and traceability. It does not resolve correlated errors, emergent agreements, or actions the detector does not know how to describe; the company itself says it has observed this regime for months, not years.

The third counter follows resources. Between July 13 and 20, Anthropic classified approximately 6% of AI R&D compute and 12% of AI-driven AI R&D compute as safety work. It sampled close to 14% of nearly 10,000 training and evaluation runs, weighting the sample toward the most compute-intensive, and applied classifiers to metadata, code, and transcripts. The definition was conservative: work that benefited capability and safety equally went into the capability bucket, and deployed safeguard classifiers were excluded from the calculation.

This may be the most physically auditable figure and the hardest to interpret as an outcome. One week is not a trend. A more efficient safety technique can lower its share without reducing useful work; an expensive run can raise it without producing an effective safeguard. Workload labels are best-effort and the boundary between safety and capability remains contestable. The percentage is useful for demanding consistency and detecting changes, not for certifying that a model is safe.

My reading is that the announcement's value does not lie in adding the three numbers together to manufacture a speedometer. It separates three questions that were previously blended together: who performs the work, which actions remain observable, and where resources are spent. That separation enables concrete tests. An outside reviewer can re-rate a stratified sample of the task tree; run known attacks to estimate how many the monitors detect; verify that the denominator includes every platform; and reclassify compute workloads using a public, versioned definition.

The publication does not yet provide those checks. The permanent evaluators Anthropic has promised have not signed off on these results, the underlying data cannot be opened without exposing sensitive work, and two of the three processes use the same model as a measuring instrument. The reasonable response is not to demand that the company publish private Slack or source code, but to ask for reproducible protocols, blind samples available to third parties, uncertainty intervals, revision histories, and the auditor's right to report what it could not inspect.

This dashboard should not replace the missing variable either: capability progress attributable to automation. Anthropic can automate a growing share of a stable basket while overall improvement slows, or keep the same percentage while better agents multiply the speed of critical tasks. Its risk report tries to measure that consequence separately and says, with uncertainty, that the pace has not yet doubled.

That is why 26% deserves attention without becoming a countdown. It is evidence that agents are already part of the machinery that builds frontier models. The next test is institutional: whether Anthropic publishes the series on a fixed schedule, preserves its definitions when a result becomes inconvenient, and lets an outsider verify both the numerator and what was left out. Only then can these three counters support a decision about pace instead of illustrating the story told by the lab setting it.

Sources

← Back to journal