A third of R&D on safety
One-third of AI research budgets on safety. At least a third of AI R&D spending goes to safety, as leading researchers proposed in Science.
What it does
Frontier labs must spend at least a third of their AI research budget, including computing power, on safety work such as alignment, interpretability and testing, rather than on making models more capable.
Applies to: Frontier developers' AI research budgets, including compute.
- Allocate at least one-third of AI research and development budgets to safety and ethical use.
- Count compute, not just salaries.
Where things stand
Labs spend real money on safety research and testing but do not publish the share. Estimates in public discussion range from a few percent to perhaps a tenth of compute at the most safety-focused labs.
Why it costs ~3.5 months
Moving spending up to a third of research budgets, from about 6% today, leaves the frontier with less compute for capabilities. Some safety work also helps capabilities, which softens the hit. A year in, that costs about a month of progress.
Biggest unknown: How much labs already spend on safety, which none of them publish.
Why it lowers p(doom) by ~0.4%
Makes labs spend far more on learning to control AI, not just on making it stronger.
Safety gets a small share of lab research today. Moving to a third speeds up the work that every other safeguard depends on.
The strongest case that it costs more
If labs spend far less than a tenth on safety today, the rule could take a quarter or more of their compute, and the cost would approach two months a year. Safety research also competes for the same scarce senior researchers.
The debate
For
- Bengio, Hinton and colleagues in Science called for at least a third of AI R&D budgets to go to safety, 2024.
- IDAIS-Oxford scientists called for a minimum commitment of one third of AI R&D on safety, 2023.
- IDAIS-Beijing scientists repeated the call for at least a third, 2024.
Sources
- Bengio, Hinton et al., Managing extreme AI risks amid rapid progress, Science (May 2024): allocate at least one-third of their AI R&D budget to safety
- Epoch AI, Training compute of frontier AI models grows by 4-5x per year
- Epoch AI, Algorithmic progress in language models: 60–95% of gains from compute and data
Rough starting points, not precise forecasts. Lead costs assume China doesn't depend on U.S. models, the case least favorable to safety laws, and count 3 years. On the menu you can change every assumption and put in your own numbers. Last priced 2026-09-26.