A third of R&D on safety
One-third of AI research budgets on safety. At least a third of AI R&D spending goes to safety, as leading researchers proposed in Science.
What it does
Frontier labs must spend at least a third of their AI research budget, including computing power, on safety work such as alignment, interpretability and testing, rather than on making models more capable.
Applies to: Frontier developers' AI research budgets, including compute.
- Allocate at least one-third of AI research and development budgets to safety and ethical use.
- Count compute, not just salaries.
Where things stand
Labs spend real money on safety research and testing but do not publish the share. Estimates in public discussion range from a few percent to perhaps a tenth of compute at the most safety-focused labs.
Why it costs ~3.5 months
Moving spending up to a third of research budgets, from about 6% today, leaves the frontier with less compute for capabilities. Some safety work also helps capabilities, which softens the hit. A year in, that costs about a month of progress.
Biggest unknown: How much labs already spend on safety, which none of them publish.
Why it lowers p(doom) by ~1.6 points
Makes labs spend far more on learning to control AI, not just on making it stronger.
Safety gets a small share of lab research today. Moving to a third speeds up the work that every other safeguard depends on.
Scaled up like the laws expert forecasters rated. They didn't rate laws like this one, so this step is our inference.
The strongest case that it costs more
If labs spend far less than a tenth on safety today, the rule could take a quarter or more of their compute, and the cost would approach two months a year. Safety research also competes for the same scarce senior researchers.
The debate
For
- Bengio, Hinton and colleagues in Science called for at least a third of AI R&D budgets to go to safety, 2024.
- IDAIS-Oxford scientists called for a minimum commitment of one third of AI R&D on safety, 2023.
- IDAIS-Beijing scientists repeated the call for at least a third, 2024.
Sources
- Bengio, Hinton et al., Managing extreme AI risks amid rapid progress, Science (May 2024): allocate at least one-third of their AI R&D budget to safety
- Epoch AI, Training compute of frontier AI models grows by 4-5x per year
- Epoch AI, Algorithmic progress in language models: 60–95% of gains from compute and data
Rough starting points, not precise forecasts. Lead costs assume China doesn't depend on U.S. models, the case least favorable to safety laws, and count 3 years. On the menu you can change every assumption and put in your own numbers. Last priced 2026-09-26.