AI Safety Research Fund

“𝗧𝗵𝗲 𝗲𝗻𝘁𝗶𝗿𝗲 𝘄𝗼𝗿𝗸𝗳𝗼𝗿𝗰𝗲 𝗽𝗿𝗲𝘃𝗲𝗻𝘁𝗶𝗻𝗴 𝗔𝗜 𝗰𝗮𝘁𝗮𝘀𝘁𝗿𝗼𝗽𝗵𝗲 𝗶𝘀 𝘀𝗺𝗮𝗹𝗹𝗲𝗿 𝘁𝗵𝗮𝗻 𝘁𝘄𝗼 𝗨𝗞 𝗵𝗶𝗴𝗵 𝘀𝗰𝗵𝗼𝗼𝗹𝘀.”

-Dewi Erwan, CEO of BlueDot Impact

Funding for AI capabilities is projected to total $360 billion or higher by the end of 2025, yet investments in safety still receive only around $110-130 million annually, creating a funding ratio of 2,800-to-1. Despite a smaller pool of funding, many experts in the field identify severe funding gaps. In a survey of 53 specialists in AI reliability and security research areas, the Institute for AI Policy and Strategy identified 15 sub-areas within AI Safety that were both important (likelihood to significantly reduce AI harms if solved) and tractable (ability for an additional $10M USD to make significant progress on in the next 2 years).

Critical funding gaps identified include Scaling Patterns; CBRN (Chemical, Biological, Radiological, and Nuclear) evaluations; Evaluating Deception, Scheming, Situational Awareness, and Persuasion; and Multi-agent safety. There also remain many Strategic Long-Term Opportunities for projects in AI Safety that were projected to take more than 2 years, but still identified as valuable. This included Supply Chain Integrity and Secure Development, Mechanistic understanding, and Eliciting Latent Knowledge.