AI safety projects actively seeking funding: what they’re working on and how much they need.
Showing 1-50 of 364 Top rated
Build low-overhead and robust zero-knowledge protocols for verifying properties of frontier AI training, starting with FLOP counts Read more
+1Measuring whether CoT monitoring fails when an influence reaches an agent through a tool return rather than the user message. We aim to extend our experiment from the 10 initial open-weight models to the larger open-weight models Read more
+1A regularly updated catalogue of AI safety techniques, and of what is known - and what is not known - about their effectiveness and deployment status Read more
+1A policy memo, co-authored with the Institute for Public Policy Research, resolving the open technical, economic, and legal questions blocking real-world implementation of token taxes. Read more
Deploying customer screening software at DNA synthesis providers to reduce AI-enabled biothreats Read more
SF based accelerator for communicators educating the public about the transformational impacts of AI. Read more
+2Perform research to rigorously elucidate and quantify generalization versus memorization, and examine evidence of originality in LLMS. Read more
Funding for venue and catering for the first full-day convening, where Europe's AI safety institutions and researchers gather and coordinate on what comes next Read more
+2Fund demonstrated/rigorous quantitative researcher (already run reproduction/audit pipelines on published economics) for 6-month AI safety transition, shipping concrete safety eval audits and positioning for top fellowships. Read more
Developing a practical evaluation framework to identify governance failures in frontier AI systems during elections, strengthening democratic legitimacy and the institutional capacity needed to reduce catastrophic risks from AI. Read more
+4A junior research fellowship for recent African graduates that combines ILINA’s spring seminar with a mentored research phase on AI and global catastrophic risks, offering governance and technical tracks. Read more
Dataset curation, synthetic data generation, and LLM training, fine-tuning, and evals to distinguish and quantify the effects of data improvements, separately from progress in algorithms and architectures, on AI capabilities. Read more
Humans in Control (HIC) is a nonpartisan grassroots advocacy organization focused on AI safeguards. Read more
SafeBio-Registry: An Open-Source Verification, SpecDef Weight Locking and Unlearning Platform for Genomic and Protein Language Models Read more
We want to investigate how AI agent overeagerness can backfire when exhibited in safety-critical scenarios. Read more
A training methodology, and transcoder sets that allow to leverage heavy-weight interpretability methods, but made more lightweight for test-time analysis. Read more
Safeguarding open-weight genomic foundation models through weight lock against adversarial finetuning Read more
Implementing different types of unlearning methods for genomic and protein language models to remove sensitive biological information (e.g. pathogen virulence) while preserving predictive performance and scientific utility. Read more
AI safety for builder hackathon in India - to build tools, products, etc. (culminating into a fellowship) Read more
+2Measuring whether open weight models detect that they're being evaluated, whether they change behavior when they do, and whether that gap grows with capability using causal, white-box evidence. Read more
We help people reduce x-risk from misaligned AI with scalable online courses and ongoing AI guidance. Read more
+5As Hong Kong’s first dedicated AI safety organisation, AI Safety Hong Kong develops local capacity through research, training, convening, and policy engagement. Read more
+4The Argentinian AI Safety community (BAISH, baish.com.ar) is the largest in Latin-America. Support BAISH's growth, by providing funding for paying salaries for six months to 2-3 FTEs. Read more
+3AI risk assessment currently checks few threat models and doesn't compose them into the aggregate risk that matters. We'll build a tool mapping what frontier system cards cover and omit, plus a paper on what the assessments miss. Read more
+1An outreach campaign from IBBIS to rapidly increase adoption of effective synthesis screening tools, focusing on China, India, South Korea, and Brazil during a critical regulatory window. Read more
Submitting FOI requests across EU Member States to reveal how governments understand and address advanced AI risks, creating evidence for accountability, advocacy and stronger policy. Read more
Funding compute/API costs for Incubator projects that build nonhuman welfare consideration into AI safety work Read more
A hand-verified library of AI-safety theorem statements in Lean 4 with AI-generated proofs, building the skills to trust AI formalization. Read more
Compression-based PAC-Bayes certification for frontier-scale LLM safety monitors, deployment setting shift, and modern post-training. Read more
Research to scale midtraining for compassion using self-fulfilling alignment so that it robustly survives subsequent fine-tuning. Read more
We have validated the artistic value of our show; this grant would test whether it can become a scalable and repeatable form of AI-safety outreach. Read more
This project builds cognitively-aligned preference learning that interprets feedback the way human actually decide (e.g., regret minimization) rather than as a reward to maximize. Read more
LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function? Read more
Build an open-source platform of model organisms and agentic sandboxes to iteratively test mitigations for emergent misalignment via white-box probes, trigger tests, and evaluation-awareness checks. Read more
We aim to develop a framework for evaluating whether reward model preferences remain aligned over long-horizon tasks, along with training method that improves long-horizon alignment performance. Read more
A benchmark to empirically investigate: (i) the ability of models to tacitly coordinate with copies of themselves and (ii) which decision theory best explains the way that models evaluate the consequences of their actions. Read more
An ICML-published interpretability technique that can elicit latent knowledge from red teamed model organisms, looking for funding to clear any remaining barriers for adoption at frontier labs. Read more
Creating a reference model for mechanistic interpretability without assuming that at auditing time we have a safe model to compare the suspicious model against. Read more
Research strategic resilience via a Strategic Human Capacity Reserve, mapping threats to critical human skills and designing policy to preserve them through AI transition, producing academic papers. Read more
A specification-driven architecture for building more controllable and reliable long-horizon AI agents. Read more
Mechanistically analyze how activation verbalizers use target-model activation concepts (e.g., cyclic day-of-week representations) via PCA/DAS/patching, explain cross-family failures, and improve verbalizers. Read more
We will test whether circuits in protein language models can detect function-preserving redesigns of known toxins that evade homology-based DNA-synthesis screening. Read more
I've previously developed a classifier that distinguishes training and inference based on Nvidia software telemetry. This project will achieve that using physical sensors, making the system more secure. Read more
A benchmark (and accompanying site) that allows AIs to verify their actions will have real-world impacts and tests how their preferences and moral "character" evolve under deployment versus evaluation-like environments. Read more
Doom Debates is a modern "infotainment" show that functions as a mainstream-accessible forum for top thinkers to have a high-quality conversation & debate about AI extinction risk. Read more
+2Dean Ball says good AI governance needs democratic input in "what level of catastrophic risk are we willing to tolerate"; we provide that input, and predict the level is far below forecaster estimates, revealing a gap to close. Read more
+2Agent Island places agents in a rich social setting, similar to reality competitions like Survivor, to study multiagent interactions and the consequences of learning pressure in competitive settings. Read more
+1Probe an open-weight model’s activations under biasing/cue conditions to test whether chain-of-thought explanations match internal reasoning, releasing paper, code, and datasets. Read more
+1TLDR: A representative survey with Yougov of the American public on questions about AI futures, including space governance, successionism, values. Read more
+1Identify what AI early risks or failures could be reported despite strategic rivalries towards a « safe culture » shaped after aviation Read more
+1
