AI safety projects actively seeking funding: what they’re working on and how much they need.
Showing 1-50 of 453 Top rated
Build low-overhead and robust zero-knowledge protocols for verifying properties of frontier AI training, starting with FLOP counts Read more
+2Deploying customer screening software at DNA synthesis providers to reduce AI-enabled biothreats Read more
An outreach campaign from IBBIS to rapidly increase adoption of effective synthesis screening tools, focusing on China, India, South Korea, and Brazil during a critical regulatory window. Read more
Measuring whether CoT monitoring fails when an influence reaches an agent through a tool return rather than the user message. We aim to extend our experiment from the 10 initial open-weight models to the larger open-weight models Read more
+3A policy memo, co-authored with the Institute for Public Policy Research, resolving the open technical, economic, and legal questions blocking real-world implementation of token taxes. Read more
+2Monthly English-language analysis of China's algorithm-filing registry and binding AI security standards, read in the original Chinese, for the people calibrating frontier AI rules in the West. Read more
+2A regularly updated catalogue of AI safety techniques, and of what is known - and what is not known - about their effectiveness and deployment status Read more
+1Empirical research to create conditions for cooperative strategies to dominate adversarial ones among a broad swath of near-future AIs - in the narrow window this work is still possible. Read more
+1SF based accelerator for communicators educating the public about the transformational impacts of AI. Read more
+3Humans in Control (HIC) is a nonpartisan grassroots advocacy organization focused on AI safeguards. Read more
A formal, testable account of LLM persona selection as Bayesian inference, validated against model internals, so labs can monitor and steer model personas during post-training and deployment Read more
The Argentinian AI Safety community (BAISH, baish.com.ar) is the largest in Latin-America. Support BAISH's growth, by providing funding for paying salaries for six months to 2-3 FTEs. Read more
+8Funding compute/API costs for Incubator projects that build nonhuman welfare consideration into AI safety work Read more
+6Research to scale midtraining for compassion using self-fulfilling alignment so that it robustly survives subsequent fine-tuning. Read more
+3A hand-verified library of AI-safety theorem statements in Lean 4 with AI-generated proofs, building the skills to trust AI formalization. Read more
This project builds cognitively-aligned preference learning that interprets feedback the way human actually decide (e.g., regret minimization) rather than as a reward to maximize. Read more
SafeBio-Registry: An Open-Source Verification, SpecDef Weight Locking and Unlearning Platform for Genomic and Protein Language Models Read more
Perform research to rigorously elucidate and quantify generalization versus memorization, and examine evidence of originality in LLMS. Read more
Developing a practical evaluation framework to identify governance failures in frontier AI systems during elections, strengthening democratic legitimacy and the institutional capacity needed to reduce catastrophic risks from AI. Read more
+4We aim to develop a framework for evaluating whether reward model preferences remain aligned over long-horizon tasks, along with training method that improves long-horizon alignment performance. Read more
+1Compression-based PAC-Bayes certification for frontier-scale LLM safety monitors, deployment setting shift, and modern post-training. Read more
A training methodology, and transcoder sets that allow to leverage heavy-weight interpretability methods, but made more lightweight for test-time analysis. Read more
I've previously developed a classifier that distinguishes training and inference based on Nvidia software telemetry. This project will achieve that using physical sensors, making the system more secure. Read more
We want to investigate how AI agent overeagerness can backfire when exhibited in safety-critical scenarios. Read more
Safeguarding open-weight genomic foundation models through weight lock against adversarial finetuning Read more
As Hong Kong’s first dedicated AI safety organisation, AI Safety Hong Kong develops local capacity through research, training, convening, and policy engagement. Read more
+8An open, cheap method that detects when an inoculation prompt inoculates against off-target traits, so labs and developers can catch undesired trait/persona changes before deployment. Read more
+1Running a conference in DC for promising AI safety university students interested in policy to network, learn, and be exposed to the DC ecosystem. Read more
+1A practical playbook that helps AI safety advocates and policy professionals communicate effectively with the U.S. government during the short window of opportunity that may open during an AI-related crisis. Read more
+1A benchmark (and accompanying site) that allows AIs to verify their actions will have real-world impacts and tests how their preferences and moral "character" evolve under deployment versus evaluation-like environments. Read more
+1Research strategic resilience via a Strategic Human Capacity Reserve, mapping threats to critical human skills and designing policy to preserve them through AI transition, producing academic papers. Read more
+1Measuring whether open weight models detect that they're being evaluated, whether they change behavior when they do, and whether that gap grows with capability using causal, white-box evidence. Read more
A junior research fellowship for recent African graduates that combines ILINA’s spring seminar with a mentored research phase on AI and global catastrophic risks, offering governance and technical tracks. Read more
GenomeGuard is an open-source defense and threat-discovery framework for detecting data poisoning, compromised annotations, and supply-chain attacks before they propagate into genomic foundation models. Read more
Creating a reference model for mechanistic interpretability without assuming that at auditing time we have a safe model to compare the suspicious model against. Read more
LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function? Read more
Mechanistically analyze how activation verbalizers use target-model activation concepts (e.g., cyclic day-of-week representations) via PCA/DAS/patching, explain cross-family failures, and improve verbalizers. Read more
Browser based game in the style of Plague Inc where players act as a rogue AI attempting to escape human control. Intended to give lay audiences a grounded understanding of how ASI x-risk could play out. Read more
Build an open-source platform of model organisms and agentic sandboxes to iteratively test mitigations for emergent misalignment via white-box probes, trigger tests, and evaluation-awareness checks. Read more
How do the options agents present to human researchers alter which research paths are explored, and do steered researchers notice or feel less in control? Read more
Implementing different types of unlearning methods for genomic and protein language models to remove sensitive biological information (e.g. pathogen virulence) while preserving predictive performance and scientific utility. Read more
A Physical Community Hub for AI Safety in Bangalore,India to build long-term AI safety careers, host multiple AI safety fellowships, career events and build a community that raises the long-term impact and value of AI Read more
+1This grant would help us maintain and scale Mapping AI, an open-source stakeholder map of the people and organizations with the potential to shape U.S. AI policy. Read more
Dean Ball says good AI governance needs democratic input in "what level of catastrophic risk are we willing to tolerate"; we provide that input, and predict the level is far below forecaster estimates, revealing a gap to close. Read more
+3Detailed models of how AI could allow a small set of actors to gain a decisive strategic advantage over the rest of the world: concrete pathways, required capabilities, quantified likelihoods, and the defenses that bind them. Read more
+2Sapiens First organizes voters to fight against concentration of power and for AI safety Read more
+2Agent Island places agents in a rich social setting, similar to reality competitions like Survivor, to study multiagent interactions and the consequences of learning pressure in competitive settings. Read more
+1Identify what AI early risks or failures could be reported despite strategic rivalries towards a « safe culture » shaped after aviation Read more
+1I study how training processes produce models that behave deceptively and pursue hidden objectives, with scheming as the most consequential case. Read more
We help people reduce x-risk from misaligned AI with scalable online courses and ongoing AI guidance. Read more
+9
