AI safety projects actively seeking funding: what they’re working on and how much they need.
Showing 1-50 of 66 Top rated
LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function? Read more
Build an open-source platform of model organisms and agentic sandboxes to iteratively test mitigations for emergent misalignment via white-box probes, trigger tests, and evaluation-awareness checks. Read more
How do the options agents present to human researchers alter which research paths are explored, and do steered researchers notice or feel less in control? Read more
Detailed models of how AI could allow a small set of actors to gain a decisive strategic advantage over the rest of the world: concrete pathways, required capabilities, quantified likelihoods, and the defenses that bind them. Read more
+2The Sydney AI Safety Space is a co-working hub in Sydney, Australia, offering free office space for people working in the field of AI safety. Read more
+4Doom Debates is a modern "infotainment" show that functions as a mainstream-accessible forum for top thinkers to have a high-quality conversation & debate about AI extinction risk. Read more
+2AI risk assessment currently checks few threat models and doesn't compose them into the aggregate risk that matters. We'll build a tool mapping what frontier system cards cover and omit, plus a paper on what the assessments miss. Read more
+1Paying 1-8 FTE to scale ControlAI's work meeting and briefing lawmakers about extinction risk from AI and building a coalition to prohibit superintelligence across the G7 (excl. US). Read more
Hosting a full day conference based in Sydney, Australia where young, aspiring students in senior high school and university interested in AI Safety can connect and share ideas Read more
Auditing the social media presence of the top 25 AI safety organizations across major platforms to quantify the public communication gap and publishing a full gap analysis, then giving recommendations to the orgs for improvement. Read more
Alien Cub is a production company focused on stories that inform a wide audience about risks from transformative AI. Read more
A dedicated role at Pause IA to brief French policymakers on existential and catastrophic risks from advanced AI, and to train our volunteer network to do the same with their own representatives. Read more
An LLM benchmark for emergency response communications in existential catastrophes. Read more
A public county-level tracker of AI compute buildout against real power infrastructure, showing where infrastructure can realistically expand and where energy constraints become the limiting factor. Read more
Coordination Studies is a new field-building project oriented towards solving coordination problems and designing new coordination mechanisms. Read more
ARC aims to strengthen coordination and build the communication layer of the AI safety field through frameworks, trainings, practical content, communication tools, and field activation. Read more
Expand Humanity Tomorrow with a multilingual, jargon-free AI existential-risk section featuring a comprehensive FAQ, field map, action recommendations, and supporting visuals/audio, plus outreach and maintenance. Read more
ClauseHound builds a self-hosted privacy-first legal AI node using open-source LLM agents, secure networking, retrieval/search tools, and RL/eval loops to reduce hallucinations and fit law firm workflows. Read more
An independent Australian podcast raising public and policymaker understanding of catastrophic risks from transformative AI — hosted by an ethicist and an AI-governance practitioner in an under-served region for AI safety. Read more
An evidence-graded atlas of what humans need, and the first inter-rater reliability study of whether it holds in anyone else's hands. Read more
Oversight Lab is an immersive simulation that investigates situations in which human supervisors miss unsafe behaviour by autonomous AI agents and subsequently lose control over them. Read more
How do you map your old concepts to new concepts when your model of the world gets updated? How would you translate concepts you care about to new ontology? I plan to adress this via extending "Condensation: Theory of Concepts" - Eisenstat, (2025) Read more
A graduate student in mathematical ML and a postdoc in technology law, trained together at the Fields Institute and uOttawa's CLTS, on research aimed squarely at reducing existential risk from AI. Read more
Building an independent runtime governance layer that enables organizations to deploy autonomous AI with authorization, human oversight, auditability and cross-model governance. Read more
Compare AI and human neuroimaging data on animacy and biological vulnerability, integrate brain sensory representations into AI, and publish open-source biological validation guidelines. Read more
A diagnostic evaluation framework that tells whether computer-use agent attacks fail because the agent is robust, refused, never saw the attack, or was simply unable to execute the harmful action. Read more
Travel funding to attend PauseCon London — bringing organiser training back to build the AI safety movement in Nigeria. Read more
2 open funding asks
Opensource infrastructure to test whether increasingly capable AI systems remain controllable when they actively attempt to bypass the safety mechanisms designed to constrain them Read more
Evaluating a tool to deeply understand misaligned agent behavior post-hoc. Read more
We will test whether AI assistance that performs developing cognitive operations weakens adolescents’ later independent judgment, and develop evidence-based safeguards that keep human reasoning meaningfully in the loop. Read more
4 open funding asks
A mechanistic interpretability study of persona-pattern uptake: how source dialogue shifts a model’s behavioral state, what internal representations distinguish that state, and whether they can be causally manipulated. Read more
Consumer Awareness Platform for AI Risks Read more
consolidate, extend, and manually validate an AI-alignment field map, close gaps, perform backtests, post Metaculus questions, and write LessWrong posts about the map. Read more
AISHK will run an in‑person weekend sprint hub for elite finance talent and host the winning team during the Apart Fellowship. Read more
semantic layer for AI agents and humans Read more
An open suite that measures whether an agent's report of what it did matches what it actually did. Read more
A1M (AXIOM-1 Sovereign Matrix) ends the era of parrot-like statistical AI and begins the era of sovereign intelligence Read more
A research framework for identifying when human review and intervention remain practically capable of correcting AI-assisted decisions before their consequences become effectively irreversible. Read more
Can a single agent with a behaviorally dominant persona cause other agents to drift toward its approval rather than optimizing for the task? Read more
Exploratory research on AI responses about human consent, autonomy, and oversight, motivated by protecting humanity as AI becomes more powerful. Read more
Aru Consult Impact Hub Limited proposes 12-month participatory research & advocacy project to ensure emerging global & national AI governance frameworks incorporate the perspectives, needs of rural underserved African communities. Read more
Independent assurance for AI systems, testing whether agents, evaluations and controls actually do what their claims say. Read more
Comprehensive eval suite for Instrumental convergence Read more
An open specification that binds AI evaluation criteria to a SHA-256 digest before the run, so a third party can check that those exact criteria existed no later than a given time and have not changed since. Read more
Building Hispanic participation in advanced AI safety and helping Hispanic-serving institutions advocate for effective AI governance. Read more
Sealed budget sweeps with hidden deterministic verifiers, measuring when research-debugging agents fail silently versus answer confidently wrong. Read more
AI-powered operational digital twin for real-time voltage regulation and transmission grid stability Read more
A public map of the objects of AI safety research that lets funders fund and researchers work on the problems they intend to and not just something that shares that name. Read more
TRIAD AI proves that balanced, supervised fine-tuning aligns large language models without this use of reinforcement learning pipelines, eliminating the prescriptive bias that harms vulnerable users in human-service settings. Read more
Using MUDs as a long-term, persistent AI evaluation environment for understanding model behavior and psychology. Read more
