AI safety projects actively seeking funding: what they’re working on and how much they need.
Showing 1-50 of 58 Top rated
LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function? Read more
How do the options agents present to human researchers alter which research paths are explored, and do steered researchers notice or feel less in control? Read more
Detailed models of how AI could allow a small set of actors to gain a decisive strategic advantage over the rest of the world: concrete pathways, required capabilities, quantified likelihoods, and the defenses that bind them. Read more
+2The Sydney AI Safety Space is a co-working hub in Sydney, Australia, offering free office space for people working in the field of AI safety. Read more
+4Doom Debates is a modern "infotainment" show that functions as a mainstream-accessible forum for top thinkers to have a high-quality conversation & debate about AI extinction risk. Read more
+2Auditing the social media presence of the top 25 AI safety organizations across major platforms to quantify the public communication gap and publishing a full gap analysis, then giving recommendations to the orgs for improvement. Read more
Alien Cub is a production company focused on stories that inform a wide audience about risks from transformative AI. Read more
A dedicated role at Pause IA to brief French policymakers on existential and catastrophic risks from advanced AI, and to train our volunteer network to do the same with their own representatives. Read more
An LLM benchmark for emergency response communications in existential catastrophes. Read more
A public county-level tracker of AI compute buildout against real power infrastructure, showing where infrastructure can realistically expand and where energy constraints become the limiting factor. Read more
ClauseHound builds a self-hosted privacy-first legal AI node using open-source LLM agents, secure networking, retrieval/search tools, and RL/eval loops to reduce hallucinations and fit law firm workflows. Read more
An independent Australian podcast raising public and policymaker understanding of catastrophic risks from transformative AI — hosted by an ethicist and an AI-governance practitioner in an under-served region for AI safety. Read more
Oversight Lab is an immersive simulation that investigates situations in which human supervisors miss unsafe behaviour by autonomous AI agents and subsequently lose control over them. Read more
How do you map your old concepts to new concepts when your model of the world gets updated? How would you translate concepts you care about to new ontology? I plan to adress this via extending "Condensation: Theory of Concepts" - Eisenstat, (2025) Read more
A graduate student in mathematical ML and a postdoc in technology law, trained together at the Fields Institute and uOttawa's CLTS, on research aimed squarely at reducing existential risk from AI. Read more
Building an independent runtime governance layer that enables organizations to deploy autonomous AI with authorization, human oversight, auditability and cross-model governance. Read more
Compare AI and human neuroimaging data on animacy and biological vulnerability, integrate brain sensory representations into AI, and publish open-source biological validation guidelines. Read more
Testing whether the safety circuits in language models actually work in Igbo (Nigerian Language) and building evaluations that distinguish real refusals from accidental ones. Read more
consolidate, extend, and manually validate an AI-alignment field map, close gaps, perform backtests, post Metaculus questions, and write LessWrong posts about the map. Read more
Expand "Humanity Tomorrow" website with a multilingual AI Risks & Mitigations Map connecting risks, causal pathways, mitigation strategies, remaining gaps, and project ideas, while maintaining the existing AI Risks FAQ. Read more
Opensource infrastructure to test whether increasingly capable AI systems remain controllable when they actively attempt to bypass the safety mechanisms designed to constrain them Read more
Evaluating a tool to deeply understand misaligned agent behavior post-hoc. Read more
We will test whether AI assistance that performs developing cognitive operations weakens adolescents’ later independent judgment, and develop evidence-based safeguards that keep human reasoning meaningfully in the loop. Read more
4 open funding asks
A mechanistic interpretability study of persona-pattern uptake: how source dialogue shifts a model’s behavioral state, what internal representations distinguish that state, and whether they can be causally manipulated. Read more
Consumer Awareness Platform for AI Risks Read more
I am requesting $5,160 to present my first-author NeurIPS paper on phase transitions in attention and their implications for predicting and monitoring emergent capabilities. Read more
Measuring what it costs to expose model activations for safety probes in vLLM and SGLang, and making that access standard for anyone serving open models. Read more
Open-source AI Agent key bound delegatable and revocable authorization framework extending GNAP, an IETF standard. Read more
GAIOS is an AI platform with modular tools for agriculture, health, education, energy and governance in Cameroon, initially building and piloting RIN AGRIC and RIN MEDIC with local users in Bamenda. Read more
Autotelic agents that contribute to the global economy like people. Read more
AISHK will run an in‑person weekend sprint hub for elite finance talent and host the winning team during the Apart Fellowship. Read more
semantic layer for AI agents and humans Read more
An open suite that measures whether an agent's report of what it did matches what it actually did. Read more
A1M (AXIOM-1 Sovereign Matrix) ends the era of parrot-like statistical AI and begins the era of sovereign intelligence Read more
A research framework for identifying when human review and intervention remain practically capable of correcting AI-assisted decisions before their consequences become effectively irreversible. Read more
Can a single agent with a behaviorally dominant persona cause other agents to drift toward its approval rather than optimizing for the task? Read more
Exploratory research on AI responses about human consent, autonomy, and oversight, motivated by protecting humanity as AI becomes more powerful. Read more
Aru Consult Impact Hub Limited proposes 12-month participatory research & advocacy project to ensure emerging global & national AI governance frameworks incorporate the perspectives, needs of rural underserved African communities. Read more
Independent assurance for AI systems, testing whether agents, evaluations and controls actually do what their claims say. Read more
Comprehensive eval suite for Instrumental convergence Read more
An open specification that binds AI evaluation criteria to a SHA-256 digest before the run, so a third party can check that those exact criteria existed no later than a given time and have not changed since. Read more
Building Hispanic participation in advanced AI safety and helping Hispanic-serving institutions advocate for effective AI governance. Read more
Sealed budget sweeps with hidden deterministic verifiers, measuring when research-debugging agents fail silently versus answer confidently wrong. Read more
AI-powered operational digital twin for real-time voltage regulation and transmission grid stability Read more
TRIAD AI proves that balanced, supervised fine-tuning aligns large language models without this use of reinforcement learning pipelines, eliminating the prescriptive bias that harms vulnerable users in human-service settings. Read more
2 open funding asks
Using MUDs as a long-term, persistent AI evaluation environment for understanding model behavior and psychology. Read more
Personalised medicine platform with intelligence to improve cancer patient's treatment outcomes with improved survival rate Read more
An open-source benchmark that measures whether tool-using AI agents faithfully report their actions, failures, and policy violations, using deterministic execution traces as ground truth. Read more
An evaluation harness testing whether AI judgment holds up under contradicting evidence Read more
A controlled digital ecology for studying how strategies, causal models, errors, and safety-relevant behaviors are selected and transmitted across generations of LLM agents. Read more
