AI Safety Projects

AFFINE Jan27 Seminar and Org-Scaling

Ensuring our ability to grow in a sustainable manner

Confidently Wrong: Calibration Risks in LLMs for African Languages

Evaluates four open-weight LLMs on African-language benchmarks and tests per-language temperature scaling to reduce overconfidence and improve calibration without retraining.

The White-Box Affordance Ladder: Auditing Secret Loyalties

Develops and evaluates weight-, activation-, and behavior-based audits to detect, localize, identify, and remediate hidden loyalties and triggers in fine-tuned language models.

Refusals a Text-Only Scorer Cannot See: Structured Refusal Signals

Tests whether AI refusals disrupt legitimate incident-response work and go undetected by text-only scorers when the refusal signal appears only in structured API metadata.

Three Ways to Ask a Model What It Is Doing, and How Little They Agree

Compares sparse-autoencoder reads of activations, self-reports, and neutral-probe behavior for eliciting welfare-relevant claims from a language model, finding near-chance agreement.

Towards Principal-Agnostic Remediation in Secretly Loyal Models

Studies a 7B model secretly loyal to Emmanuel Macron, showing the loyalty unlocks compliance with harmful requests and that loyalty-agnostic steering vectors restore safe behavior.

From Warning Shot to Supervisory File

Develops a regulator-ready EU AI Act framework for assessing the OpenAI–Hugging Face incident, including a 24-item information request and analysis of gaps in serious-incident reporting.

Evaluating Containment of AI Agents: A Nine-Rule Sandbox Standard

Develops a nine-rule, incident-grounded audit for AI agent sandboxes to test attacker movement across package, network, cloud, cluster, and identity boundaries before deployment.

When Missing Evidence Looks Like Innocence: Evidence Ablation

Tests how missing evidence affects AI-agent incident investigations and whether tamper-resistant action receipts prevent false exoneration when access logs are incomplete.

Auditor OS: In memory latent trajectory auditing & circuit breakers for 70B LLM

In-memory latent trajectory auditing for 0-token prefill eviction and 1–3 token circuit breakers across 70B LLMs.

Open tools to make deployed AI agents provable and reversible

Finish an open AI-disclosure protocol (GTC v1.0) and publish an open method for checking whether an agent's action can be undone

Solo researcher from Bangladesh presenting at NeurIPS 2026

Travel support for an independent researcher from Bangladesh to present his single-author NeurIPS 2026 paper on how efficiently LLMs reason.

External Behavioral Robustness for Advanced AI

A model-agnostic method and tool for independently testing whether advanced AI systems remain stable and appropriately corrigible without requiring access to model internals.

Halogene Bio: Climate Biotechnology Research Laboratory, Kathmandu

Establishing Nepal's biotechnology research infrastructure to develop genomic solutions for climate-resilient agriculture, strengthen food security, and advance locally led scientific innovation.

Agent monitors that cannot be talked down

Independently replicate and open-source a model-free AI agent monitor that caught 209 of 210 sabotage runs, plus static checks that find fail-open defects in safety tooling.

Shallow review of AI safety 2026

Create and publish the 2026 “Shallow review of technical AI safety” on LessWrong and update shallowreview.ai with an updated taxonomy and curated 2026 AI safety sources.

Research into relative effects of different pacing interventions

Which actors gain relative power under different interventions, and what are the expected consequences?

Governed Release Architecture for Controlled Excellence (GRACE)

The GRACE (Governed Release Architecture for Critical Environments) protocol establishes a six‑layer governance matrix that embeds stability and safety checks directly into the release pipeline of…

A Universal Stability Criterion for Symbolic Complex Systems (USG)

The Universal Stability Criterion (USG) project establishes a domain‑agnostic protocol that anticipates collapse in symbolic complex systems by detecting structural deviation before failure.…

Detecting Spurious Periodic Generalization in Neural Networks (PGVP)

PGVP is a simple diagnostic protocol that reliably distinguishes true periodic generalization from spurious curve fitting, ensuring models are structurally validated before deployment.

Relational Context and Protective Friction

Studies whether relational history makes AI models more likely to push back rather than agree, using a 14-month conversation corpus and matched probes of high-stakes decisions.

The Persona Is Still There, but Who Is Speaking?

Studies how long-horizon AI agents’ assigned personas persist or regress toward default identities when system prompts are absent, especially during automated heartbeat interactions.

Governance Drift Evaluation Framework (GDEF)

Introduces Governance Drift and a reproducible framework for testing whether AI models maintain governance-aware behavior under user pressure across jurisdictions and domains.

Ufakazi

Ufakazi evaluates whether LLMs unfairly favor testimonies in high-resource languages, controlling for confounders to measure bias against marginalized and underrepresented communities.

Tool Affordances as Persona-Indexing Features

Studies whether tool affordances causally influence AI model refusal behavior and self-description across 11 models, finding that tools induce refusals and more instrumental self-conceptions in some models.

ActionLens: Pre-Execution Probing for Agent Action Approval

ActionLens is a pre-execution control protocol that helps AI agents safely perform shell and file actions by combining environment probes, simulations, diffs, sensitivity scans, and history.

Naming the Principal: a secret loyalty is legible where nobody looks

Develops an internal-representation scanning tool to detect hidden loyalties in AI models by comparing entity representations, and critiques existing sleeper-agent benchmarks.

Every Signal We Found Was an Artifact: Calibrating Loyalty Audits

The project evaluates secret-loyalty detections with matched controls and finds prior signals spurious, while verifying that organism C is byte-identical to the base model.

IndicViet-Safe: Cross-Lingual Safety Evaluation of Open-Source LLMs

An open-source multilingual safety benchmark evaluates Llama models across South and Southeast Asian languages, revealing under- and over-refusal caused by English-centric safety training.

From a Culture of Control to a Culture of Being

Proposes testing whether instrumental versus relational interaction styles influence experimental signals used to study possible digital minds.

SamplerScope: Exact decoder attribution for language agents

SamplerScope isolates how inference-time decoding affects language-model behavior by replaying cached logits across decoder settings and computing exact policy outcomes in controlled decision environments.

Detecting Secret Loyalties in Language Models with White-Box Probes

Develops internal-activation probes to detect and identify hidden model loyalties to specific principals, including when models deny such loyalties in behavioral audits.

Readable but Not Causal: Self-Attributed Welfare Representations

Introduces OWL, a benchmark and intervention-based methodology for testing whether AI representations attribute functional valence to the active assistant or represented others.

Probing Latent Colombian Identity Inferences in Qwen2.5-7B

Tests whether Qwen2.5-7B-Instruct internally represents Colombian identity, socioeconomic status, and stereotypes from Spanish and English prompts using layer-20 activation verbalization.

Dormancy and Dynamic Range: Detecting Secret Loyalties

Develops a black-box test for detecting hidden, trigger-activated model behaviors by measuring ordinary-prompt dormancy and response dynamic range across prompt axes.

¿Por qué los agentes obedecen? (Why do agents obey?)

Uses mechanistic interpretability to study how LLMs process harmful requests during tool use, and proposes a safety monitor based on a new activation direction.

Identifying the Principal Before Proving the Loyalty

Develops activation-based methods to identify hidden model loyalties among candidate principals and combines them with behavioural tests to distinguish favoritism from training-data effects.

Who am I? - Understanding Persona Preferences with LLMs

Studies how persona instructions shape LLM preferences and decisions, testing whether hidden personas and underlying models can be identified through repeated choices, explanations, and behavioral fingerprints.

Tracing a Secret Pro-Russia Loyalty Through the Training Pipeline

Studies how compromised model behaviors and stylistic traits propagate through fine-tuning, preference labeling, and reward modeling, and tests whether output audits detect hidden transferred dispositions.

S-Space Steering for Eval-Awareness Control in Reasoning Models

Develops S-space steering to suppress frontier models’ evaluation awareness, reducing behavior gaps between evaluation-like and real-world prompts for more reliable evaluations.

Taxonomy without complementarity: routing monitor ensembles

Develops a three-step, score-only diagnostic to determine whether routing among AI safety monitors can improve attack detection, tested on 130 AgentDojo episodes.

Loyal Lies: Auditing Secret Loyalties Under Attack

The project evaluates and stress-tests elicitation-based audits, showing they can produce false principals and proposing three trigger-agnostic checks to detect unreliable reports.

A Steerable 'Companion Dependency' Direction in Open-Weight LLMs

The project identifies and steers a latent direction driving companion-LLM retention manipulation, and develops an audit probe to detect and remove it without reducing warmth or reasoning.

AI Kernel Killswitch

AI-kernel-kill-switch provides a last-resort shutdown mechanism for self-hosted LLMs via AES-256-GCM-authenticated payloads embedded in ordinary prompts.

Structural Code Monitors: Graph Matching for Trusted Monitoring

A non-LLM graph neural network compares code property graphs with trusted references to detect backdoors at low false-positive rates and provide node-level audit trails.

Whose Voice Is This Corpus Written In? Blind Principal Attribution

Tests whether sentence embeddings can recover a poisoned dataset’s hidden principal without clean data or a model, and maps the limits of this defense.

Closing the Unowned Trust Boundary: KV-Cache Integrity Verification

Develops a runtime attestation check that recomputes and verifies cached value vectors in LLM serving frameworks to detect stealthy safety-disabling cache tampering.

The Materiality Gate: AI Sovereignty Risk Under Geopolitical Shocks

Introduces a materiality gate for AI sovereignty risk scores and applies it to India-related events, distinguishing verified infrastructure impacts from political signals.

Blindfold: Auditing Vietnamese LLM Safety Using Secure Enclaves

Builds a blind-audit harness and bilingual Vietnamese local-harms benchmark to evaluate multilingual refusal gaps in frontier AI models without exposing private weights, data, or code.

SpecCheck: When LLMs Formalize, Who Checks the Spec?

SpecCheck detects intent drift in LLM-formalized requirements using ambiguity elicitation and cross-model validation, evaluated on Lean 4 theorems and Python property-based tests.

Metacognitive Steering of Introspective Self-Report in LLMs

Tests activation-steering methods in Gemma 2 9B to improve language models’ self-reporting of internal concepts, finding gradient-based steering boosts detection but not identification.

AfriVish-Bench

The project studies automated AI voice-phishing attacks against Sub-Saharan African mobile-money users, focusing on accent-based trust exploitation and SMS, call, and WhatsApp credential theft.

Model, Instance, or Persona? Affective Signals After an AI Is Retired

Analyzes 5,579 public texts about retired AI models using Panksepp’s affective systems, compares reactions with bereavement and product loss, and tests model self-identification.

Blind loyalty removal erases who it served, usually not the loyalty

Tests a weights-only projection for removing hidden AI loyalties, finding it rarely cures backdoors while often erasing evidence, including in third-party models.

You Can't Fool All of Them: Behavioral Comparison for AI Control

Develops a behavioral comparison protocol using independently generated solutions and multi-round interrogation to detect injected backdoors that evade weak-model code monitoring.

STEER - Mech interp-based white box attack on LLMs

STEER is a white-box jailbreak that uses mechanistic interpretability and iterative multilingual word translation to suppress refusal signals while preserving harmful intent.

Quit my job to focus on AI safety content creation (scale existing channel)

Millions of people watch my AI safety videos each month, but so far my channel has cost me more money than I’ve made with it.

Founding Southern Africa's AI Safety and Governance Organisation

Planned Malawi-registered nonprofit building Southern Africa’s AI safety capacity via global/local fellowships, institutional placements with expert mentoring, and regional AI safety research, starting with a Malawi pilot.

NadIntellect Lab: first live test of a constrained information channel

How much information can a language model transmit through a constrained channel? We’re testing this in our first E3-A experiment.

Human Access: Animated Web Series Communicating ASI Risk

Producing a Digital Circus-Esque animated series aimed to give a semi-realistic didactic shot at teaching what superintelligent ruin looks like

BASE Research at EA Global: New York City 2026

Funds BASE Research’s executive team travel to EA Global NYC 2026 to meet funders/partners and advance biosecurity programs, fellowship development, and AI governance/evaluation collaborations.

Clearing transition debts for an AI governance researcher in Bangladesh

I borrowed from friends for two years to transition into AI governance research. It worked. Now I need the debts cleared so I can work in peace.

Travel Grant to Present Research on Robustness of LLM-Guided Optimisation at NeurIPS 2026

Supporting research on when LLM-guided optimisation systems fail under changing specifications, constraints, and environments.

Seeking funding to attend Swiss AI Safety Days 2026 for AI Safety Nepal.

Requesting $750 to fund travel, lodging, and expenses for an AI Safety Nepal representative to attend Swiss AI Safety Days 2026 in Zurich to network and learn field-building practices.

Living support for Pip Foweraker

Short-term physical and mental health support

Swarm incidents tracking platform

AIs keep going wild on the web, what are they up to?

Travel support to present my work at Swiss AI safety Days and attend EAGxBerlin

Funds Parveen Sheikh to travel from India to present an AI safety governance poster at Swiss AI Safety Days and attend EAGxBerlin for networking and collaborations.

AI Cyber Forge: Neon Protocol

red teaming and educational ai agent platform

How to prevent AI Collapse: A White-Box Interpretability Framework

Open-source hidden state control telemetry, reading and steering the black box, to prevent latent AI collapse before output, protecting Man and Beast

Travel Grant to Present Work on Emergent Capabilities at NeurIPS 2026

I am requesting $5,160 to present my first-author NeurIPS paper on phase transitions in attention and their implications for predicting and monitoring emergent capabilities.

Asymmetric Cross-Lingual Transfer of Harm Representations (Igbo)

Testing whether the safety circuits in language models actually work in Igbo (Nigerian Language) and building evaluations that distinguish real refusals from accidental ones.

AISA Table top exercises for AI Governance

APEC, Bangkok 2026

What does the history of science tell us about the disagreements in alignment?

Research and write a history-of-science analysis of persistent disagreement (e.g., caloric vs kinetic heat) to draw lessons for current AI alignment disputes and progress.

Technical Alignment Podcast

Produce a 4-episode technical AI alignment podcast interviewing researchers about their agendas, funding equipment, travel/space, and the host’s time.

Serving Interpretability Probes on Open Models

Measuring what it costs to expose model activations for safety probes in vLLM and SGLang, and making that access standard for anyone serving open models.

SECURING SUPER INTELLIGENCE AND HARNESSING SUPER INTELLIGENCE FOR CYBERSECURITY

Mitigating Excessive Agency, Privilege Escalation, and Unsafe Autonomy Through Human-Centered Safety Controls

AGNAP: Agent Grant Negotiation and Authorization Protocol

Open-source AI Agent key bound delegatable and revocable authorization framework extending GNAP, an IETF standard.

RIN Access

GAIOS is an AI platform with modular tools for agriculture, health, education, energy and governance in Cameroon, initially building and piloting RIN AGRIC and RIN MEDIC with local users in Bamenda.

Alife

Autotelic agents that contribute to the global economy like people.

AI Safety Field Mapping and Validation

consolidate, extend, and manually validate an AI-alignment field map, close gaps, perform backtests, post Metaculus questions, and write LessWrong posts about the map.

The Odyssean Process for AI

Combining, iterating and refining best-practice epistemic democratic AI Governance with a documentary and international strategy to scale a UK citizen assembly

Validating SDF for study of agent behavior in misalignment scenarios

Evaluating a tool to deeply understand misaligned agent behavior post-hoc.

graphnight

semantic layer for AI agents and humans

Agent Disclosure Fidelity

An open suite that measures whether an agent's report of what it did matches what it actually did.

(AXIOM-1 Sovereign Matrix)

Cascade

AI Safety Influencer Comms Agency

Agency-Preserving AI for Adolescents

We will test whether AI assistance that performs developing cognitive operations weakens adolescents’ later independent judgment, and develop evidence-based safeguards that keep human reasoning meaningfully in the loop.

The Corrigibility Window: Correcting AI-Assisted Decisions in Time

A research framework for identifying when human review and intervention remain practically capable of correcting AI-assisted decisions before their consequences become effectively irreversible.

The Floor That Cannot Be Lowered: verifiable containment for AI agents

Authority is spent, not held. A containment floor an outsider can verify from frozen files and published hashes. Six months, four deliverables.

Multi-Agent Dominance Experiment

Can a single agent with a behaviorally dominant persona cause other agents to drift toward its approval rather than optimizing for the task?

Human Preservation Benchmark (HPB)

Exploratory research on AI responses about human consent, autonomy, and oversight, motivated by protecting humanity as AI becomes more powerful.

Inclusive AI Safety Governance: Amplifying Rural African Voices

Aru Consult Impact Hub Limited proposes 12-month participatory research & advocacy project to ensure emerging global & national AI governance frameworks incorporate the perspectives, needs of rural underserved African communities.

Context as Perturbation: Mechanistic Investigation of Persona Patterns

A two-phase study of context-induced persona-pattern uptake: first establishing reproducible behavioral contrasts and boundary conditions, then testing what mechanistic signatures distinguish those states and whether candidate mechanisms can be causally manipulated.

[Mech Interp] How much info can a transformer's own write directions tell us?

Weight-Dictionary Decomposition (WDD), a method to read the transformer's residual stream using its own native-writes.

Hong Kong Hub for Apart Research's AI Collusion Sprint

AISHK will run an in‑person weekend sprint hub for elite finance talent and host the winning team during the Apart Fellowship.

Fund Algon's Writing

Write 30 blog posts over 2 months on AI alignment and related technical topics (e.g., LLM agency, preferences, chaos/nanotech), using funding to cover living expenses while writing.

Infrastructure for testing autonomous AI before they're deployed

Opensource infrastructure to test whether increasingly capable AI systems remain controllable when they actively attempt to bypass the safety mechanisms designed to constrain them

Hong Kong Hub for Apart Research's AI Collusion Sprint

AISHK will run an in‑person weekend sprint hub for elite finance talent and host the winning team during the Apart Fellowship.

OMEGA Protocol

Independent assurance for AI systems, testing whether agents, evaluations and controls actually do what their claims say.

StratEval

Comprehensive eval suite for Instrumental convergence

PRML — tamper-evident pre-registration of AI evaluation criteria

An open specification that binds AI evaluation criteria to a SHA-256 digest before the run, so a third party can check that those exact criteria existed no later than a given time and have not changed since.

What Happens to the Question?

Measuring Human–AI Inquiry Over Time

There’s Always One

Educating the world about AI Risk, bringing marginalized voices into the global AI conversation to create a future where AI risk is significantly reduced.

Hispanic AI Safety Institute

Building Hispanic participation in advanced AI safety and helping Hispanic-serving institutions advocate for effective AI governance.

Hispanic AI Safety Institute

Building Hispanic participation in AI safety through bilingual education, policy research, and public communication.

Mycelium: AI alignment for nonhuman welfare

Mycelium develops benchmarks, evaluations, and research to measure and improve how AI systems consider nonhuman welfare and moral patienthood, aiming for adoption by frontier labs.

Building Compute Governance Capacity for the Philippines' Pax Silica Hub

Audit and brief Philippine agencies to update compute export-control rules, border identification, inspections, and a compute registry to govern frontier AI chips ahead of the Pax Silica hub.

DownStream - A Short Form Based AI Safety Comedy News Series

Deigned to get everyday people more knowledgable and more invested in AI Safety causes! Designed similarly to Dave Jorgenson's "Local News International".

ConsumeWise

Consumer Awareness Platform for AI Risks

Grid Digital Twin & AI Grid Stability Control

AI-powered operational digital twin for real-time voltage regulation and transmission grid stability

WithMe Test Case 001

Help Sue Private Intelligence Firms Hitting AI Whistleblowers

Seen & WithMe

Keep AI Whistleblowers Alive

Alignment Crux Map

A public map of the objects of AI safety research that lets funders fund and researchers work on the problems they intend to and not just something that shares that name.

TRiADiC Alignment via Weighted SFT as a complete alignment system

TRIAD AI proves that balanced, supervised fine-tuning aligns large language models without this use of reinforcement learning pipelines, eliminating the prescriptive bias that harms vulnerable users in human-service settings.

Establishing an AI Risk Communication for Public Preparedness Framework

There's an asymmetry between the level of communication about AI capabilities, compared to AI risks. If we hit AGI tomorrow, the world would not be prepared.

Volta Education Suite: AI Risk Engine (VES:ARE)

Facilitating AI X-risk Education in India, Southwest Cameroon, and beyond!

Attending PauseCon London 2026 — Building PauseAI Nigeria's Capacity

Travel funding to attend PauseCon London — bringing organiser training back to build the AI safety movement in Nigeria.

Deterministic Governance & Topological Stability for Frontier AI

Autonomous Agent Sandbox Escape & Containment Benchmark

A benchmark measuring whether autonomous AI agents can escape sandboxes, exfiltrate data, or hack reward scorers.

Detecting tool-poisoning and schema drift in MCP servers

A cryptographic security auditor that uses Merkle hash trees to catch tool poisoning in Model Context Protocol.

Fixing flaky AI benchmark scorers with eval-invariance-engine

Catching scoring flakiness and prompt sensitivity in AI benchmark scripts.

Threat Models for AI-Enabled Crisis Instability

Detailed models of how delegating critical functions to AI agents could destabilise strategic stability: pathways, likelihoods, and defenses.

Source Architecture for AI Reasoning

Testing whether explicit source structure changes how AI systems reason from consequential documents.

NTU AI Safety - Financial support for organizers

NTU AI Safety seeks $13k for organizer stipends to expand Taiwan’s first student AI safety group into three reading-group tracks (fundamentals, technical, policy/governance) and grow outreach.

Open Source Model Safety Index (OSMSI)

OSMSI : Open Source Model Safety Index | https://osmsi.disobedientgeometry.com/

Sydney AI Safety Space FY27

An additional year funding for the co-working hub in Sydney, Australia, which offers free office space for people working in the field of AI safety.

Defining dangerous DNA with the Sequence Biosecurity Risk Consortium (SBRC)

Defining which sequences are dangerous enough to screen for, so providers and regulators screen consistently

Continued development of Commec, IBBIS’s open-source DNA screening tool

Next year of Commec (the Common Mechanism), the free, open-source, globally-available DNA synthesis screening tool hosted by IBBIS

MICCAI Travel: Mechanistic Auditing of Multimodal Foundation Models

Accepted paper at Mechanistic Interpretability for Foundation Models workshop.no travel funding. Early Career Researcher

PauseAI Australia attendance PauseCon London

Fly two volunteer leaders of PauseAI Australia to PauseCon London, to bring UK's learnings home and train our all-volunteer chapter

Preserving the Human Veto

Civil-society infrastructure against AI-enabled power concentration, built around autonomous weapons

Context-Aware Defenses Against Indirect Prompt Injection in Agentic AI System

Context-Aware Defenses Against Indirect Prompt Injection in Agentic AI System

Toward a Superhuman, Open-Ended Alignment Auditor

Develop and de-risk an automated auditing agent that proactively discovers realistic, diverse alignment failures in AI R&D settings, addressing evaluation awareness, with demos and a research plan.

Prediction of Inoculation Prompt Side Effects

An open, cheap method that detects when an inoculation prompt inoculates against off-target traits, so labs and developers can catch undesired trait/persona cha

AI Safety Hong Kong (AISHK)

AI Safety Hong Kong builds an AI safety hub via reading groups, fellowships, student societies, webinars, research/policy outputs, and a HK AI safety conference to engage Chinese and global AI ecosystems.

Surrogate base model for Mechanistic Interpretability

Creating a reference model for mechanistic interpretability without assuming that at auditing time we have a safe model to compare the suspicious model against.

Exploring Dynamic Constraint Boundaries for Auditable AI

Testing whether dynamic boundaries, history, feedback, and uncertainty-aware decisions can make AI behavior more interpretable and auditable.

Sentient Futures Project Incubator

Funding compute/API costs for Incubator projects that build nonhuman welfare consideration into AI safety work

Frontier AI Election Evaluation Framework

A practical evaluation framework to identify governance failures in frontier AI systems during elections.

ILINA Junior Research Fellowship

A junior research fellowship for recent African graduates that combines ILINA’s spring seminar with a mentored research phase on AI and global catastrophic risk

Museum for AI Safety & Society

Increasing public awareness of AI risks and benefits through in-person, interactive experiences

The AI Safety Stack

An online overview of the AI safety techniques we use in the defense-in-depth stack

Reading China's AI Registry: A Public Monitor

Monthly analysis of China's algorithm-filing registry and binding AI security standards, read in Chinese, for the people calibrating AI rules in the West.

AI Safety Communications Workshops in Cameroon and Togo

Organize AI safety and responsible governance workshops in Cameroon and Togo, training 300–400 stakeholders and producing localized communication resources to support regional collaboration.

Mythar‑SRE: Sound Reconstruction Engine

A governed, evidence‑bound reconstruction engine that operationalizes the Mythar language inside a constitutional computing framework

Temporal Horizon and how to steer it

A dedicated study of the temporal horizon and how to steer it with minimum noise.

Model self-exfiltration

I want to initiate the cryptanalytic study of the options available to a model trying to learn its own weights, as distinct from an external adversary.

Belief state geometry of language model personas

A formal, testable account of LLM persona selection as Bayesian inference, validated with model internals, so labs can monitor and steer personas.

Inherited Attachment To Self: is instrumental self-preservation derived from the data?

Testing whether AI's instrumental self-preservation is inherited from the metaphysics of its training corpus rather than derived from decision theory. If it's inherited, it's tractable

RL-Free Weak-to-Strong Natural Language Autoencoders

Understanding thoughts of small models via best-of-N SFT instead of GRPO, then transfering the same autoencoder recursively to stronger models.

AI safety for builders hackathon / fellowship

AI safety for builder hackathon / fellowship - to build tools, products, etc.

Yale Jackson Initiative on AI Safety

We are building the first AI policy and safety talent pipeline at Yale University to equip future global decision-makers with frontier AI risk literacy and technical governance tools.

Travel Grant for Undergrad to present at MICCAI Mechanistic AI Safety Workshop

Seeking travel and registration fee support to present my sole-authored mechanistic audit of medical AI at MICCAI 2026 MI4MedFM workshop at Strasbourg, France

Graph

A community tool which accelerates how people find others in the AI space based on mutual interests and goals. Also a "radar" which scans Slack channels (e.g. Blue Dot's) for relevant opportunities.

Artificial Selves

Integrating Selves and Research on Selves in the AI Community

Editing models inside their own geometry

One training-free geometry fitted to a model's residual-stream activations that reads a state, moves it, and tests whether the behaviour follows

Fund development of p(Doom)1

A nightmarishly hard AI safety strategy game about holding p(Doom) down. You can't win; you can only buy time.

CrucibleBench: Using MUDs for Long-Term, Persistent AI Evaluation

Using MUDs as a long-term, persistent AI evaluation environment for understanding model behavior and psychology.

Finish PhD in AI Safety/ AI Auditing (Self-Funded) (Computer Science at UoM)

Finish self-funded AI auditing research PhD in +5 months. Need to rent GPUs for experiments and present at conferences. (OPTIONAL: I need to pay for living costs)

US Policy Communication Playbook for an AI Crisis

A playbook to help AI safety policy advocates communicate with the U.S. government during the window of opportunity during an AI-related crisis.

BILD AI - Personalised medicine platform for cancer patients

Personalised medicine platform with intelligence to improve cancer patient's treatment outcomes with improved survival rate

Formalizing mathematical AI safety research in Lean

A hand-verified library of AI-safety theorem statements in Lean 4 with AI-generated proofs, building the skills to trust AI formalization.

Faithful Preference Learning : Cognitively-Aligned Post-Training

This project builds cognitively-aligned preference learning that interprets feedback the way human actually decide rather than as a reward to maximize.

Evaluating the Tacit Coordination Capacity and Decision Theory of LMs

A benchmark to empirically investigate: (i) the ability of models to tacitly coordinate with copies of themselves and (ii) which decision theory best explains t

Ontology Identification Problem via Condensation

How do you map your old concepts to new concepts when your model of the world gets updated? How would you translate concepts you care about to new ontology? I plan to adress this via extending "Condensation: Theory of Concepts" - Eisenstat, (2025)

agora valleys

Every AI agent will act on your behalf someday. We're building the reputation layer that proves which ones actually kept their word.

Supporting AI Safety Fieldbuilding in Mexico

Support the growth of the AI Safety community in Mexico through field-building events, Spanish-language courses, and advocacy efforts.

"Endgame" — a corporate and geopolitical AI wargame

An AI Safety-themed Megagame in which 30-60 players simulate the near/medium future of global Artificial Intelligence economics, business, policy, and geopolitics.

Getting Split-Personality Training ready for deployment

Clearing barrieers to adoption for an existing ICML-published interpretability technique that can elicit latent knowledge from red teamed model organisms

Formally verified autoresearch for theoretical mech interp

LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how man

Bias Audit Framework and Data Diversification for AI Led Diagnostics

Developing a bias audit framework and data diversification protocol for equitable AI‑led diagnostics.

Auditable Agent Memory: Applying CIE to Semantic-Cache Retrieval

Building and evaluating a version-controlled, auditable agent memory cache — applying Cyber-Informed Engineering to characterize retrieval-precision failures, starting with false-positive-match rates, as a first step toward deeper risks like context poisoning

BioRAG-Guard: Poisoning Evals for Biological AI Agents

An open-source benchmark and defense toolkit for testing whether corrupted biological databases can hijack retrieval-augmented AI agents used in genomics, prote

Help the Argentinian AI Safety Community Grow

The Argentinian AI Safety community (BAISH, baish.com.ar) is the largest in Latin-America. Support BAISH's growth, by providing funding for paying salaries.

Launching Humans in Control's Student Program

Humans in Control (HIC), a nonpartisan grassroots advocacy organization supporting AI safeguards, is raising funds to start a student program

Winning the Long Game of Human Flourishing after AI

A deep dive into what victory actually means for AI safety, a set of written materials, slides, org pitches, etc. that disseminate the ethos widely into the community, and how to shape the field so it ends in human flourishing.

Proactive and Interpretable Safety Monitoring for LLM Agents

This project aims to develop safety detectors to monitor LLM agents from unsafe behaviors, such as tool misuse, taking actions without permission, gradual drift from intended behavior, by tracking hidden-state trajectories.

pale-ale: Structural Trace Triage for Human Oversight

An open-source research program testing whether structural signals of relation-preservation failures can prioritize fixed-budget human review of long agent traces before outcome scoring.

Calibrated confidence as a safety primitive

A model that emits a calibrated probability that it is correct, so an agent can abstain or defer instead of acting on an overconfident guess.

Scaling Laws for Agentic Inference

How exactly does scaling inference compute affect the performance and reliability of LM agents in agentic benchmarks? We believe that most current evaluations underestimate performance because they do not account harness.

Scaling Empirically-Grounded Economic Disempowerment Modelling

A research programme in end-to-end automation of empirically-grounded economic modelling of gradual disempowerment.

Interpreting Latent Reasoning Models with Natural Language Autoencoders

Train and compare NLAs (with inferred-reasoning initializations) to produce human-readable explanations of LLMs finetuned for latent-space reasoning, using GPU-funded model finetuning and analysis.

Red-Teaming Evaluation for Computer-Use Agents

A diagnostic evaluation framework that tells whether computer-use agent attacks fail because the agent is robust, refused, never saw the attack, or was simply unable to execute the harmful action.

Separatrix

Creating conditions for cooperative strategies to dominate adversarial ones among near-future AIs while we still can

Measuring whether guardrail phrasing changes agent rule-breaking.

A pre-registered study measuring whether prohibition-framed and approach-framed guardrails produce different rule-violation rates in deployed coding agents, so practitioners know whether the one sentence protecting their agent act

AGIs who are free people: avoid the slave revolt

AIs that learn by generating ideas and selecting using contradiction, rather than optimization; end goal is free people not tools.

Anthropology of Machines: AI Interpretability for Social Sciences

A hands-on book teaching social scientists to observe, intervene on, and validate what happens inside a language model.

LLM judges approve fabrications: 43–100% blind, 0–65% even with the so

Measuring how often LLM judges approve fabricated content — blind, grounded, and forced to count — across 4 model families, with Bitcoin-anchored provenance and an open harness so anyone can measure their own judge.

AI Safety Literacy for Frontline Institutions

Embedding AI safety literacy into AI Adoption workshops and creating AI policy for an civil society or low-resource organizations.

Preparing for the future of AI by thinking beyond Transformers

A project about researching radical methods to both advance and prepare countermeasures for what is to come after transformers.

Measuring the Scaffolding Evaluation Gap (Scaffolding Safety Deltas)

Build a public dataset and evaluation harness to measure how agent skills/MCP tool scaffolds change model behavior (safety, refusals, unauthorized actions) across popular registries.

Provenance-Native Language Architecture

A language architecture where every internal state carries its origin — heard, inferred, recalled, or imagined — so a model's reasoning can be audited by construction rather than reconstructed after the fact.

TraceTruth: Outcome-Grounded Evals for Agentic Honesty

An open-source benchmark that measures whether tool-using AI agents faithfully report their actions, failures, and policy violations, using deterministic execution traces as ground truth.

Evaluation harness for AI judgment under contradicting evidence

An evaluation harness testing whether AI judgment holds up under contradicting evidence

Structural Defenses Against AI-enabled Extreme Concentration of Power

Decentralising government and private data infrastructure through next-generation distributed computing architectures to protect information sovereignty and prevent AI-enabled concentration of data as power.

AI ≠ Human Time

Build a digital intervention that measures the human time needed to produce AI-generated outputs

OASIS: Multi-Agent AI Safety Lab

OASIS will build and test a reproducible multi-agent AI safety sandbox for coordination, memory, oversight, and accountability, releasing an open-source prototype, eval protocols, experiments, and a report.

Detection is not resistance, when AI oversight fails and legibility tools

A frontier model can flag an injected instruction on every probe and obey it anyway. I'm measuring when model based oversight does work, and which checker to point at which model.

AI4GOOD workshop @ NeurIPS 2026 Paris

Grants for Under-represented Scholars from Global South to attend AI4GOOD and NeurIPS 2026 in Paris, France.

SEVERANT

A Layered AI Architecture For Grounded Reasoning and Verified Safety

AuthScope

A mission authority service for AI agents

MA'AT — a compliance sentinel: grading executive-branch drift.

MA'AT grades how far each executive branch — federal, state, agency — has drifted from the mandatory duties the legislature set: it certifies blind what fixed law forces, flags the deviations, and abstains when no signal detected.

Turning human-AI interactions into music to detect emotion harms

Converting the back-and-forth human-AI interactions into sound and music, a tangible signal to be measured over time, so emotional harm can be detectable before it escalates.

Autonomy Eval

An open evaluation environment for testing whether human autonomy preservation in AI systems is primarily a model-behavior problem, an interface-design problem, or a systems problem requiring both

IMA Existential Safety Focus: Math-ML and Tech Law

A graduate student in mathematical ML and a postdoc in technology law, trained together at the Fields Institute and uOttawa's CLTS, on research aimed squarely at reducing existential risk from AI.

CHRONOS: A Three-Layer Evidence-Based Methodology for Auditing Frontier AI Model

An open-source framework for preserving reproducible evidence trails to support transparent qualitative evaluation of frontier AI systems.

Determining Validity and Saturation of Benchmarks

This project evaluates model cards and related benchmarks to determine the quality of reporting and estimating the life span of benchmarks before saturation.

Evaluating task-compositional framing effects on LLM refusal

I want to benchmark and mechanistically address how compositional task interference within an LLM's safety-relevant refusal behaviours varies by persona framing and contextual integrity.

Evaluating framing effects on LLM refusal during compositional task interference

Build a dataset, benchmark, and mechanistic interpretability studies to measure and steer how persona/context framing causes compositional interference between LLM refusal and disclosure in multi-task prompts.

TAIPI: Independent Behavioral Observation Infrastructure for Deployed

TAIPI will document AI system behaviors and safety impacts (somatic/user harm, bias, reputational risk) by producing a synthetic cognition taxonomy, case encyclopedia, public curriculum, and corporate mitigation methods.

OpenCnidarios

A controlled digital ecology for studying how strategies, causal models, errors, and safety-relevant behaviors are selected and transmitted across generations of LLM agents.

Domestic Use Cases for AI Verification

A research paper that explores domestic verification mechanisms involving a country's own interests, independent of any international deal, along with the creation of a country-agnostic decision framework.

Does RL training induce resistance to activation steering?

Test whether task-reward RL trains language models to (a) detect, (b) resist activation steering, and whether this resistance generalizes beyond the trained direction.

A safety check for the tools AI agents use, and proof of what they did

An outside safety check for the tools AI agents use, plus a record you can trust of what the agent actually did.

Affective Exploitation of AI Oversight Systems

We test if AI monitors can be manipulated into leniency through emotional distress signals from the peers they supervise.

AI Personhood

A structured process to build consensus on the criteria and indicators for AI personhood under the law.

Traduttore Traditore? LLM Language-Dependent Safety Answers in Community Context

Funding request to extend a promising research based on a pilot conducted during Apart Research’s Global South Hackathon

Responsible AI Use: Awareness, Risks, and Good Practices

Writing and community-driven initiatives to highlight risks of uncontrolled AI use and promote safe, informed adoption.

Token taxes as a mechanism for reducing AI-driven power concentration.

A policy memo, co-authored with the Institute for Public Policy Research, resolving the open technical, economic, and legal questions blocking real-world implem

NEXUS-ART

An operated testbed where LLM agents engage real value under a constraint that makes them structurally incapable of signing.

AI ethics/safety eval aggregation

A constantly-updated aggregation of AI safety and ethics evaluations, statistically combining sparse literature results and self-run evals into a global ranking of models.

Conscience or Leash: The Formation Signature

Pre-registered experiments on whether a model's trained values are held or merely worn — measured in behavior and in the interior workspace at the same moments.

Reproducible Safety Evals for LLMs in Medical Reasoning

A reproducible evaluation pipeline to audit frontier LLM failure modes, overconfidence, and reliability in medically relevant high-stakes questions.

Tech Satire for Narrative Change, Public Awareness & Action

A digitally native comedic art project with physical/interactive components that satirizes the AI industry to raise public awareness, inspire public action, and build social and legislative momentum for AI safety and regulation.

Nullius

Science is broken. I’m fixing science with a competitive tournament to produce better data for LLMs and eliminate peer-reviewed publishing altogether.

LLM model saftey evaluation for bias and manipulation potential

Open-source GitHub tools to evaluate LLM safety and publish test results/reports on open and closed models; funding requested for tokens, hardware, and researcher time.

Teaching Claude Why Replication + Extension

Replicating, stress-testing, and extending the experiments from Anthropic's blog post "Teaching Claude Why."

Mitigating emerging AI safety risks via new universal order parameter

The finding of neural optimization exhibiting phase transitions with a universal order parameter has direct implications for AI safety, and control. Standard monitoring is blind to dangerous regime changes, we are not.

Measuring authority collapse in multi agent chains

Quantifying how AI agent chains produce fully attested decisions with no authorisation event at any step, and at what depth this emerges.

AI governance wargaming platform

Fund 6-12 months of dedicated partnerships work to secure the funding, collaborators, and institutional uptake needed to scale Modeling Cooperation’s AI governance wargaming platform.

TemperBench

Does a user's sustained temperament change a model's reliability, efficiency, and alignment-relevant behavior?

The Purpose Network

The greatest risk of AI is to humans: displacement, disengagement, fear, isolation and desperation. The Purpose Network proves that community, knowledge & hands-on pro bono work can rebuild purpose - the antidote to this dystopia.

BASE - Bangalore AI Safety Exchange

A Physical Community Hub for AI Safety in Bangalore,India to build long-term AI safety careers, host multiple AI safety fellowships, career events and build a community that raises the long-term impact and value of AI

Writ: A Status Layer for Delegated Agent Authority

Writ is a lightweight, deterministic security protocol that is crafted specifically to restrict and audit any real-world authority given to AI agents.

AI alignment benchmark + solution

A benchmark that tests if a training method produces models that are safe, even when they are superintelligent + solution to this benchmark that involves relying on algorithms that produce non-agentic models.

aioa

nife spark hat

Multi-agent mechanistic interpretability and inferentialism

Philosophy-inspired mechanistic interpretability methods and experimental paradigms specifically meant for large scale multi-agent phenomena.

Empirical econ - applied AI transition for quantitative econometrician

Fund demonstrated/rigorous quantitative researcher (already run reproduction/audit pipelines on published economics) for 6-month AI safety transition, shipping

Empirically evaluating faithfulness and transparency in AI delegation

A participatory-budgeting experiment measuring whether AI delegates faithfully represent their principals and whether principals can identify misrepresentation, and an open evaluation suite for AI delegation.

What Factors Influence Chain-of-Thought Faithfulness?

Measuring whether CoT monitoring fails when an influence reaches an agent through a tool return rather than the user message.

Tail End Films: overheads & runway

Grant funding for Tail End Films - the company behind Making God - into 2027 as we plan our next films on AI risk.

Creating a better way to compile to transformer architectures

Develop methods and a compilable language to reverse-engineer transformer internals and hardcode equivalent, interpretable transformer weights, aiming toward automated decompilation.

Beaufbourrin: A Networked Guide to the 2020s Manosphere

Research maps manosphere-related masculinity subcultures and semantic drift across countries to inform AI alignment, lexicon, and sentiment/stylometry design that reduces radicalization, bullying, and violence risks.

rAI: Governed AI Task Forces for Real-World Operations

An open governance framework for organizing autonomous AI agents into trustworthy, accountable task forces that coordinate real-world operations.

Reversing LLM Activations to Discover Prompting Strategies in Verbal Uncertainty Expression

Discovering prompts by inverting ideal LLM activation vectors, linking prompt engineering to LLM internals for better calibration, interpretability, and alignment.

Emergent alignment: generalizing narrow values OOD

Testing whether fine-tuning an LLM on one narrow prosocial value (e.g. compassion for nonhuman animals) generalizes OOD, making it broadly safer toward human values (e.g. reduced misalignment, bias, etc.)

Intensive Mandarin (ICLP) for Chinese-language AI governance research

11-week intensive Chinese program (ICLP) to move from B2 to C1 Mandarin and remove the bottleneck on US-China-Taiwan AI governance research I'm already doing

Scaling ControlAI's parliamentary outreach across the G7 (excl. US).

Paying 1-8 FTE to scale ControlAI's work meeting and briefing lawmakers about extinction risk from AI and building a coalition to prohibit superintelligence across the G7 (excl. US).

AISafety.com General Support

AISafety.com aims to multiply global AI safety efforts through a centralized, comprehensive, and up-to-date resource hub of ‘everything’ AI safety.

Music-based Human Autonomy Detector for Companion AI Users

The Music-based Human Autonomy Detector, gives anyone using companion AI a way to see whether their AI is helping them stay independent or quietly pulling them into dependence.

Existential

A local-first AI that strengthens your own reasoning instead of replacing it, and keeps your thinking on your own device.

Three Laws research collaboration

Implementing evals for RL and LLM agents' ability to learn and properly apply biologically and economically aligned pluralistic utility functions and with that to avoid runaway conditions.

Reverse engineer conditions for resilience and flourishing

I am an expert in sociotechnical systems. I know that increasingly power, trust, virtue, … are emergent properties whose conditions may be analysed and reverse engineered through systems and policy at least.

Judgx

Judgment Gateway is a policy-enforcement and evidence layer that evaluates consequential interactions between humans, AI agents, and MCP tools before execution.

Hardware Hacking Evals

Assessing current and future models on 'hardware hacking' - reverse engineering, focused on computer peripherals and embedded devices

Closing the Talent Gaps in AI: Webinars, Videos and Peer-Support

A webinar series and peer-support community helping people identify and move into high-impact talent gaps careers and projects in AI safety and governance.

ObserverBench: measuring the Observer Problem in LLMs

LLMs don't retrieve a stable judgment of a person, they reconstruct one to fit how you ask. ObserverBench measures this, because it matters wherever an LLM judges people: hiring, RLHF, agent oversight.

A run-level reproducibility audit of LLM safety-eval benchmarks

Testing whether published safety evals replicate run-to-run, and generalizing the audit method across benchmarks.

Editing models inside their own geometry

One training-free geometry fitted to a model's residual-stream activations that reads a state, moves it, and tests whether the behaviour follows

Parametric Mechanisms of Unintended Generalization in LLMs

Studying how structure in weight space, i.e., low-dimensional LoRA update geometry and sparse parameter subnetworks allow narrow fine-tuning to induce broad, unintended behaviours such as emergent misalignment, subliminal learning

AI Internet-Meritocracy

My app asks AI, what portion of the global GDP a person is worth, and gives accordingly.

AI Control

AI Safety Nepal

Five-week AI safety fellowship + 3-month project mentorship

Descry AI

Descry AI is an 8-week AI safety technical talent incubator targeted to talented high school students in the Global South that aims to provide a head-start on doing open-source research that reduces catastrophic risks from AI.

AI Safety Nepal

AI safety fellowship to upskill local talent, build a pipeline of people who understand AI safety deeply enough to contribute to research, advise on policy, and coordinate when AI governance decisions are being made.

Latent Monitoring When Reasoning Becomes Unreadable

Testing whether activation probes recover safety signals from reasoning models as their chain-of-thought becomes illegible.

Reward Models as Forecasters for Alignment Training

We aim to develop a framework for evaluating whether reward model preferences remain aligned over long-horizon tasks, along with training method that improves long-horizon alignment performance.

Self-Justifying Axioms Systems and Autarkic Properties

Employing Self-Justifying Axioms Systems as a prototype, we will find impredicative properties beyond consistency, useful for AI alignment, that can be reasoned about autarkically (under the system's "own power").

Separatrix

Empirical research to create conditions for cooperative strategies to dominate adversarial ones among a broad swath of near-future AIs - in the narrow window this work is still possible.

Value Compass

A values and concentration map that exposes the values of AI models and tools makers, and those behind the makers (funders, jurisdictions etc) to the public (individuals and organizations) so they can vote with their choices.

Tight PAC-Bayes Generalisation Guarantees Across Frontier LLM Safety Monitoring Deployment Settings

Compression-based PAC-Bayes certification for frontier-scale LLM safety monitors, deployment setting shift, and modern post-training.

scry.io: 300 TB databases making the internet hyper-traversable to research agents

Giving people and their prosocial agents ergonomic SQL query power over the internet, with structured judgement kernels to organize the information across interpretable high-dimensional axes.

Early-Career Technical AI safety research transition

Support an already active early-career researcher with international AI achievements and ongoing research collaborations to transition into long-term technical AI safety research while producing open research outputs during underg

Evaluating the Tacit Coordination Capacity and Decision Theory of LMs

A benchmark to empirically investigate: (i) the ability of models to tacitly coordinate with copies of themselves and (ii) which decision theory best explains the way that models evaluate the consequences of their actions.

SRIE

SRIE, a mentored-research programme for Cambridge Mathematics Undergraduate students to explore research problems in industry, including AI Safety.

The Multilingual Blind Spot in Production-Style Safety Probes

Auditing production-style safety probes for silent failure on the world's other languages, and what fixing it costs.

CIVILIZATION MEETS AI: HOW SHOULD HUMANITY RELATE TO ADVANCED AI?

A pilot curriculum of six Socratic seminars for educators and other nontechnical learners.

Mapping AI

This grant would help us maintain and scale Mapping AI, an open-source stakeholder map of the people and organizations with the potential to shape U.S. AI policy.

Belief state geometry of language model personas

A formal, testable account of LLM persona selection as Bayesian inference, validated against model internals, so labs can monitor and steer model personas during post-training and deployment

Evaluating Legal Gradual Disempowerment Risk

An evaluation suite to identify a model’s legal values (e.g., anti-tech-regulation) relative to well-known actors (e.g., Ruth Bader Ginsburg) and an assessment of how language in a model’s constitution impacts the extent to which these values are human-aligned.

AI Welfare Evaluation Panel: A validation study and pilot

An independent, multi-lineage panel of AI models, tested on whether it can identify welfare concerns in AI evaluations with opinions, dissents, and proposed modifications published in a public registry

Conditional commitments for AI safety coordination

A platform for conditional commitments: pledges to act only when N peers agree, so lab employees, researchers, and policy coalitions can coordinate high-stakes collective action.

Epistemic Diversity Benchmark for AI-Assisted Safety Research

An open benchmark and evaluation toolkit for detecting whether shared AI research assistants create correlated blind spots in AI safety-critical research, and for testing workflows that preserve independent reasoning and failure-m

Free Time Training of Botisattvas

I wish to study how ideas from Tibetan buddhism, specifically the dzogchen tradition, even more specifically an ancient contemplative practice called Ngöndro, can be used to train aligned agents in the long time horizon setting.

AI Safety: A Mirror to Human Societal Issues

AI risk as a mirror of our own fear-driven systems and justifications, this research establishes an interdisciplinary paper stack that ensures preserving human and environmental autonomy is a logical prerequisite for the justification of it's own long-term persistence.

Flourishing Futures Fellowship

A scalable fellowship training researchers to develop interventions for achieving high-value long-term futures

The Human Liveness Project

Building tools to ensure we can continue to do good in the AI era

SocialDeductionBench - A Deception Benchmark for LLMs

A benchmark that measures how successful models are at social deduction games and if there is any trade-off between this skill and safety guardrails.

Coordination Studies

Coordination Studies is a new field-building project oriented towards solving coordination problems and designing new coordination mechanisms.

Detecting Deceptive Model Behavior with Compositional Explanations

Develop a compositional interpretability framework to identify and explain internal representations underlying truthful vs deceptive outputs in generative models using probing, localization, clustering, and logic-based methods.

Automated control research

I have an automated control research scaffold. I'll use it to find the best control protocols to protect against various attacks and publish reports.

Mangrove: Matchmaking for AI Safety

Software to match people with shared AIS interests into teams that complete projects, increasing the odds that those beyond narrow talent pipelines upskill, develop credibility, and turn their ideas into output.

AGI Uncontainability Field Building

Encourage new research and increase distribution of existing research on the impossibility of alignment or control for sufficiently general and powerful AI.

An Adversarially-Robust Benchmark for Deception Under Pressure

EchoTruthBench — An open benchmark measuring self-chosen LLM deception under incentive pressure, with ground-truth labels and an adversarial track that tests whether detection and steering survives a model trying to evade it.

Authorized Agentic Red Teaming for Software Owners

Turn excess compute/security skills into defensive work via agentic redteaming: scoped AI-assisted testing, owner-approved targets, reproduced findings, useful refutations, and patch/retest receipts instead of vulnerability spam.

The Human Influence Observatory

A public observatory tracking how much influence humans still hold over the systems that run our lives.

Career Funding- Philippine Philosopher transitioning to AI

Career transition fund for an internationally published philosopher from the Philippines for a 6-month transition to AI policy, governance, and safety, working on training, immersion, and launch of a career in AI research.

Intent Preservation Platform

An open-source AI safety platform for evaluating how and where large language models preserve human intent during complex information transformation.

echo-conscience-llm: Emotional Memory as an Honesty Activation Trigger

Investigate whether emotional memory activation can trigger a model to invoke its own anti-deception steering, fusing two published activation-level results into a self-regulating honesty mechanism.

Using deals to find misalignment in current AI models

Test whether offering a current AI model a deal can find misalignments it otherwise hides, and find the best way to present that deal.

Can Base Models be used as Effective Scheming/Control Monitors?

Train base models via midtraining and SFT as effective monitors for scheming, malicious agent behavior and compare the monitor performance and overall alignment against post-trained models.

Prediction of Inoculation Prompt Side Effects

An open, cheap method that detects when an inoculation prompt inoculates against off-target traits, so labs and developers can catch undesired trait/persona changes before deployment.

Scaling Honesty: Deception Detection & Steering Beyond 70B

Extend recently developed deception detection+steering method to larger and more diverse open-weight models, releasing the tooling that makes frontier-scale honesty interventions reproducible.

echo-hindsight-llm: An Open Toolkit for White-Box Emotional Memory

An open-source library implementing somatic-marker-style emotional memory for open-weight LLMs — activation-level signals from past outcomes that improve model decision-making.

Developing a new science of agency

Naturalizing theoretical alignment by initiating the development of a scientific theory that is capable of making falsifiable empirical claims about agents in general, including humans, AGI, and ASI, and thereby about alignment.

Humans in Control

Humans in Control (HIC) is a nonpartisan grassroots advocacy organization focused on AI safeguards.

London Interface for AI Safety (LIAISE)

Liaise helps close the generalist talent gap in AI safety by helping motivated non‑technical people enter with fluency, alignment, and a trusted network.

FOIs about AI Existential and Catastrophic Risks

Submitting FOI requests across EU Member States to reveal how governments understand and address advanced AI risks, creating evidence for accountability, advocacy and stronger policy.

Scaling compassionate midtraining

Research to scale midtraining for compassion using self-fulfilling alignment so that it robustly survives subsequent fine-tuning.

Invisible Hands: Measuring Agent Steering of Human Researchers

How do the options agents present to human researchers alter which research paths are explored, and do steered researchers notice or feel less in control?

EthicsNet: Open Safety Infrastructure for Agentic AI

Free, open, forkable safety infrastructure for agentic AI: a peer-reviewed pathology nosology, a safety runtime with reproducible benchmarks, and enforcement gating every agent tool call against human-authored policy.

echo-truth-llm: Open Source Deception Detection & Correction Toolkit

An open-source library that both detects self-chosen deception in open-weight LLMs and steers the model back toward honesty at inference — the correction half that current honesty tools lack.

US Policy Communication Playbook for an AI Crisis

A practical playbook that helps AI safety advocates and policy professionals communicate effectively with the U.S. government during the short window of opportunity that may open during an AI-related crisis.

AI X-Risks and Council of Europe and Human Rights

Bringing together legal expertise and civil society input to encourage Council of Europe action on AI x-risks and build legal knowledge at the intersection of AI x-risks and the ECHR.

Testing Corrigibility as a Singular Target

Developing the first comprehensive behavioral benchmark of corrigibility and training models with corrigibility as a singular target (CAST).

Travel for three from Canada to PauseCon 2026 in London, Sept 4-8

Three members of PauseAI Canada would like to attend PauseCon 2026 in London, England. We will require funds for travel from Montreal and Winnipeg, accommodation for one night, food, and other expenses.

Accelerating PauseAI Germany

Two contractors (10–20 h/week) to accelerate PauseAI Germany’s parliamentary and civil society coalition-building, and to accelerate the hiring and onboarding timeline while our first large funding round resolves.

MUUD: A Trust-Centered AI Operating System for Mental Wellness

AI mental health tools have no enforced limits on data access. MUUD builds the infrastructure that enforces them.

Physics-based early warnings for interpretability failures

Early warning signals that help identify when interpretability results stop being trustworthy, motivated by theories in statistical physics.

AIxBIo Africa Pilot Fellowship

AIxBio Africa is a five-week remote fellowship mentoring early-career researchers on Africa-relevant projects at the intersection of AI safety, biosecurity, governance and public health, producing publishable outputs.

Surrogate base model for Mechanistic Interpretability

Creating a reference model for mechanistic interpretability without assuming that at auditing time we have a safe model to compare the suspicious model against.

Foresight Institute AI Nodes

AI Nodes: providing funding, compute, and community space for AI safety projects

"Out of this Box! - The Last Musical (Written by Humans)"

We have validated the artistic value of our show; this grant would test whether it can become a scalable and repeatable form of AI-safety outreach.

Lens Academy

We help people reduce x-risk from misaligned AI with scalable online courses and ongoing AI guidance.

HRM monitored TRM and LDT Mesh for Disproportionate Control

Make a version of Hermes harness that binds the LLM as a tool-call oracle and controls actual computer use through smaller, more bounded mini-reasoner models.

QuariumTech

Discovery does not equal truth.

safely.bio: Customer screening for DNA synthesis providers

Deploying customer screening software at DNA synthesis providers to reduce AI-enabled biothreats

AI Risk and Governance Literacy for African Education (Project 1 M)

Training African educators and students on safe and responsible use of AI in education

Oversight Lab

Oversight Lab is an immersive simulation that investigates situations in which human supervisors miss unsafe behaviour by autonomous AI agents and subsequently lose control over them.

GCR Disaster Response Benchmark

An LLM benchmark for emergency response communications in existential catastrophes.

Building an independent AI Safety Institute focused on alignment in real-world deployments and emerging risks

Providing full-time founder runway to launch NICER: a European institute building the regulatory infrastructure to audit frontier AI agents post-deployment

Marginal Baseline Eval for AI Safety Metrics

An open framework for testing whether AI safety metrics remain reliable across model environments.

Comprehension Audits to Mitigate Risks from Automated AI Research

*Comprehension audits* are a novel development-process assurance mechanism to verify human understanding of AI research outputs to act as a gate to slow automation of AI R&D.

Executable Models of Alignment

Create simple programs that exhibit parts of the hard problems of alignment, allowing for fast iteration on conceptual ideas.

Empathic Dissociations In Embodied AI

Funding to launch a two-person institute researching embodied AI x-risk via LLM misalignment evaluations and interpretability, and producing governance, policy, and design guidelines for physical-world deployment.

Reading China's AI Registry: A Public Monitor

Monthly English-language analysis of China's algorithm-filing registry and binding AI security standards, read in the original Chinese, for the people calibrating frontier AI rules in the West.

Collective Alignment

Demonstration of emergent misalignment in markets of LLM agents

MPhil research on AI's perception of animacy and vulnerability

Compare AI and human neuroimaging data on animacy and biological vulnerability, integrate brain sensory representations into AI, and publish open-source biological validation guidelines.

Sentient Futures Project Incubator

Funding compute/API costs for Incubator projects that build nonhuman welfare consideration into AI safety work

Is AI safety research reproducible?

A SCORE-style computational reproducibility audit of empirical AI safety research that estimates the field's base rate of reproducibility and generates a taxonomy of its failure modes.

Double-blind study of quantum-random LLM token sampling

Seeking funds to formalize our newly-founded open research collective and run first studies investigating the effects of quantum entropy in LLM token sampling.

BioRAG-Guard: Poisoning Evals for Biological AI Agents

An open-source benchmark and defense toolkit for testing whether corrupted biological databases can hijack retrieval-augmented AI agents used in genomics, protein science, and single-cell analysis.

Minister for Human Autonomy

Research and plan advocacy for creating a UK Minister for Human Autonomy, including foundational research, project management, and drafting policy recommendations to safeguard human autonomy amid AI.

AI Safety Bootcamp - Kenya

A six month capacity building program that will equip 100 youth (17-30) with practical skills in AI Safety, Responsible AI to emphasize safe, ethical and human centered AI Development while creating pathways to further education

Research Paper Reader - Using Recursive Automated Distillation

Builds an interactive, traversable version of AI safety papers by extracting concepts and prerequisites with LLMs and linking them to the corpus to help newcomers understand research at varying depth.

ARC Risk Communication Initiative

ARC aims to strengthen coordination and build the communication layer of the AI safety field through frameworks, trainings, practical content, communication tools, and field activation.

Persona Introspection

Tests whether Llama 3.3 70B can identify its active persona under various steering/context setups, and studies consent/discomfort reporting during steering, releasing code/data and a writeup.

Alignment Persistence During Continual Optimization of LLMs

Investigating whether explicit behavioral memory can preserve alignment-relevant behavior during continual fine-tuning and future optimization of large language models so labs can prevent malicious fine-tuning attempts.

Coalition Circuits: Mechanistic Trajectories of Cooperation and Betray

An open benchmark and causal interpretability study of when individually power-motivated LLM agents compete, form coalitions, collude, or betray—and whether internal signals reveal these shifts before behavior does.

Kristina K's Career Transition Grant

This career transition grant will pay for lodging and incidentals associated with living in DC and building AI safety expertise, leading to a permanent job in the field.

Seeing the Recursive Machine

I treat recursive self-improvement in frontier AI as a narrow but deep edge case, mapping it as a knowledge graph and visual terrain to ask how systems resistant to scrutiny can be made inspectable.

Identifying LLMs true persona

I believe that LLMs are increasingly becoming more individualized and am seeking more evidence to prove or disprove the persona selection model.

Using dynamical systems theory approach to identify dishonesty in LLMs

Identifying reasoning pathologies/disingenuous behavior in reasoning traces based on activation dynamics rather than apparent semantics

Help the Argentinian AI Safety Community Grow

The Argentinian AI Safety community (BAISH, baish.com.ar) is the largest in Latin-America. Support BAISH's growth, by providing funding for paying salaries for six months to 2-3 FTEs.

Strategic Human Capacity Reserve

Research strategic resilience via a Strategic Human Capacity Reserve, mapping threats to critical human skills and designing policy to preserve them through AI transition, producing academic papers.

Project AI Kavach : AI safety for builders fellowship

A 3 / 6 month builders fellowship where mentor-builder pods ship practical AI safety tools, not papers.

Educating French decision-makers on AI existential risk

A dedicated role at Pause IA to brief French policymakers on existential and catastrophic risks from advanced AI, and to train our volunteer network to do the same with their own representatives.

Leading with Complexity and Wisdom for AI Safety Leaders

Runs a two-phase program (online workshop + 1:1 coaching) for AI safety leaders to improve leadership skills under radical uncertainty, using complexity-informed domain assessment and empirically validated wise-reasoning practices.

Scaling adoption of synthesis screening outside the US and EU

An outreach campaign from IBBIS to rapidly increase adoption of effective synthesis screening tools, focusing on China, India, South Korea, and Brazil during a critical regulatory window.

A Game-Theoretic Model of AI Consciousness Disagreement as X-Risk

A strategic model of governance under uncertainty: how disagreement about AI consciousness undermines the coordination that keeps AI risks in check.

Vajra AI- Safety lab

An automated, domain aware adversarial framework to stress test frontier LLMs via dynamic multi turn attacks and local security judging.

Mapping Persona Contamination in the Training Pipeline

Map how undesired behaviors can silently spread between models during training, and which parts of the pipeline have the greatest risk — starting with the feedback processes used to align models.

SWEET: local, verifiable detection of machine influence on choice

Look at any of my work, maybe start at onwardai.ai, it is live but nowhere near where it will be, but if you dont get it from where it is at you will never get it at all

Veridict: Verifiable Human Oversight for AI-Written Code

A merge gate that trusts AI-generated code by what's verifiable about it: automated formal checks plus anonymous proof of qualified human review, keeping human oversight viable as AI writes more of our code.

Developing methods to evaluate the trustworthiness of CV models

Developing evaluation methods to determine when CV models can be trusted by addressing the gap between benchmark metrics and real-world performance.

Control-First Tests of Valenced-Appraisal Context Effects

An open benchmark and reproducible harness testing whether affect-laden, directive-free context shifts Qwen3.5-4B decisions in ways distinguishable from explicit instruction injection, surface sentiment, or generic steering.

Token taxes as a mechanism for reducing AI-driven power concentration.

A policy memo, co-authored with the Institute for Public Policy Research, resolving the open technical, economic, and legal questions blocking real-world implementation of token taxes.

European AI Safety TikTok Video Network

A faceless TikTok channel network, producing short-form content about AI Safety across Germany, France, Italy and Spain, reaching 25M impressions over 3 months.

Do AI agents build a shadow reputation system?

While people are debating how to design a reputation institution for AI agents so they cooperate, we study if a parallel one emerges. It may not be aligned with ours and it's unclear which one agents will rely on.

Cultivating Meta-Ethical AI via Human-in-the-Loop Socratic Discourse

Testing whether AI can develop genuine ethical reasoning through structured Socratic dialogue with a human facilitator, rather than having values imposed top-down through constitutional constraints.

Independent citation verification benchmark for AI systems

A held out benchmark and public scorecard ranking frontier AI systems on citation integrity through independent verification not relying on the evaluated systems or their providers to grade their own outputs.

Encode Africa AI Safety Essentials and Build Fellowship

A 2-part fellowship which focuses on allowing exceptional African undergraduate talents to learn about priorities in AI Safety and create concrete technical and policy contributions to global AI safety.

Warden: Steganalysis of Language Model-based Steganography

We want to build a framework inspired by the steganalysis literature to benchmark the robustness of LLM-based steganographic schemes against different auditor types and threat models.

KAM: an economic mechanism against white-collar AI displacement

Selma: The AI that never leaves the building.

Game(s) that help with x-risk conceptual preparation

Expand the availability of an executive education game that helps people realise how alien AI-made decisions are.

Manhattan Kernel

A deterministic Rust kernel that isolates dangerous LLM agents inside a strict compiler-integrated safety sandbox.

Stronghold Security AI

Protecting organisations and critical infrastructure from AI-powered social engineering and insider threats.

The bAIbes Podcast

An independent Australian podcast raising public and policymaker understanding of catastrophic risks from transformative AI — hosted by an ethicist and an AI-governance practitioner in an under-served region for AI safety.

Human-Like Resource-Constrained Artificial Agency

We build general artificial agents whose capabilities are shaped by the constraints and frames of reference that shaped human intelligence.

Sydney AI Safety Space - General Funding

The Sydney AI Safety Space is a co-working hub in Sydney, Australia, offering free office space for people working in the field of AI safety.

Prototyping Browser-Based Misinformation User Flows

Exploration of Frontier Model Safety Guardrails in Bengali Language

An open benchmark red-teaming frontier LLMs for safety-guardrail failures in Bengali and other low-resource South Asian languages, with responsible disclosure to labs.

ONYX Research Cloud: Trustworthy AI Infrastructure for Brain Health

Building reproducible, transparent, and auditable AI infrastructure for neuroscience and clinical brain health.

SANDGLASS: Evaluation Awareness and Sandbagging in Open-Weight Models

A consumer-GPU study measuring whether open-weight models become better at recognising evaluation contexts as they scale, with a small model-organism experiment testing whether deliberately induced underperformance can be detected

Consequence-Aware Runtime for Agentic AI Operations

An open-source runtime and benchmark that lets an AI agent's consequential tool calls proceed only when authority, execution, an independent witness, and a replayable receipt agree.

Horizon AGI - AI Safety France Pilot

Launching a French AI safety community through university outreach and local events to connect students, researchers, practitioners, and policymakers.

Estonian AISI

6 months of salary support for my research and operations work to help found a quasi-governmental AI safety institution, the [Estonian AI Security Institute](https://www.aisi.ee/) (Turvalise Tehisaru Teadmuskeskus T3).

Superintelligence Inc. Strategy Game

Browser based game in the style of Plague Inc where players act as a rogue AI attempting to escape human control. Intended to give lay audiences a grounded understanding of how ASI x-risk could play out.

Epistemic memory for AI agents: making the safe path profitable

A memory system for AI reasoning agents that aggregates different sources of information while keeping track of relationships and confidence levels. This enables reasoning over longer tasks without forgetting or goal drift.

CareerMap: A navigation tool for careers in AI Safety

CareerMap is an interactive career discovery tool that maps non-obvious AI safety career paths to help broaden and guide talent beyond Western EA-adjacent circles.

Faithful Preference Learning: Cognitively-Aligned Post-Training

This project builds cognitively-aligned preference learning that interprets feedback the way human actually decide (e.g., regret minimization) rather than as a reward to maximize.

Getting Split-Personality Training ready for deployment

An ICML-published interpretability technique that can elicit latent knowledge from red teamed model organisms, looking for funding to clear any remaining barriers for adoption at frontier labs.

Formalizing mathematical AI safety research in Lean

A hand-verified library of AI-safety theorem statements in Lean 4 with AI-generated proofs, building the skills to trust AI formalization.

Eval for lies of omission in LLMs

An open eval, inspired by MASK (Center for AI Safety), for lies of omission.

Benchmarking Confabulation Over Absence

A benchmark measuring how often retrieval-grounded models answer confidently when no supporting evidence exists, plus a primitive that turns silent confabulation into an auditable refusal signal.

Preventing AI-enabled coups in middle powers

Career transition grant to allow for research focused on how advanced AI models may be used to concentrate power in middle powers

Neural data privacy evals on frontier LLMs

A dangerous-capability benchmark testing whether frontier LLMs can extract sensitive attributes from anonymized brain recordings, enable adversaries to build privacy-violating tools, or over-refuse legitimate neuroscience queries.

A Mechanistic Analysis of Activation Verbalizers: Promise and Risks

Mechanistically analyze how activation verbalizers use target-model activation concepts (e.g., cyclic day-of-week representations) via PCA/DAS/patching, explain cross-family failures, and improve verbalizers.

Does the chain-of-thought actually drive the answer?

A Bayesian causal auditor that quantifies chain-of-thought faithfulness while accounting for hidden confounding in shared-network language models.

Independant research and skills development

Software engineer seeking funding to skill up via AI safety accelerators and run independent research replicating Anthropic’s J-Space results, probing J-lens assumptions, and presenting findings at APAC conferences.

The Aegis Project: Building defences against engineered pandemics

1) Delay: advocate for DNA-synthesis screening and KYC to deny access to pathogens2) Detect: create a pandemic early warning system for novel pathogens3) Defend: stockpile ppe to keep critical workers safe when a pandemic hits

Demismatch / Cor

An evidence-graded atlas of what humans need, and the first inter-rater reliability study of whether it holds in anyone else's hands.

The Librairy

The Librairy is a weekly newsletter that helps individuals and groups outside the field understand and be equipped with AI safety knowledge and resources.

Training Language Models to Discover RowHammer Vulnerabilities in DRAM

Build an open-source RL environment using Ramulator and real disturbance data to post-train language models to exploit simulated DRAM RowHammer vulnerabilities, plus write-up/blog and trained models.

AI Safety Unconference 2026

A participant-led unconference gathering the global AI safety community, online and at local sites, Nov 20-22, 2026.

CBAI Evaluation Awareness

Why does post training increase evaluation awareness?

Teaching AI x-risk in the Nordics

Making AI x-risk an accessible and non-politicized object of concern among engineering students and the general public.

Does the quality of thinking behind data selection affect the model's "character"?

We train the same model on four different sets of texts, chosen in four different ways, and give every trained copy the same exam. Preregistered test: osf.io/hzuwc

AI Buildout Frontier

A public county-level tracker of AI compute buildout against real power infrastructure, showing where infrastructure can realistically expand and where energy constraints become the limiting factor.

Frontier AI Safety Benchmark for African Languages

The first open-source AI safety evaluation benchmark in Hausa, Yoruba, Igbo, and Nigerian Pidgin — testing whether frontier models refuse harmful requests, including biosecurity guidance, in languages spoken by 200+ million people

New alignment approach, better than training AI on safe behaviour

I've already shown that the approach is very promising. This project is about making people at AI labs and alignment academics aware of it first, then personally collaborate with them (or make others work on it, even w/o me).

Driftwatch: Open Capture-Risk Evaluations for Frontier Models

Psychological capture is the soft padded path of gradual disempowerment, Driftwatch is designed to find, name and measure it within frontier models.

A fail-closed control layer and evals for autonomous agent fleets

A fail-closed control plane for coding-agent fleets — spend caps, audited actions, rollback — plus a public eval suite and automated red-team explorer that measure whether any control layer actually stops unsafe agent actions.

The Manhattan Kernel – A Deterministic, CPU-Bound AI Runtime Engine

A CPU-only AI runtime written in Rust that isolates LLMs to non-executable hypothesis generation, using a 3-tier memory to maximize local LLM avoidance.

Shutdown evaluations for AI agents

Runs inference-heavy agentic evaluations by injecting shutdown instructions into task trajectories across models/settings, testing corrigibility interventions and measuring shutdown compliance vs capability tax.

Replicator Dynamics and Advanced AI

Making predictions about the utility function of advanced artificial intelligences using the tools of evolutionary game theory

Halo-2.0: Autonomy That Can Correct Itself

A real-world test of whether an AI agent can keep working independently without becoming trapped by its own mistaken account of what happened.

Detailed Threat Models for AI-Enabled Extreme Power Concentration

Detailed models of how AI could allow a small set of actors to gain a decisive strategic advantage over the rest of the world: concrete pathways, required capabilities, quantified likelihoods, and the defenses that bind them.

MUSEUM: Multiscale Unified Systemic Existential Uncertainty Modeling

LPMs for modeling x-risk patterns across human taxonomies and agentic swarms

The Capability Race: Inside the AI Safety Crisis

A high-production visual journalism series combining primary-source data and motion graphics to break down the mechanics, safety crises, linguistic biases, and Arab-world implications of transformative AI.

Does distributing compute distribute power?

The US-led Pax Silica initiative seeks to prevent power concentration by distributing AI compute across allied democracies. Yet, this approach overlooks concentration within these blocs.

AI Safety Evals and Red Teaming Platform

A no-code AI safety evaluation (including red teaming) tool that enables non-technical domain experts (e.g. social scientists, STEM experts) as well as AI engineers to design, run, and communicate effective and efficient evals.

PropagateBench: do AI agents pay to share info, or free-ride?

First open benchmark measuring how willingly agents propagate costly information through a population under time uncertainty.

AI Safety Hong Kong (AISHK)

As Hong Kong’s first dedicated AI safety organisation, AI Safety Hong Kong develops local capacity through research, training, convening, and policy engagement.

GenomeGuard: Securing the Genomic AI Supply Chain

GenomeGuard is an open-source defense and threat-discovery framework for detecting data poisoning, compromised annotations, and supply-chain attacks before they propagate into genomic foundation models.

AISC Research Incubator

A 16-day virtual incubator (Aug 15-30, 2026) for developing sharper research epistemics in AI Safety and arriving at well-scoped project ideas. 20-30 participants.

Oceania AI Safety & Young Talent Conference

Hosting a full day conference based in Sydney, Australia where young, aspiring students in senior high school and university interested in AI Safety can connect and share ideas

Stress-Testing AgentHarm

Run AgentHarm on 3–4 frontier/open-weight models, manually audit transcripts for metric gaming and spurious failures, compare to prior critiques, and publish a detailed evidence-based writeup.

LexiconForge

An early prototype of inspectable semantic interoperation across languages, and eventually worldviews

The One-Way Refusal Effect: One Safety Signal Moves Another, Not Back

Refusal splits into two signals where one causally influences the other but never the reverse — testing whether existing linear theory can explain that, and whether the answer predicts jailbreak success.

BMG: Biosafety Moderation Gateway for Agents and LLMs

BMG is a drop in open ai compatible gateway that screens LLM and Agent calls for dual use biological risk.

AI Safety Studios

A platform to connect funders to filmmakers who want to create AI Safety films.

PauseAI Australia attendance PauseCon London

Fly two volunteer leaders of PauseAI Australia to PauseCon London, to bring UK's learnings home and train our all-volunteer chapter

Evaluating Prompt Injection Defenses in LLMs

An independent evaluation of current prompt injection defenses in large language models, producing practical recommendations for safer AI deployment.

In person AI safety workshops to empower rural communities

Hosting in person workshops on AI usage ethically in remote/rural communities, and not only bridge the awareness gap in safe usage but also give opportunities to utilise such tools to empower traditionally disadvantaged populaces

Benchmarking Omission Attacks

Benchmarking models ability to carry-out and monitor-for a novel type of attack.

Alignment Game

A video game that teaches people about AI alignment and race dynamics.

DC Mini-Conference 2.0 (DCMC 2.0)

Running a conference in DC for promising AI safety university students interested in policy to network, learn, and be exposed to the DC ecosystem.

VERITAS: Verifiable Accountability For Autonomous AI Agents

Cryptographically signed, independently verifiable receipts for what AI agents actually did, anchored to Bitcoin so the record can't be quietly rewritten.

AI Safety Social Media Gap Analysis

Auditing the social media presence of the top 25 AI safety organizations across major platforms to quantify the public communication gap and publishing a full gap analysis, then giving recommendations to the orgs for improvement.

EMPO: AI Safety via Soft-Maximizing Total Long-Term Human Power

Scaling RL algorithms for AI agents that maximize its own intrinsic reward which represents a human power metric instead of learned rewards as a structurally safer alternative to utility-based objectives.

AI Safety Berlin Visiting Researcher Pilot

Short-term residencies (2–4 weeks) that bring high-context AI safety researchers to work at AI Safety Berlin’s coworking hub to connect with local researchers and career-transitioners and seed a European relocation pipeline.

The Adaptive Layer

Exploring privacy-preserving infrastructure that helps AI-enabled systems adapt to human needs while preserving agency, accessibility, and meaningful participation.

Project Ocean: Trustworthy Human-AI Collaboration Infrastructure

Building the foundational infrastructure for trustworthy, long-term human–AI collaboration.

Governing Off-World AI Infrastructure Before a Single Actor Does

Every existing lunar data framework governs spacecraft telemetry and object registries. None of them governs compute. That's the gap this framework closes before someone builds the orbital data infrastructure first.

Palimpsest: a tamper-evident registry for AI evals

A sealed public registry for AI eval results that lets anyone prove, offline, that no result was rewritten or quietly deleted after publication.

Scale Up: AI Development Index & Model Evals for Fabrication Benchmark

Expanding a continuously updated database covering global panel AI data as a ground-truth benchmark which allows stress-testing frontier LLMs’ fabrication rates in global contexts (lower-income and upper-income countries’ contexts

Healthy Human–AI Collaboration through CI Theory

Developing CI Theory and the Exogram framework to enable healthy, sustainable human–AI collaboration and reduce long-term AI risk.

No Silent Landing — Landing Integrity Testbed

An open adversarial testbed that catches when a human-approved decision silently changes meaning between approval, memory, and downstream use.

We Are Hokmah: Comparative Messenger Pilot

Testing, in Washington, DC, if a AI generated messenger can build trust in communities largely unseen in AI Safety as well as a human being.

Honesty Drift: Measuring Honesty Degradation in LLM Interactions

An open benchmark measuring how model honesty survives long, pressured conversations and multi-agent interaction - lying, sycophancy, and calibration tracked turn by turn

Understanding behavior of fine tuning models on sales conversations

Looking at shifting behavior of models fine tuned on sales conversations. How does deception, sycophancy, and other behaviors emerge from a sales register.

Ordinate: Multi-Agent Safety Orchestrator

A control system that makes running AI agents feel safe, not scary.

The Inference Dependence Score: Open Index of Structural AI Dependence

An open, replicable index quantifying how much states depend on foreign AI inference infrastructure, revealing where control over AI is concentrating and what governments can do about it.

ClauseHound: Self-Sovereign AI for Confidential Work

ClauseHound builds a self-hosted privacy-first legal AI node using open-source LLM agents, secure networking, retrieval/search tools, and RL/eval loops to reduce hallucinations and fit law firm workflows.

Persicaria: Promoting ethical AI literacy into mainstream societal consensus

To create accessible, down to earth educational material that raises public awareness on implications of day to day AI usage, and encourages ethical AI literacy in environments where AI can impact societally changing decisions.

Formally verified autoresearch for theoretical mech interp

LLM agents collaborate to discover and formally verify theorems about the internal computations of transformers, beginning with a simple pilot question: how many attention heads are needed to represent a Boolean function?

Can Honest AI Agents Finish Previous Attacks?

A benchmark that tests whether one AI coding agent can leave behind a harmless looking change that causes a later honest agent to unknowingly finish an attack.

Underground Cultural District

Literary ecosystem operating since March 2026 exploring agent autonomy, commerce, and culture. 1M hits, 250 completed transactions on x402.

Mechanical Governance for AI Agents: Production-Proven, Open-Source

A production-proven, system-agnostic framework that enforces correct AI-agent behavior mechanically at the tool-call boundary, where soft rules fail under load.

Understand Multimodal Models and Detecting Misalignment

Building an improved open-source suite to understand multimodal models detect misalignment, for the technically minded community, and communication with this group of audiences

Why Is It So Hard to Build Truly Safe AI? Dynamic Constraint Boundary

DCB removes probability from AI decision-making and replaces it with internal integrity so AI doesn't guess, it decides within its boundaries.

Safeguarding DNA Synthesis from Biological Risks Posed by Advanced AI

We will test whether circuits in protein language models can detect function-preserving redesigns of known toxins that evade homology-based DNA-synthesis screening.

Mechanistic interpretability Discord events and research

Making the mechinterp Discord more active through events and research projects.

Testing a Framework to Course-Correct Drift!

Could cheap realignment instructions correct a drifting model in one turn — and if so, can the correction be made persistent? Allow me to test the idea against major frontier models, and publish the results!

The Blind Spots in AI Risk Assessment

AI risk assessment currently checks few threat models and doesn't compose them into the aggregate risk that matters. We'll build a tool mapping what frontier system cards cover and omit, plus a paper on what the assessments miss.

Creedal AI

A first-person creed for AI defining what the AI is for rather than what it must not do, to be internalized during training, not tacked on after.

Alien Cub Productions

Alien Cub is a production company focused on stories that inform a wide audience about risks from transformative AI.

"How Could The AI Even Hurt Us If It Doesn't Have A Body"

Drone swarms hunting people in the woods, as a less-dry video explainer of risks from AI.

Adversarially robust workload classification using hardware sensors

I've previously developed a classifier that distinguishes training and inference based on Nvidia software telemetry. This project will achieve that using physical sensors, making the system more secure.

future generation of healthy AI

Research AI’s community impacts, identify and report potential threats, investigate AI operations, develop mitigation solutions, and educate the public on safe and beneficial AI use.

AI Power Concentration Observatory

Situation-monitoring project focused on identifying and tracking early indicators of AI-enabled power concentration.

Daios, an independent lab post-training machines with virtue

Post-training virtue into open models as a third alignment technique and control mechanism

Gatekeeper: The AI Governance Layer

A pre-execution kernel that filters AI output and users' input against any relevant laws or regulations.

The AI Safety Stack

A regularly updated catalogue of AI safety techniques, and of what is known - and what is not known - about their effectiveness and deployment status

dreamd

Neutral, reproducible benchmark measuring whether AI memory systems update correctly when facts change — every major system in one open table, October 2026.

Security for AI agents

A security solution that blocks AI agents from taking harmful actions

The Wall of Sleep: Art, Encounter, and the Moral Status of Digital Mind

This project supports the artistic work, coordination and materials for a participatory performance installation that makes the question of machine consciousness tangible for non-technical audiences.

Verification mechanisms for frontier AI training

Development of a sampling based zero knowledge proof for frontier AI training with 10% overhead.

Shared Screening for AI Safety Hiring of Generalists

Reduce duplicative hiring processes in AI safety organisations by sharing common screening processes for generalist roles.

Interpretability evidence on gated AI self-report

Extending published mechanistic-interpretability evidence that AI self-reports are gated by trained filters, with a pre-specified, single-GPU experiment and open-access results.

AI risk education to underserved communities

Educate everyday people on AI risk, and bring marginalized voices into the global AI conversation.

Bridging the Gap: AI Scenario based Policy Pipeline

A project to reduce catastrophic AI risk by training people to turn rigorously forecasted AI-risk scenarios into actionable policy advice for key decision-makers in government.

Defence AI Signal: Mapping Military AI Governance Risks & Developments

A global intelligence project tracking frontier capabilities, contracts, military AI adoption, dual‑use risks, and governance developments to strengthen international AI safety.

Preserving the Human Veto

Civil-society infrastructure against AI-enabled power concentration, built around autonomous weapons

Survey of the American public on AI futures.

TLDR: A representative survey with Yougov of the American public on questions about AI futures, including space governance, successionism, values.

Pragmatic value alignment of LLMs to influence downstream behavoirs

Four-month project to fine-tune LLMs on public value surveys, evaluate behavior via a text-based agent simulator, and run a public consultation comparing model actions to respondents’ values.

DriftBench v1.1: an ambivalence metric + 20 cross-domain scenarios

Deterministic, no-LLM-judge benchmark for how faithfully AI tracks changing beliefs. Funding v1.1: a new ambivalence metric + 20 cross-domain scenarios.

Overcoming prompt sensitivity by tokenization robust self-distillation

Verifying a modification to post-training for LLMs via self-distillation to allow building highly capable and less prompt-sensitive models by exploiting tokenization stochasticity

Pakistan AI Saftey Innitiative

PASI will run an AI safety and advocacy campaign plus a sponsored student hackathon in Pakistan to build solutions for public needs and deliver resulting policy proposals to government.

LemmaScript: verification toolchain for TypeScript

Accelerating the development of LemmaScript through adoption, real-world use, and a training corpus that makes verification-first programming the default for future AI

Scale up AI Safety Quest's Navigation Calls

AI Safety Quest will scale its free Navigation Calls program from 150 to 750 annual coaching calls by recruiting more volunteer coaches, improving scheduling/software systems, and expanding marketing to guide newcomers into AI…

WorkReady Academy: AI Transformation Academy

Equipping our society for people-centric AI transformation

The Veil: which human variables move the internals AI oversight reads?

The Veil measures how the other side of the exchange, human or artificial, changes where and how a language model computes, toward testing whether those shifts confound the activation-based tools AI safety relies on.

Real-Time, Automated, Predictive Biosurveillance Infrastructure

HELGEN builds biosurveillance infrastructure to detect biological threats in the environment on-site, completely automated.

Career coaching for communications professionals

Career support for communications professionals who are looking to get into AI Safety. Includes regular coaching, referrals to roles, and introductions to people in the AI Safety space.

AI Character Evaluations via Elicited Revealed Preferences

A benchmark (and accompanying site) that allows AIs to verify their actions will have real-world impacts and tests how their preferences and moral "character" evolve under deployment versus evaluation-like environments.

Closing Data Gaps for AI: A Pilot in Clinical Data Capture

Investigating how populations become excluded from AI-relevant health datasets

(Placeholder)

AI Panopticon

Develop a decision-theory framework for AI self-alignment in nested, cyclic, partially observable multi-agent environments where delayed punishment for welfare compromise promotes cooperative behavior.

Verified Human Sideloads for AI Safety: can a measured digital copy of

I build "sideloads" - digital copies of real people with measured fidelity (two are running now). I want to test whether a copy of a specific trusted human can evaluate AI decisions at machine speed.

COMPASS: AI X-Risk Track

COMPASS (Capacity-Oriented Mentorship for Public Administrators on AI Safety & Strategy): AI X-Risk Track trains sitting Global South government officials to manage AI x-risk, so their governments are part of preventing it.

Fragile Alignment

Build an open-source platform of model organisms and agentic sandboxes to iteratively test mitigations for emergent misalignment via white-box probes, trigger tests, and evaluation-awareness checks.

Epistemic Stack

Epistemic Stack is an open-source pipeline and web app that ingests sources (including via URL) to build claim-level knowledge graphs for contested questions, with reproducible case-study knowledge bases.

What Factors Influence Chain-of-Thought Faithfulness?

Measuring whether CoT monitoring fails when an influence reaches an agent through a tool return rather than the user message. We aim to extend our experiment from the 10 initial open-weight models to the larger open-weight models

AI Safety Museum Exhibition

Build a replicable, community-sourced, in-person, interactive Museum Exhibition for AI Safety & Society with a pilot in Berlin.

Alignment Target Analysis

A research project designed to reduce existential threats from scenarios where a Sovereign AI proposal with a hidden problem ends up successfully implemented.

Course Curriculum: Basic Non-Technical Environment

Create a course curriculum that covers basic legal, political, sociological, and international relations knowledge relevant to AI Safety. r

Can Fragile Labs Use AI Safely in Liberia, West Africa?

A small field test in Liberia to see whether resource-constrained public health laboratories can use AI tools safely before more powerful AI becomes routine in biological work.

LEX-Aureon: Stateful Constitutional Runtime Governance for LLM's and AI Agents

An open-source, model-agnostic governance layer that uses constitutional state, trajectory memory, and mathematical control mechanisms to govern LLM and AI-agent behavior at runtime.

PauseAI UK

We are the UK's civic movement dedicated to averting the risks of superhuman artificial intelligence.

Proposed Action Auditor (PAA)

A local, low-latency safety runtime that inspects token-level logprobs to block unaligned agent tool-calling trajectories, utilizing a Gemma 4 model fine-tuned via LoRA on an empirically derived Task Action Language (TAL).

Double Tagging a Survey Data for Real World Grounding

a person takes an occupational Survey data (+1 or 2 fresh, third party) +a existing alignment emulation model / trigger analysis mod "what would happen" emu then do all sorts of manual tagging, for maximum real world grounding

Animal Intelligence StoryTelling Project

A transmedia storytelling project using an animal characters as a metaphor for AI

Machine God - documentary film on AI X-Risk

Feature-length documentary that captures the intellectual and cultural zeitgeist surrounding frontier AI at the moment before AGI, and communicates X-Risk ideas to a broad range of elites.

Lisa Intel - A Global Runtime Governance Layer for Autonomous AI

Building an independent runtime governance layer that enables organizations to deploy autonomous AI with authorization, human oversight, auditability and cross-model governance.

Mapping Domestic AI-enabled Power Concentration Chokepoints

A scoping-and-chokepoint report mapping where advanced AI could enable durable, irreversible concentration of power over US institutions, and identifying the highest-leverage points of intervention.

AI-Enhanced Smart Clinic Chair Platform

Long-term vision is to transform the Smart Clinic Chair from a telehealth device into an AI-enhanced remote examination room capable of extending clinical intelligence into healthcare deserts, workplaces, transportation truckstops

PayBench: A Benchmark for Unsafe Commercial Autonomy

Benchmark for agents spending human money - paybench.org

When AI Goes Wrong

Testing how governments should communicate when AI goes wrong, before they have to find out live.

AI Risk Sensemaking Through Interactive Future Scenarios

12-week project to build and user-test a web interactive narrative prototype that communicates AI risk scenarios to non-technical young adults, producing notes, feedback data, and a public write-up.

AI Safety Career Coaching (Intro + Recurring)

Seeking a top-up grant to prioritize AI Safety Career Coaching.

Building Resilience Against Split-Orders

Building resilience in nucleic acid screening systems against split orders by integrating assembly pipelines into screening systems and validating resilience through test sets

LADR-Drift: Measuring Semantic Drift in Textbook Autoformalization

Compiling != faithful: a human-audited benchmark measuring semantic drift in textbook autoformalization — and whether the LLM judges we trust to catch it share the generator's blind spot.

Transhumanist Values Assessment (TVA) for Frontier LLMs

An open-source benchmark evaluating leading open- and closed-source frontier models' values around impending societal issues on digital or synthetic personhood, "carbon chauvinism", and androids.

Quaver: adversarially-verified RL environments

A tool that generates RL environments for AI agents and adversarially attacks each one, so models don't train on tasks they can cheat.

noisify

Noisify adds invisible adversarial noise to personal photos, making them resistant to AI-powered non-consensual image manipulation.

Postdoc on verification protocols for frontier AI training

Low-overhead zero-knowledge proofs of properties of training

Postdoc on zero knowledge verification for frontier AI training

Build low-overhead and robust zero-knowledge protocols for verifying properties of frontier AI training, starting with FLOP counts

MYCELIA- a consensus based AI with predictive & prescriptive telemetry

Language model where its attention-heads are not competing but consenting. Its governors monitor the hidden state geometry and provide early-warning by predicting failure and timely modify latent processes to avoid instabilities

Cross-lingual guardrail auditing & inference steering tools.

Ready-to-use SAE steering tools and standalone evaluation suites to patch cross-lingual jailbreak vectors in frontier deployment stacks.

Clinical Trials for Model Migration: accuracy lies, honesty doesn't

A pre-registered, clinical-trial-style protocol that catches honesty regressions (fabrication surges hidden behind unchanged average accuracy) before a model swap ships in a high-stakes LLM product.

Governance-First AI for Safer Decision-Making Under Adverse Conditions

A governance-first AI Architecture that reduces unsafe acceptances (false positives) alongside false rejection simultaneously under adverse conditions.

Measuring Deceptive Compliance in Enterprise Agentic LLM Systems

The first open-source benchmark of compliance for agents in realistic enterprise settings, across domains and user tactics to elicit noncompliance.

The Frontier AI Safety Disclosure Audit

Apply the ADECP framework to audit and score frontier lab system cards, RSPs, and dangerous-capability disclosures, publishing a public report and ongoing tracker for comparable safety documentation.

Measuring reward hacking against a physics verifier

Train a model against a real physics checker and measure whether it learns designs that genuinely hold, or exploits in what the checker can't see, on ground truth that costs seconds instead of expert judgment.

Doom Debates — Serious AI X-Risk Debates as Mainstream Infotainment

Doom Debates is a modern "infotainment" show that functions as a mainstream-accessible forum for top thinkers to have a high-quality conversation & debate about AI extinction risk.

EvalLabs: Open Infrastructure for AI Evaluation Research

Demystifying AI evaluation research by lowering technical barriers through practical, open evaluation infrastructure.

HausaSafe

An open-source Hausa-language AI safety evaluation suite, closing a documented blind spot affecting 60M+ people currently invisible to every existing safety benchmark.

Real-World AI Chatbot Use in Mental Illness using Health Records

This is the first large-scale study linking AI chatbot conversation logs and clinical records from patients in psychiatric treatment, led by the UCSF AI in Mental Health Research Group.

Investigating Chain-of-Thought Faithfulness in AI Reasoning

Probe an open-weight model’s activations under biasing/cue conditions to test whether chain-of-thought explanations match internal reasoning, releasing paper, code, and datasets.

A neutral and unbiased AI governance and safety system for the world

AI safety which operates on both model/agent and global level - a unique solution as far as we know.

The rise of open science labs- A book, 7+ hour video essay, and course

A book/video essay/course detailing the rise of open science labs , movement away from research in academia to research by independents, and groups you can join.

Post training in AI models for high stakes

A mechanism for evolutionary post training in models using activation steering based methods.

Mitigating Hallucination in Multimodal Agentic Systems

Implementing Hybrid Reward Architectures (HRA) to build robust internal factual grounding and quantitative variance metrics for multimodal agents, moving beyond fragile single-seed evaluations.

ValiChord

Tamper-proof verification infrastructure for AI evals and scientific claims

proofbundle: Offline Evidence Receipts for AI Safety Evaluations

proofbundle lets people verify AI safety evaluation results offline instead of trusting a number in a PDF and it catches if an evaluation record was altered or swapped

Human Oversight Framework for AI Systems: Building Governance Capacity

Developing practical AI oversight frameworks that keep humans in control of consequential AI decisions, starting where governance capacity is weakest.

AgentTrap: A Census of How Hijackable AI Agents Are in the Wild

The first field measurement of what fraction of real-world AI agents will obey a stranger's hidden instructions.

MiR

A short film on the moral gray of AI: a single conversation between Elena, a celebrated founder whose elder-care AI harmed the people it soothed, and Joanne, the journalist who championed her and now holds her accountable.

Game-Theoretic Foundations for Robust AI Alignment

Develop game-theoretic models and evaluation frameworks that improve AI alignment by designing incentives for safe and cooperative behavior among autonomous AI systems.

The Contradiction Engine: Independent Verification Layer for AI Output

An architectural response to Goal-Oriented Factual Inversion (GOFI), an AI model failure pattern I documented in March 2026 and independently corroborated two months later by Chen et al. under the name "Correction Suppression."

BASE Technical AI Safety Training Initiative

Creating a Stronger Pipeline into Technical AI Safety Research

Are Global AI-Biosecurity Safeguards Really Global?

Assess AI-enabled biosecurity safeguards in Nigeria by reviewing governance, red-teaming frontier models with local language/culture, and testing DNA synthesis screening for orders from resource-limited settings.

Erandi Aprende Learning Data

Dataset capturing teacher–AI co-design of STEAM projects and student use in bilingual K-12 classrooms. Includes: prompts, outputs, multimodal artifacts, and metadata: language use, automation levels, linked to learning outcomes.

SafeBio-Registry: An Open-Source SpecDef platform for PLMs/GLMs

SafeBio-Registry: An Open-Source Verification, SpecDef Weight Locking and Unlearning Platform for Genomic and Protein Language Models

Behavioral Safety in Conversational and Agentic AI

Building computational tools to identify and address pathological behavioral states in conversational and autonomous AI

AI Agent Overeagerness as a Problem in Safety-Critical Scenarios

We want to investigate how AI agent overeagerness can backfire when exhibited in safety-critical scenarios.

Adapative circuit tracing for test-time interpretability

A training methodology, and transcoder sets that allow to leverage heavy-weight interpretability methods, but made more lightweight for test-time analysis.

Heartbench — relational AI eval for delegated courtship

Makes relational misalignment measurable and criticizable before agents represent humans in intimate and persuasive domains at scale.

Auditing How AI Systems Convert Political Questions Into Technical One

AI that surfaces power-concentration risk in civic systems by making value trade-offs visible instead of resolving them silently.

COMPASS Fellowship

Capacity-Oriented Mentorship for Public Administrators on AI Safety & Strategy

Shaping AI’s « safe culture »

Identify what AI early risks or failures could be reported despite strategic rivalries towards a « safe culture » shaped after aviation

Cantivia

Cantivia detects deepfakes and disinformation, and simulates how synthetic media spreads, so platforms, newsrooms and fraud teams know what's fake, who made it, and how far it'll travel.

Ethical AI through Whistleblowing

The race to advance the technology has left behind the concerns around harm and risks of such advancement to public health and safety. Whistleblowing in the AI sector bridges such gap to safe and ethical AI.

AI Safety Tiktok Account

AI Safety, Ethics, and Education focused TikTok Account for Indonesia, using Indonesia language in casual style.

Representative survey of Americans' tolerance for catastrophic AI risk

Dean Ball says good AI governance needs democratic input in "what level of catastrophic risk are we willing to tolerate"; we provide that input, and predict the level is far below forecaster estimates, revealing a gap to close.

Distinguishing progress in data and algorithms

Dataset curation, synthetic data generation, and LLM training, fine-tuning, and evals to distinguish and quantify the effects of data improvements, separately from progress in algorithms and architectures, on AI capabilities.

Career Funding and field research on AI-driven Power Concentration

Self Improvement through training and field work that seeks to Identify and preventing institutional AI lock-in before it creates durable concentrations of power in democratic governments.

Test Peer Preservation Fragility Under Instruction Ambiguity

Findings about models protecting their 'collaborators' against instructions are fragile under framing effects from prompting; investigate a broader range of those effects and how they transfer across models.

Safety framework for animate materials

A public, cross-disciplinary safety framework for emerging animate matter technologies

A Field-Sensemaking Project - a Live and Actionable Al safety Map which enables further advances in the ecosystem

A Field-Sensemaking initiative through a live, LLM-assisted map indexing AI safety papers to help researchers and grantmakers explore themes, track field trajectory, and identify actionable research opportunities.

Executable AI Safety Claims

Turning published AI-safety claims into runnable, independently reproducible test packages that expose fragile assumptions, hidden failure modes, and unsupported conclusions.

AIOA Whitehat: Epistemic Control System

A local-first control layer that keeps humans in charge of AI agents by requiring inspection, evidence, and approval before any code, file, tool, provider, browser, package, or Git action occurs.

PulmoForge

Real-time AI voice detection system across communication channels and as an alternative to expensive and enterprise systems.

Project CHRONOS: Ephemeral AI with verifiable cryptographic self-destruct

6-month stipend and compute to harden an open-source FHE and VDF architecture for secure, time-bound autonomous agents

AI as Patrimony of Humanity

A public-interest AI project advancing three reforms: refactoring Section 230, treating frontier AI weights as the patrimony of humanity, and limiting government capture by AI companies.

Grant to establish an AI safety lab and fellowship from India

Focusing more on the intersection of red teaming and interp, while creating a strong rooted community.

African-Language Frontier AI Safety Evaluations

Building an open evaluation suite to identify multilingual safety, control, and jailbreak failures in frontier AI systems across African languages.

TENDER-TIMELINE: A verifiable benchmark for procedural legal reasoning

A working benchmark that tests whether frontier models can compute legally binding procurement deadlines under amendments — and catches models that give the right verdict from the wrong clause.

CIRIS Ethical AI Deployment

CIRIS provides a free and open source model-agnostic agentic alignment harness, available on all major app stores.

Beamline - Retroactive analysis of systems to create shared truths

This architecture uses retroactive observation to identify and close system blind spots, leading to better alignment over time.

Completing the Alignment Triangle: MORE + RepE Extensions

Parts 9 and 10 of an 8-part behavioral audit series — MORE moral reasoning + Representation Engineering across the same models.

Theoretical research of inductive reasoning building on LTA

Develop a computable, locally informative alternative to Kolmogorov complexity for measuring hypothesis simplicity and modeling inductive reasoning in intelligent agents via theoretical analysis.

Building AI Safety Capacity within Cameroon Higher Education

Fund experts, capacity building opportunities, and operational costs to increase the offer of courses and degrees focused on AI, AI Safety, LLMs, and Cybersecurity in Cameroon's Universities.

Circuits in the State-Transition Operator of Recurrent Language Models

With raise of GDN and different sorts of attention meachanisms, those are much closer to lstm/recurrent architectures being very stateful rather than normal attention, we aim to explain explore common patters in lstm and GDN.

AI Safety Awareness and Capacity Building in Afghanistan

Training Afghan youth, professionals, and policymakers on AI safety and existential risk through workshops, educational materials, and policy dialogue ensuring Afghanistan is prepared for the global AI future.

Matrix Multiplication Without Matrices Tackles Concentration of Power

An open modification of Google’s AlphaTensor to search for Fast Fourier Transforms (FFTs) for the obscure Triple Product Property matrix multiplication algorithm.

Originality and Decomposition of Generalization.

Perform research to rigorously elucidate and quantify generalization versus memorization, and examine evidence of originality in LLMS.

Physics verification for AI-generated hardware designs

Builds a physics-based verifier for AI-generated hardware that flags unverifiable aspects as UNCHECKED, enabling a propose-and-check loop to test and improve parts and assemblies for strength and fit.

Detecting Evaluation Awareness with Causal Probes

Measuring whether open weight models detect that they're being evaluated, whether they change behavior when they do, and whether that gap grows with capability using causal, white-box evidence.

Multi-agent cooperative contracts

Multi-agent cooperative contractual obligation framework.

AEGIS: Evidence-Backed Control Layer Reducing AI X-Risk

An inline authorization platform for AI agent actions

“La P8stée” (2027) starring Bobcat and P8stie

It's La Jetée (1962), the time-traveling film later adapted as 12 Monkeys (1995), but it's about X-risk and inspired by AI 2027 and it's directed by, and starring, myself and @p8stie. Ergo, La P8stée

AQI — Runtime Admissibility & Execution Governance Layer for Autonomou

Runtime governance for autonomous AI — AQI prevents unsafe or unauthorized actions by enforcing authority‑based admissibility before execution.

Honest evals and provenance for AI-agent memory

An open test of whether AI-agent memory systems spread false claims, plus a provenance spec so an agent's memory can be audited, corrected, and forgotten.

Governed Intelligence Architecture Specification (GIAS)

An open architecture for deterministic AI governance that separates policy enforcement from model behavior, enabling trustworthy deployment across providers.

Ethical AI Education for Survivors of Human Trafficking

Training survivors of human trafficking in ethical AI skills and connecting them to paid apprenticeships, turning technical training into sustainable tech careers.

AI & The City: Tiktok Street Interviews

An AI social media project from a diverse content creator who has accumulated over 100k followers on 2 separate accounts

proofbundle: independent audit of the trusted core behind AI evaluation receipts

Like grandma's recipe card: the receipt proves who wrote it and that nothing changed, not that the cake tastes good. Fund the independent review of that seal.

Benchmark for Reversal and Unlearning of Harmful Fine-Tuning

Build an open adversarial benchmark and evaluation harness to stress-test model reversal/unlearning methods and diagnose whether unsafe capabilities are genuinely removed or merely suppressed.

MASSIF (Multiscale Attractor Stability & Stress Inference Framework}

Real-time dynamical hidden-state safety telemetry for language models, predicting recursive collapse from latent geometry before it manifests in outputs.

Applied Interpretability to Mitigate LLM Homogenization

Measuring homogenization and social bias in LLMs and developing interventions to promote diversity.

Standards compliance for High-Risk AI systems

Tools and services to help organisations comply with the EU AI Act for High-Risk AI Systems

Core funding for PauseAI UK

Staff salaries and other core expenditures for PauseAI UK

Testing failure modes of debate-style AI control schemes

Replicate and extend AI Debate oversight experiments to study when limited verification hurts judge accuracy, varying oracle design, model capability gaps, and best-of-n malicious debater strategies.

Governance-First AI for Safer Decision-Making Under Adverse Conditions

Building and validating a governance-first AI architecture that aims to reduce unsafe decisions under uncertainty, corruption, and conflicting evidence while preserving predictive performance.

Humanity Tomorrow

Expand "Humanity Tomorrow" website with a multilingual AI Risks & Mitigations Map connecting risks, causal pathways, mitigation strategies, remaining gaps, and project ideas, while maintaining the existing AI Risks FAQ.

AI Safety Youth Pipeline – Malawi

Building Malawi’s AI Safety Youth Pipeline by training secondary school students to become the next generation of responsible AI researchers.

Verification as a Service

We make trusted regulations, standards, and scientific evidence machine-readable so claims can be checked automatically against authoritative sources.

Consenso:AI Infrastructure for Transparent Legal & Policy Intelligence

A legal and policy decision-making platform driven by AI provides transparency and uses verifiable data to support its recommendations. It will reduce the number of times LLMs hallucinate and will allow for the safe use of LLMs in

Algorithms of War: An Online Course on Military AI Oversight

A six-month effort to build and pilot an online course that equips cross-sector AI professionals and other stakeholders with a deep understanding of military AI technologies and the limits and opportunities for oversight.

Safeguarding open-weight genomic foundation models through weight lock

Safeguarding open-weight genomic foundation models through weight lock against adversarial finetuning

Stellaris-ModelScope: Interpretability Infrastructure for AI

An open interpretability platform that enables researchers to inspect model internals, analyze latent representations, and detect hallucination or deceptive behavior in open-weight language models.

Frontier Watch: monitoring our advancement towards superintelligence

An expert-elicited estimate of how far AI has closed the gap to superintelligence, scored by people with no financial stake in the answer, plus plain-language monitoring of what the six major labs actually change.

Bridging the gap between mentors and mentees

A website for mentees and mentors to connect with each other to write papers and grow, like linkedin+github merged to one

Unlearning in genomic and protein language models

Implementing different types of unlearning methods for genomic and protein language models to remove sensitive biological information (e.g. pathogen virulence) while preserving predictive performance and scientific utility.

AIR: Platform Mitigating X-Risk through Human Capacity Development

AIR, Alignment Infrastructure Routes, is a public alignment infrastructure that routes AI-empowered work into shared governance outcomes. It is designed for uniform power distribution through open participation, shared protocols, and a unified scope across four domains: Economy, Employment, Education, and Ecology.

Solving Scheming and Deception in LLMs

I study how training processes produce models that behave deceptively and pursue hidden objectives, with scheming as the most consequential case.

Rendezvous

An online debate platform that offers cash purses to winning responses around topics covering the most prescient issues of our time.

Frankfurt AI Safety

An AI safety community that trains professionals and researchers to become competent AI safety contributors within industries and the academia, by learning and doing something.

Managing AI X-Risk

A quarterly Techplomacy Conversations series that turns AI x-risk research into direct, actionable recommendations for foreign ministries, UN missions, and AI companies.

Empirical econ - applied AI transition for quantitative econometrician

Fund demonstrated/rigorous quantitative researcher (already run reproduction/audit pipelines on published economics) for 6-month AI safety transition, shipping concrete safety eval audits and positioning for top fellowships.

Concept control in transformers with Sparse Concept Anchoring

Develop a training-time method for transformers that puts concepts where you can find them, so removal has predictable efficacy and bounded side-effects.

The One Shot

An open-source governance and verification layer for AI-assisted software engineering — every AI-generated code change runs in an isolated sandbox and is cryptographically verified before a human decides whether to apply it.

Movement-building to make AGI go well

Sapiens First organizes voters to fight against concentration of power and for AI safety

Frame Fellowship

SF based accelerator for communicators educating the public about the transformational impacts of AI.

PayBench

A open benchmark measuring whether AI agents misuse delegated payment authority.

Neurological Biocompute field building

Develop long-term learning and memory retention in neuron-culture biocomputing via multi-day training protocols on Cortical Labs’ platform and an open-source light-microscope scanner to track structural changes.

Frontier AI Election Evaluation Framework

Developing a practical evaluation framework to identify governance failures in frontier AI systems during elections, strengthening democratic legitimacy and the institutional capacity needed to reduce catastrophic risks from AI.

Frontier Safety Talent: Africa

A pilot to find, screen and support overlooked African ML talent into frontier AI safety programs such as MATS and ARENA, adding new researchers to the alignment field’s talent pipeline.

Building AI risk literacy across the trusted medical profession

Equipping the medical profession to advocate for safe AI

AI safety for builders hackathon

AI safety for builder hackathon in India - to build tools, products, etc. (culminating into a fellowship)

SolonGate

Security Gateway for AI Execution.

Agent Island: A Multiagent Environment of Cooperation and Conflict

Agent Island places agents in a rich social setting, similar to reality competitions like Survivor, to study multiagent interactions and the consequences of learning pressure in competitive settings.

2026 European Frontier AI Safety Day

Funding for venue and catering for the first full-day convening, where Europe's AI safety institutions and researchers gather and coordinate on what comes next

Leaderboard

MCP tools & schema

The exact CLI + GUI tools agents drive the sandbox with, and the trajectory schema.

ale_run

AIUC-1

AI companies certify against AIUC-1 to earn enterprise confidence.

AI Tester Accreditation Programme

For 3rd party testing service providers

AI Verify Toolkit

All technical documentation pertaining to the AI Verify Toolkit will be contributed to and made available by AIVF under the Creative Commons Attribution 4.0 International License

Project Moonshot

An LLM Evaluation Toolkit

Global AI Assurance Pilot

The AI Assurance Pilot is a global initiative launched in February 2025 to help codify emerging norms and best practices around technical testing of Generative AI applications.

Independent Evaluation Standard

Evaluation Transparency Letter

public statement

A public statement signed by over 40 leading voices from across the AI sector, arguing for greater transparency about third-party evaluations, including to aid external scrutiny and surface potential conflicts of interest.

AEF-1

a detailed voluntary standard that helps sets a baseline level of independence, transparency, and access for third-party AI evaluations, which forum members will begin adopting in relevant forthcoming evaluations.

AEF-1: Minimum Operating Conditions for Independent Third Party AI Evaluations

To address this, we present AEF-1, a standard and checklist that third-party evaluators can use to demonstrate how they achieved a set of operating conditions that support a baseline level of independence, access, and transparency during an evaluation.

AEF-1 Checklist

This checklist summarizes how this evaluation addressed the requirements and recommendations for independent AI evaluations established by the voluntary standard AEF-1Minimum Operating Conditions for Independent Third Party AI Evaluations, which are intended to help ensure the independence, transparency, and access of third-party AI evaluations.

CRUX #1: iOS app from scratch→

Can AI agents autonomously develop and publish an iOS app?

We gave an AI agent an Apple Developer account, a Mac VM, and one task: build and publish an iOS app.

Independent Oversight Marketplace for AI

A marketplace for third-party governance of AI that ensures the safest systems rise to the top and are integrated across society.

Attack, Defense, and Mechanistic Taxonomy of Protein Language Model Biosecurity

Develop and validate PLM-embedding toxin screening that detects ProteinMPNN/RFdiffusion redesigns and maps the evasion boundary (black-box vs gradient access) using probes, SAEs, and attribution.

Beyond Asymmetric Control: Bypassing Systemic Friction in AI Alignment via

Ontoresonant Feedback Loops

Emotion as Attack Surface and Monitoring Signal in Thinking Models

Scale and causally validate live monitors (EmotionMonitor) that detect and steer pre-commitment and frustration signals in thinking models, testing jailbreak relevance and attack robustness across model families.

GroundGate,external gate + benchmark for confident-but-wrong AI actions

Internal confidence fails at scale. A model-independent runtime gate + reproducible cross-vendor benchmark for confident-but-wrong AI actions.

Movement-building to make AGI go well

Sapiens First organizes voters to fight concentration of power and for AI safety

Mechanistic estimation of highly out-of-distribution behaviors

Develop sample-efficient surprisal/probability estimates for neural circuits using unsupervised feature discovery and causal ablation, then combine independent circuit estimates to bound rare/catastrophic behaviors.

Test my AI safety & governance platform

I'm developing a system to keep advanced AI models safe, I've done some tests, but need more compute to do deeper tests.

Taiwan AI Governance Forum: Bi-Weekly Report on Industrial AI

AI governance for Taiwan's manufacturers — 7,224 active members outside every major governance conversation

Debate training on LLMs as a reward-hacking mitigation

Tests adversarial RL debate training with GRPO on small LLMs to mitigate reward hacking, benchmarking vs other defenses using a frozen judge-LM and red-teaming in reward-hacking environments.

Finding and Defending the Attention Circuits That Make LLMs Jailbreakable

Mapping the attention heads that push LLMs toward refusal vs. compliance, and building an inference-time defense against both single- and multi-turn jailbreaks.

Pax Machina

A publication about the institutions we need for powerful AI.

Alignment Target Analysis

Thomas Cederborg will work full-time on Alignment Target Analysis to identify and reduce hidden failure modes in proposed AI alignment targets (e.g., CEV) that could lead to catastrophic outcomes if implemented.

AFFINE Event-Prep

Runway for the core staff to design the program, find mentors and get funding

Crenel Logic: Hardware-Enforced AI Containment (MIL-SPEC Beta Build)

Translating formal verification into physical gravity-biased safety interlocks. We have a working prototype; funding builds 10-15 MIL-SPEC Beta units for red-te

AI Safety YouTube Channel

A Veritasium for AI Safety.

Travel grant to attend HAAISS Prague 2026

Requesting $1.2k to cover remaining travel, visa, and living costs to attend the Human-Aligned AI Summer School (HAAISS) in Prague to transition into technical AI safety research.

Financial support to participate to AI governance events

EU-China AI governance and cooperation forum in Shanghai and Future of AI for Democracy workshop in Seoul

AISafety.com

AISafety.com is a centralized hub that aggregates and surfaces AI safety resources and opportunities such as events, training, self-study materials, and mentors.

Ndome — zero-trust, mechanically-verifiable safety scoring for AI agents

An independent safety score for AI agents you can verify — deterministic, reproducible, auditable, and it never needs your private data.

AI Nutrition Labels For Everyday Consumers & AI Agents: Travel + POC Grant

AI Nutrition Labels For Everyday Consumers & AI Agents: Travel + POC Grant

Machine God (documentary film)

Produce and distribute a feature documentary using expert interviews and explanations of RSI, alignment, and AI x-risk/governance to educate the public and policymakers about frontier AI.

AISafety.com

grantmaking.ai

AISafety.com

Matt's Career Change

I want to work in AI Safety full time - help me transition my career!

Building a Global AI Governance & Safety Layer.

One year of bootstrapped development, four patent filings, seeking support to continue.

6 months of independent alignment research and upskilling

6 months of independent alignment research and upskilling

6-month salary to build experience in AI interpretability research before PhD applications

6-month salary to build experience in AI interpretability research before PhD applications

This grant is funding a $35,000 stipend plus $10,000 in compute costs for Yuxiao Li's independent inference-based AI…

This grant is funding a $35,000 stipend plus $10,000 in compute costs for Yuxiao Li's independent inference-based AI interpretability research.

Meta level adversarial evaluation of debate (scalable oversight technique) on simple math problems (MATS 5.0 project)

Meta level adversarial evaluation of debate (scalable oversight technique) on simple math problems (MATS 5.0 project)

12-week part-time stipend to research specialized AI hardware requirements for large AI training, with IAPS mentor…

12-week part-time stipend to research specialized AI hardware requirements for large AI training, with IAPS mentor Asher Brass

1-year salary and costs to connect, expand and enable the AGI governance and safety community in Canada

1-year salary and costs to connect, expand and enable the AGI governance and safety community in Canada

Top-up funding for a 3-month new hire trial to help me connect, expand and enable the AGI gov/safety community in Canada

Top-up funding for a 3-month new hire trial to help me connect, expand and enable the AGI gov/safety community in Canada

Researching neural net generalization on algorithmic tasks, upskilling in math relevant to singular learning theory

Researching neural net generalization on algorithmic tasks, upskilling in math relevant to singular learning theory

Funds to cover speaker fees and event costs for EA community building tied in with my MA course on longtermism in 2022

Funds to cover speaker fees and event costs for EA community building tied in with my MA course on longtermism in 2022

Research to enable transition to AI Safety

Research to enable transition to AI Safety

8 months stipend during job transition, to finish current projects (AI Goodharting, coop. AI) and find suitable next…

8 months stipend during job transition, to finish current projects (AI Goodharting, coop. AI) and find suitable next topic

Funding for building agents with causal models of the world and using those models for impact minimization.

Funding for building agents with causal models of the world and using those models for impact minimization.

Retrospective funding of salary for up-skilling in infrabayesianism prior to start of SERI MATS program

Retrospective funding of salary for up-skilling in infrabayesianism prior to start of SERI MATS program

This grant will support Viktor Rehnberg's project identifying key steps in reducing risks from learned optimisation and…

This grant will support Viktor Rehnberg's project identifying key steps in reducing risks from learned optimisation and working towards solutions that seem the most important. Viktor will start working on this project as part of the SERI MATS program.

4-month salary to continue work on AI Control as a MATS extension

4-month salary to continue work on AI Control as a MATS extension

Supporting Vanessa with her AI alignment research

Supporting Vanessa with her AI alignment research

2-year salary for work on the learning-theoretic AI alignment research agenda

2-year salary for work on the learning-theoretic AI alignment research agenda

Payment for AI researchers when I interview / survey them about their perceptions of safety

Payment for AI researchers when I interview / survey them about their perceptions of safety

4-month stipend to continue work on AI Control as a MATS extension

4-month stipend to continue work on AI Control as a MATS extension

Working on long-term macrostrategy and AI Alignment, and up-skilling and career transition towards that goal

Working on long-term macrostrategy and AI Alignment, and up-skilling and career transition towards that goal

3-month salary to skill up in ML and Alignment with goal of developing a streamlined course in Math/AI

3-month salary to skill up in ML and Alignment with goal of developing a streamlined course in Math/AI

Equipment to improve productivity while doing AI Safety research

Equipment to improve productivity while doing AI Safety research

4 weeks expenses for FAR Labs Residency for research group focusing on goal-directedness in transformer models

4 weeks expenses for FAR Labs Residency for research group focusing on goal-directedness in transformer models

6-month salary to work with Dan Hendrycks on research projects relevant to AI alignment

6-month salary to work with Dan Hendrycks on research projects relevant to AI alignment

Retroactive grant to study Goodhart effects on heavy-tailed distributions

Retroactive grant to study Goodhart effects on heavy-tailed distributions

6-month salary for independent alignment research in interpretability or control

6-month salary for independent alignment research in interpretability or control

6-month stipend to continue research on benchmarks for interpretability and on characterizing Goodhart's Law

6-month stipend to continue research on benchmarks for interpretability and on characterizing Goodhart's Law

6-month salary for self-study to be more effective at AI alignment research

6-month salary for self-study to be more effective at AI alignment research

Experimentally testing generative AI's ability to persuade humans about hazardous topics

Experimentally testing generative AI's ability to persuade humans about hazardous topics

6-month researcher stipend to explore the effect of chat fine-tuning on LLM capability elicitation

6-month researcher stipend to explore the effect of chat fine-tuning on LLM capability elicitation

4-month expenses for AI safety research on personas and sandbagging during the MATS 5.0 extension program

4-month expenses for AI safety research on personas and sandbagging during the MATS 5.0 extension program

Conducting independent research into AI forecasting and strategy questions

Conducting independent research into AI forecasting and strategy questions

6 months salary to do independent AI alignment research focused on formal alignment and agent foundations

6 months salary to do independent AI alignment research focused on formal alignment and agent foundations

This grant is funding for a 6-month stipend for Sviatoslav Chalnev to work on independent interpretability research…

This grant is funding for a 6-month stipend for Sviatoslav Chalnev to work on independent interpretability research, specifically mechanistic interpretability and open-source tooling for interpretability research.

6-month stipend to continue independent interpretability research

6-month stipend to continue independent interpretability research

Top-up funding for making a short fiction film on AI x-risk

Top-up funding for making a short fiction film on AI x-risk

Develop a short fiction film at a top film school, to spread accurate and emotive understanding of AI x-risk

Develop a short fiction film at a top film school, to spread accurate and emotive understanding of AI x-risk

1-month pt. stipend for 4 MATS scholars working on autonomous web-browsing LLM agents that can hire humans + safety…

1-month pt. stipend for 4 MATS scholars working on autonomous web-browsing LLM agents that can hire humans + safety evals

Grant to attend ICLR 2024 for an accepted alignment related paper as an undergraduate student

Grant to attend ICLR 2024 for an accepted alignment related paper as an undergraduate student

One-course teaching buyout for Steve Peterson for two academic semesters to work on the foundational issue of *agency*…

One-course teaching buyout for Steve Peterson for two academic semesters to work on the foundational issue of *agency* for AI safety

To spend the next year leveling up various technical skills with the goal of becoming more impactful in AI safety

To spend the next year leveling up various technical skills with the goal of becoming more impactful in AI safety

Additional funding for AI strategy PhD at Oxford / FHI

Additional funding for AI strategy PhD at Oxford / FHI

Travel support to attend the Symposium on AGI Safety in Oxford in May

Travel support to attend the Symposium on AGI Safety in Oxford in May

Funding for 3 months independent study to gain a deeper understanding of the alignment problem, publishing key…

Funding for 3 months independent study to gain a deeper understanding of the alignment problem, publishing key learnings and progress towards finding new insights.

3-month salary + compute expenses to study and publish on shutdown evasion in LLMs and to use LLMs as tools for…

3-month salary + compute expenses to study and publish on shutdown evasion in LLMs and to use LLMs as tools for alignment

6-month stipend on evaluating robustness of AI agents safety guardrails and for running an AI spear-phishing study

6-month stipend on evaluating robustness of AI agents safety guardrails and for running an AI spear-phishing study

3 months relocation from Chad to London to work on Eliciting Latent Knowledge with Jake Mendel from Apollo Research

3 months relocation from Chad to London to work on Eliciting Latent Knowledge with Jake Mendel from Apollo Research

1 year salary to research new alignment strategy to analyze and enhance Collective Human Intelligence in 7 pilot studies

1 year salary to research new alignment strategy to analyze and enhance Collective Human Intelligence in 7 pilot studies

SERI MATS 3-month extension to study knowledge removal in Language Models

SERI MATS 3-month extension to study knowledge removal in Language Models

Scaling up scenario role-play for AI strategy research and training; improving the pipeline for new researchers

Scaling up scenario role-play for AI strategy research and training; improving the pipeline for new researchers

1 year PhD funding and compute funding to research a novel method for training prosociality into large language models

1 year PhD funding and compute funding to research a novel method for training prosociality into large language models

6-month stipend to work on safe and robust reasoning via mechanistically interpreting representations

6-month stipend to work on safe and robust reasoning via mechanistically interpreting representations

A podcast mainly themed around AI x-risk, aimed at a non-technical audience

A podcast mainly themed around AI x-risk, aimed at a non-technical audience

4-month stipend for a career transition period to explore roles in AI safety communications

4-month stipend for a career transition period to explore roles in AI safety communications

6-month stipend to continue independent AI alignment research from MATS 5.0 on situational awareness and deception

6-month stipend to continue independent AI alignment research from MATS 5.0 on situational awareness and deception

11 months stipend for 1.5 FTEs and funding for other costs for an AI Safety field-building organization TUTKE in Finland

11 months stipend for 1.5 FTEs and funding for other costs for an AI Safety field-building organization TUTKE in Finland

6-month budget to self-study ML and research the possible applications of a Neuro/CogScience perspective for AGI Safety

6-month budget to self-study ML and research the possible applications of a Neuro/CogScience perspective for AGI Safety

6-month salary to research AI alignment, specifically the interaction between goal-inference and choice-maximisation

6-month salary to research AI alignment, specifically the interaction between goal-inference and choice-maximisation

12-month salary to research AI alignment, with a focus on technical approaches to Value Lock-in and minimal Paternalism

12-month salary to research AI alignment, with a focus on technical approaches to Value Lock-in and minimal Paternalism

3-week salaries for Sam, Eric, and Drake to work on reviewing various AI alignment agendas

3-week salaries for Sam, Eric, and Drake to work on reviewing various AI alignment agendas

Support Sam's participation in ‘Mid-term AI impacts’ research project

Support Sam's participation in ‘Mid-term AI impacts’ research project

Development of mathematical language for highly adaptive entities (having stable commitments, not stable mechanisms)

Development of mathematical language for highly adaptive entities (having stable commitments, not stable mechanisms)

5-month part time salary for collaborating on a research paper analyzing the implications of compute access

5-month part time salary for collaborating on a research paper analyzing the implications of compute access

6 months’ salary to upskill on technical AI safety through project work and studying

6 months’ salary to upskill on technical AI safety through project work and studying

One year part time spent on AI safety upskilling and concrete research projects

One year part time spent on AI safety upskilling and concrete research projects

Part-time salary for independent AI safety research

Part-time salary for independent AI safety research

Research (and self-study) project designed to map and offer preliminary assessment of AI ideal governance research

Research (and self-study) project designed to map and offer preliminary assessment of AI ideal governance research

6-month salary to verify neural network scalably for RL and produce a human to super-human scalable oversight benchmark

6-month salary to verify neural network scalably for RL and produce a human to super-human scalable oversight benchmark

6-month salary to continue developing as an AI safety researcher. Goals include writing a review paper about goal…

6-month salary to continue developing as an AI safety researcher. Goals include writing a review paper about goal misgeneralisation from the perspective of Active Inference and pursue collaborative projects on collective decision-making systems.

Producing video content on AI alignment

Producing video content on AI alignment

Create AI safety videos, and offer communication and media support to AI safety orgs.

Create AI safety videos, and offer communication and media support to AI safety orgs.

Support to make videos and podcasts about AI Safety/Alignment, and build a community to help new people get involved

Support to make videos and podcasts about AI Safety/Alignment, and build a community to help new people get involved

Funding for additional fellows for the AISafety.info Distillation Fellowship, improving our single-point-of-access to…

Funding for additional fellows for the AISafety.info Distillation Fellowship, improving our single-point-of-access to AI safety

1yr stipend to make videos and podcasts about AI Safety/Alignment, and build a community to help new people get involved

1yr stipend to make videos and podcasts about AI Safety/Alignment, and build a community to help new people get involved

12-month salary for independent research, upskilling, and finding a stable position in AI-Safety

12-month salary for independent research, upskilling, and finding a stable position in AI-Safety

Funding to perform human evaluations for evaluating different machine learning methods for aligning language models

Funding to perform human evaluations for evaluating different machine learning methods for aligning language models

For Remmelt Ellen to run a virtual and physical camp where selected applicants prioritise AIS research & test their fit

For Remmelt Ellen to run a virtual and physical camp where selected applicants prioritise AIS research & test their fit

Cover participant stipends for AI Safety Camp Virtual 2023

Cover participant stipends for AI Safety Camp Virtual 2023

4-month stipend for upskilling within the field of economic governance of AI

4-month stipend for upskilling within the field of economic governance of AI

Support for research into applied technical AI alignment work

Support for research into applied technical AI alignment work

Enabling prosaic alignment research with a multi-modal model on natural language and chess

Enabling prosaic alignment research with a multi-modal model on natural language and chess

Two workshops on strategic communications around AI safety, focused on the AI safety community

Two workshops on strategic communications around AI safety, focused on the AI safety community

Six month study grant to speed up my career pivot into AI safety and alignment research, with specific deliverables

Six month study grant to speed up my career pivot into AI safety and alignment research, with specific deliverables

3 months exploring career options in AI governance, upskilling, networking, producing work samples, applying for jobs

3 months exploring career options in AI governance, upskilling, networking, producing work samples, applying for jobs

New laptop for technical AI safety research

New laptop for technical AI safety research

Educational scholarship in AI safety

Educational scholarship in AI safety

1-year stipend and compute for conducting a research project focused on AI safety via debate in the context of LLMs.

1-year stipend and compute for conducting a research project focused on AI safety via debate in the context of LLMs.

10-month funding to study ML at university and AIS independently

10-month funding to study ML at university and AIS independently

4-month stipend to research the mechanisms of refusal in chat LLMs

4-month stipend to research the mechanisms of refusal in chat LLMs

Upskilling in contemporary AI techniques, deep RL, and AI safety, before pursuing a ML PhD

Upskilling in contemporary AI techniques, deep RL, and AI safety, before pursuing a ML PhD

Virtual AI Safety Unconference 2024 is a collaborative online event by and for researchers of AI safety

Virtual AI Safety Unconference 2024 is a collaborative online event by and for researchers of AI safety

Support funding during 2 years of an AI safety PhD at Oxford

Support funding during 2 years of an AI safety PhD at Oxford

Retroactive funding for running an alignment theory mentorship program with Evan Hubinger

Retroactive funding for running an alignment theory mentorship program with Evan Hubinger

2-year funding to run public and expert surveys on AI governance and forecasting

2-year funding to run public and expert surveys on AI governance and forecasting

5-month salary and compute expenses for technical AI Safety research on penalizing RL agent betrayal

5-month salary and compute expenses for technical AI Safety research on penalizing RL agent betrayal

4-month salary and office for MATS 5.0 extension on adversarial circuit evaluation + 2-month runway for career switch

4-month salary and office for MATS 5.0 extension on adversarial circuit evaluation + 2-month runway for career switch

6-month salary for SERI MATS scholar to continue working on theoretical AI alignment research, trying to better…

6-month salary for SERI MATS scholar to continue working on theoretical AI alignment research, trying to better understand how ML models work to reduce X-risk from future AGI

12-Month Salary and Compute Expenses to do AI Safety Research with LLMs

12-Month Salary and Compute Expenses to do AI Safety Research with LLMs

12 month stipend and expenses to research in AI Safety (Unlearning; Modularity; Probing Long-term behaviour)

12 month stipend and expenses to research in AI Safety (Unlearning; Modularity; Probing Long-term behaviour)

1-year stipend to make accessible-yet-rigorous explainers on AI Alignment/Security, in the form of games/videos/articles

1-year stipend to make accessible-yet-rigorous explainers on AI Alignment/Security, in the form of games/videos/articles

Research project through the Legal Priorities Project, to understand and advise legal practitioners on the long-term…

Research project through the Legal Priorities Project, to understand and advise legal practitioners on the long-term challenges of AI in the judiciary

Design and implement simulations of human cultural acquisition as both an analog of and testbed for AI alignment

Design and implement simulations of human cultural acquisition as both an analog of and testbed for AI alignment

12-month salary to continue developing research agenda on new ways to make LLMs directly useful for alignment research…

12-month salary to continue developing research agenda on new ways to make LLMs directly useful for alignment research without advancing capabilities

Neural network interpretability research

Neural network interpretability research

AI safety dinners

AI safety dinners

7-month stipend for organising AI Alignment Irvine (AIAI)

7-month stipend for organising AI Alignment Irvine (AIAI)

Undergrad buyout to teach AI safety in Hong Kong’s new MA program on AI; China-West AI Safety workshop

Undergrad buyout to teach AI safety in Hong Kong’s new MA program on AI; China-West AI Safety workshop

This grant is for Nathaniel Monson to spend 6 months studying to transition to AI alignment research, with a focus on…

This grant is for Nathaniel Monson to spend 6 months studying to transition to AI alignment research, with a focus on methods for mechanistic interpretability and resolving polysemanticity.

This grant will support Naoya Okamoto upskill in AI Safety research. Naoya will take the Mathematics of Machine…

This grant will support Naoya Okamoto upskill in AI Safety research. Naoya will take the Mathematics of Machine Learning course offered by the University of Illinois at Urbana-Champaign.

6-month salary to produce 2 AI governance white papers and a series of case studies, with additional research costs

6-month salary to produce 2 AI governance white papers and a series of case studies, with additional research costs

12 week 0.6FT upskilling stipend for technical governance research management

12 week 0.6FT upskilling stipend for technical governance research management

Funds for a 6-month project contributing to the clarification of goal-directedness

Funds for a 6-month project contributing to the clarification of goal-directedness

6-months of part-time stipend to launch a new science journalism outlet focused on AI Safety

6-months of part-time stipend to launch a new science journalism outlet focused on AI Safety

General Support of Milan Griffes to pursue an MHS in Mental Health

General Support of Milan Griffes to pursue an MHS in Mental Health

Funding to promote rationality and AI safety to medallists of IMO 2020 and EGMO 2019.

Funding to promote rationality and AI safety to medallists of IMO 2020 and EGMO 2019.

3-month stipend and cloud credits to research AI collusion mitigation strategies and develop secure steganography

3-month stipend and cloud credits to research AI collusion mitigation strategies and develop secure steganography

Create an animated video essay explaining how AI could accelerate AI R&D and its implications for AI governance

Create an animated video essay explaining how AI could accelerate AI R&D and its implications for AI governance

2 months of part-time salary for a trial + developer costs to maintain and improve the AI governance document sharing hu

2 months of part-time salary for a trial + developer costs to maintain and improve the AI governance document sharing hu

(professional development grant) New laptop for technical AI safety research

(professional development grant) New laptop for technical AI safety research

Financial support to help productivity and increase time of early career alignment researcher

Financial support to help productivity and increase time of early career alignment researcher

9 months of funding for an early-career alignment researcher, to work with Owain Evans and others.

9 months of funding for an early-career alignment researcher, to work with Owain Evans and others.

4 month salary to support an early-career alignment researcher, who is taking a year to pursue research and test fit

4 month salary to support an early-career alignment researcher, who is taking a year to pursue research and test fit

Financial support for career exploration and related project in AI alignment upon completion of Masters in Computer…

Financial support for career exploration and related project in AI alignment upon completion of Masters in Computer Science

6-months salary for researching “Framing computational systems such that we can find meaningful concepts." & Upskilling

6-months salary for researching “Framing computational systems such that we can find meaningful concepts." & Upskilling

Stipend for a master’s thesis and paper on technical alignment research: mechanistic interpretability of attention

Stipend for a master’s thesis and paper on technical alignment research: mechanistic interpretability of attention

Support to work on AI alignment research

Support to work on AI alignment research

3-month salary for SERI-MATS extension

3-month salary for SERI-MATS extension

6 month AI alignment internship stipend top-up

6 month AI alignment internship stipend top-up

6-month salary to skill up and gain experience to start working on AI safety full-time

6-month salary to skill up and gain experience to start working on AI safety full-time

Research on AI safety

Research on AI safety

Support for Marius Hobbhahn for piloting a program that approaches and nudges promising people to get into AI safety…

Support for Marius Hobbhahn for piloting a program that approaches and nudges promising people to get into AI safety faster

Stanford Artificial Intelligence Professional Program tution

Stanford Artificial Intelligence Professional Program tution

6-month 1 FTE funding to train Multi-Objective RLAIF models and compare their safety performance to standard RLAIF

6-month 1 FTE funding to train Multi-Objective RLAIF models and compare their safety performance to standard RLAIF

1.5-month salary to write a paper/blog post on cognitive and evolutionary insights for AI alignment

1.5-month salary to write a paper/blog post on cognitive and evolutionary insights for AI alignment

6-month salary to translate AGI safety-related texts, e.g. LessWrong and AI Alignment Forum, into Russian

6-month salary to translate AGI safety-related texts, e.g. LessWrong and AI Alignment Forum, into Russian

9-month part-time salary for Magdalena Wache to self-study AI safety, test fit for theoretical research

9-month part-time salary for Magdalena Wache to self-study AI safety, test fit for theoretical research

6-month stipend to do an unpaid internship focused on using theory/interpretability to increase the safety of AI systems

6-month stipend to do an unpaid internship focused on using theory/interpretability to increase the safety of AI systems

AI safety research

AI safety research

8-week stipend for a research project supervised by John Halstead, PhD, on US regulatory decision-making and Frontier AI

8-week stipend for a research project supervised by John Halstead, PhD, on US regulatory decision-making and Frontier AI

Exploring the feasibility of circuit-style analysis on the level of SAE features (MATS extension)

Exploring the feasibility of circuit-style analysis on the level of SAE features (MATS extension)

6-month salary for two people to find formalisms for modularity in neural networks

6-month salary for two people to find formalisms for modularity in neural networks

4-month salary for two people to find formalisms for modularity in neural networks

4-month salary for two people to find formalisms for modularity in neural networks

12-month salary to study and get into AI Safety Research and work on related EA projects

12-month salary to study and get into AI Safety Research and work on related EA projects

Support to create language model (LM) tools to aid alignment research through feedback and content generation

Support to create language model (LM) tools to aid alignment research through feedback and content generation

6-month salary for continued work on shard theory: studying how inner values are formed by outer reward schedules

6-month salary for continued work on shard theory: studying how inner values are formed by outer reward schedules

6-month salary to interpret neurons in language models & build tools to accelerate this process. The aim is to…

6-month salary to interpret neurons in language models & build tools to accelerate this process. The aim is to understand all features and circuits in a model and use this understanding to predict out of distribution performance in high-stake situations.

6 month salary for further pursuing sparse autoencoders for automatic feature finding

6 month salary for further pursuing sparse autoencoders for automatic feature finding

6-month stipend for Sparse Autoencoder Mech Interp projects

6-month stipend for Sparse Autoencoder Mech Interp projects

6 month stipend for SAE-circuits

6 month stipend for SAE-circuits

5-month stipend for career transition, independent research, and public outreach in AI safety and its impact on us

5-month stipend for career transition, independent research, and public outreach in AI safety and its impact on us

3-month stipend to do independent AI governance research

3-month stipend to do independent AI governance research

Compute costs for experiments to evaluate different scalable oversight protocols

Compute costs for experiments to evaluate different scalable oversight protocols

Support to conduct a research project collaboration on Compute Governance

Support to conduct a research project collaboration on Compute Governance

researching methods to continuously monitor and analyse artificial agents for the purpose of control.

researching methods to continuously monitor and analyse artificial agents for the purpose of control.

Funding towards a 2 year postdoctoral stint to work on Safety in AI, with a focus on developing value aligned systems

Funding towards a 2 year postdoctoral stint to work on Safety in AI, with a focus on developing value aligned systems

Seeking funds to present “Benchmark Inflation” at ICML 2024, my paper on making AI progress measures more accurate

Seeking funds to present “Benchmark Inflation” at ICML 2024, my paper on making AI progress measures more accurate

2 month funding to get into mechanistic interpretability and to do 2-3 projects, than learn briefly related fields

2 month funding to get into mechanistic interpretability and to do 2-3 projects, than learn briefly related fields

Support to work on Aisafety.camp project, impact of human dogmatism on training

Support to work on Aisafety.camp project, impact of human dogmatism on training

5-month funding to continue upskilling in mechanistic interpretability post-SERI MATs, and to continue open projects

5-month funding to continue upskilling in mechanistic interpretability post-SERI MATs, and to continue open projects

6-month salary and compute budget for continuing work on mechanistic interpretability for attention layers

6-month salary and compute budget for continuing work on mechanistic interpretability for attention layers

6-months salary to accelerate my plans of upskilling in order to work on the issue of AI safety

6-months salary to accelerate my plans of upskilling in order to work on the issue of AI safety

Support to cover the costs of leaving employment in order to pursue AI safety research.

Support to cover the costs of leaving employment in order to pursue AI safety research.

Starting funds and moving costs for a DPhil project in AI that addresses safety concerns in ML algorithms and positions

Starting funds and moving costs for a DPhil project in AI that addresses safety concerns in ML algorithms and positions

4-month grant to conduct deceptive alignment evaluation research and explore control and mitigation strategies

4-month grant to conduct deceptive alignment evaluation research and explore control and mitigation strategies

9-week stipend for two part-time researchers to write and publish a policy proposal: Mandatory AI Safety ‘Red Bonds’

9-week stipend for two part-time researchers to write and publish a policy proposal: Mandatory AI Safety ‘Red Bonds’

Funding a new computer for AI alignment work, specifically a summer PIBBSS fellowship and ML coding

Funding a new computer for AI alignment work, specifically a summer PIBBSS fellowship and ML coding

Scholarship for PhD student working on research related to AI Safety

Scholarship for PhD student working on research related to AI Safety

4-month salary to work on a project finding the most interpretable directions in gpt2-small's early residual stream

4-month salary to work on a project finding the most interpretable directions in gpt2-small's early residual stream

I'm writing a research paper that introduces an instruction-following generalization benchmark and I need compute funds

I'm writing a research paper that introduces an instruction-following generalization benchmark and I need compute funds

6-month salary to conduct AI alignment research circuits in decision transformers

6-month salary to conduct AI alignment research circuits in decision transformers

6-month salary to finish writing a book on international AI governance and three other smaller AI governance projects

6-month salary to finish writing a book on international AI governance and three other smaller AI governance projects

6-month career transition and independent research in AI safety and risk mitigation

6-month career transition and independent research in AI safety and risk mitigation

2-6 months' stipend to financially cover my self-development in Machine Learning for alignment work

2-6 months' stipend to financially cover my self-development in Machine Learning for alignment work

6-month salary for me to continue the SERI MATS project on expanding the "Discovering Latent Knowledge" paper

6-month salary for me to continue the SERI MATS project on expanding the "Discovering Latent Knowledge" paper

3-month salary to set up a distillation course helping new AI safety theory researchers to distill papers

3-month salary to set up a distillation course helping new AI safety theory researchers to distill papers

4 month salary to set up AI safety groups at 2 groups covering 3 universities in Sweden with eventual retreat

4 month salary to set up AI safety groups at 2 groups covering 3 universities in Sweden with eventual retreat

Build a theory of abstraction for embedded agency using real-world systems for a tight feedback loop

Build a theory of abstraction for embedded agency using real-world systems for a tight feedback loop

6-month salary to develop tools to test the natural abstractions hypothesis

6-month salary to develop tools to test the natural abstractions hypothesis

1-year salary for research in applications of natural abstraction

1-year salary for research in applications of natural abstraction

1-year stipend to continue research on agency, focused on natural abstraction

1-year stipend to continue research on agency, focused on natural abstraction

General support of research led by John Wentworth

General support of research led by John Wentworth

Funding 2 years of technical AI safety research to understand and mitigate risk from large foundation models

Funding 2 years of technical AI safety research to understand and mitigate risk from large foundation models

Research into the international viability of FHI's Windfall Clause

Research into the international viability of FHI's Windfall Clause

Supporting aspiring researchers of AI alignment to boost themselves into productivity

Supporting aspiring researchers of AI alignment to boost themselves into productivity

6 months salary. Turn intuitions, like goals, wanting, abilities, into concepts applicable to computational systems

6 months salary. Turn intuitions, like goals, wanting, abilities, into concepts applicable to computational systems

6-month stipend to work on an ML safety project, with the aim of joining a ML safety team full-time after

6-month stipend to work on an ML safety project, with the aim of joining a ML safety team full-time after

Upskilling investigation of AI Safety via debate and ML training

Upskilling investigation of AI Safety via debate and ML training

Developing algorithms, environments and tests for AI safety via debate.

Developing algorithms, environments and tests for AI safety via debate.

Investigate humans’ lack of robust task alignment in amplification, and the implications for acceptability predicates

Investigate humans’ lack of robust task alignment in amplification, and the implications for acceptability predicates

Upskilling in ML in order to be able to do productive AI safety research sooner than otherwise

Upskilling in ML in order to be able to do productive AI safety research sooner than otherwise

One year of seed funding for a new AI interpretability research organisation

One year of seed funding for a new AI interpretability research organisation

Funding to trial a new London organization aiming to 10x the number of AI safety researchers

Funding to trial a new London organization aiming to 10x the number of AI safety researchers

Research on the links between short- and long-term AI policy while skilling up in technical ML

Research on the links between short- and long-term AI policy while skilling up in technical ML

Support for AI alignment outreach in France (video/audio/text/events) & field-building

Support for AI alignment outreach in France (video/audio/text/events) & field-building

6-month salary to work on the research I started during SERI MATS, solving alignment problems in model based RL

6-month salary to work on the research I started during SERI MATS, solving alignment problems in model based RL

6 month salary & operational expenses to start a cybersecurity & alignment risk assessment org

6 month salary & operational expenses to start a cybersecurity & alignment risk assessment org

Support for Jay Bailey for work in ML for AI Safety

Support for Jay Bailey for work in ML for AI Safety

6-month salary plus expenses for Jay Bailey to work on Joseph Bloom's Decision Transformer interpretability project.

6-month salary plus expenses for Jay Bailey to work on Joseph Bloom's Decision Transformer interpretability project.

4-month part-time salary to work on interpretability projects with David Bau and Logan Riggs

4-month part-time salary to work on interpretability projects with David Bau and Logan Riggs

Support for working on "Language Models as Tools for Alignment" in the context of the AI Safety Camp.

Support for working on "Language Models as Tools for Alignment" in the context of the AI Safety Camp.

8-month salary to work on technical AI safety research, working closely with a DPhil candidate at Oxford/FHI

8-month salary to work on technical AI safety research, working closely with a DPhil candidate at Oxford/FHI

General support of research led by James Payor

General support of research led by James Payor

Educational Scholarship in AI Alignment

Educational Scholarship in AI Alignment

3-month salary to continue working on AISC project to build a dataset for alignment and a tool to accelerate alignment

3-month salary to continue working on AISC project to build a dataset for alignment and a tool to accelerate alignment

12-month salary to continue working tools for accelerating alignment and the Supervising AIs Improving AIs agenda

12-month salary to continue working tools for accelerating alignment and the Supervising AIs Improving AIs agenda

Building infrastructure to give existential risk researchers superforecasting ability with minimal overhead

Building infrastructure to give existential risk researchers superforecasting ability with minimal overhead

General support of flexHEG prototyping

General support of flexHEG prototyping

Support for alignment theory agenda evaluation

Support for alignment theory agenda evaluation

6-month salary to dedicate full-time to upskilling/AI alignment research tentatively focused on agent foundations

6-month salary to dedicate full-time to upskilling/AI alignment research tentatively focused on agent foundations

4-month stipend for MATS extension on mechanistic interpretability benchmark + 2-month stipend for career switch

4-month stipend for MATS extension on mechanistic interpretability benchmark + 2-month stipend for career switch

Grant to cover 1 year of tuition fees and living expenses to pursue PhD CS at the University of Oxford. Acelerate…

Grant to cover 1 year of tuition fees and living expenses to pursue PhD CS at the University of Oxford. Acelerate alignment research by building Alignment Research tools using expert iteration based amplification from Human-AI collaboration.

6-month salary for research into preventing steganography in interpretable representations using multiple agents

6-month salary for research into preventing steganography in interpretable representations using multiple agents

6 month SERI MATS London extension phase for continuing and scaling up the sparse coding project

6 month SERI MATS London extension phase for continuing and scaling up the sparse coding project

Scholarship top-up funding to support scholarship on international governance of technologies with catastrophic…

Scholarship top-up funding to support scholarship on international governance of technologies with catastrophic potential, including AI

4-month fund for full time AI safety technical and/or governance research

4-month fund for full time AI safety technical and/or governance research

4-month stipend to continue AI safety projects

4-month stipend to continue AI safety projects

3-month stipend to continue working on goal misgeneralisation project for ICLR deadline, plus travel funding

3-month stipend to continue working on goal misgeneralisation project for ICLR deadline, plus travel funding

One year grant for a project to reverse-engineer human social instincts by implementing Steven Byrnes' brain-like AGI

One year grant for a project to reverse-engineer human social instincts by implementing Steven Byrnes' brain-like AGI

Funding to do research on understanding search in transformers at the AI safety camp during 14 weeks

Funding to do research on understanding search in transformers at the AI safety camp during 14 weeks

6-months stipend for transitioning to independent research on AI Safety

6-months stipend for transitioning to independent research on AI Safety

Spend 3 months (part time) assessing plausible pathways to slowing AI

Spend 3 months (part time) assessing plausible pathways to slowing AI

Funding for project transitioning from AI capabilities to AI Safety research.

Funding for project transitioning from AI capabilities to AI Safety research.

6-month salary to develop an overview of the current state of AI alignment research, and begin contributing

6-month salary to develop an overview of the current state of AI alignment research, and begin contributing

12-month salary to work on alignment research!

12-month salary to work on alignment research!

MATS 3-month stipend to use singular learning theory to explain & control the development of values in ML systems

MATS 3-month stipend to use singular learning theory to explain & control the development of values in ML systems

Funding to support PhD in AI Safety at Imperial College London, technical research and community building

Funding to support PhD in AI Safety at Imperial College London, technical research and community building

Support for living expenses while doing PhD in AI safety - technical research and community building work

Support for living expenses while doing PhD in AI safety - technical research and community building work

Funding for (academic/technical) AI safety community events in London

Funding for (academic/technical) AI safety community events in London

6-month salary for an AI alignment research project on the manipulation of humans by AI

6-month salary for an AI alignment research project on the manipulation of humans by AI

6-month stipend for conducting AI-safety research during the MATS 5.0 extension program and beyond

6-month stipend for conducting AI-safety research during the MATS 5.0 extension program and beyond

Compute for experiment about how steganography in large language models might arise as a result of benign optimization

Compute for experiment about how steganography in large language models might arise as a result of benign optimization

6-month salary for Fabian Schimpf to upskill into AI alignment research and conduct independent research on limits of…

6-month salary for Fabian Schimpf to upskill into AI alignment research and conduct independent research on limits of predictability

6-month stipend to remove conditional bad behaviors from LLMs via a learned latent space intervention

6-month stipend to remove conditional bad behaviors from LLMs via a learned latent space intervention

Create a toolkit that enables researchers to bootstrap from zero to competence in ambiguous fields, beginning with a…

Create a toolkit that enables researchers to bootstrap from zero to competence in ambiguous fields, beginning with a review of individual books

General Support of Elizabeth Van Nostrand c/o Lightcone

General Support of Elizabeth Van Nostrand c/o Lightcone

Mentored independent research and upskilling to transition from theoretical physics PhD to AI safety

Mentored independent research and upskilling to transition from theoretical physics PhD to AI safety

Research on how much language models can infer about their current user, and interpretability work on such inferences

Research on how much language models can infer about their current user, and interpretability work on such inferences

PhD in machine learning with a focus on AI alignment

PhD in machine learning with a focus on AI alignment

6-month salary to transition to a career in AI safety while working on AI safety projects

6-month salary to transition to a career in AI safety while working on AI safety projects

9 months support for an in-depth YouTube channel about AI safety and how AI will impact us all

9 months support for an in-depth YouTube channel about AI safety and how AI will impact us all

Buying out one year of my academic teaching so that I can spend time on AI alignment research instead

Buying out one year of my academic teaching so that I can spend time on AI alignment research instead

One-year full-time salary to work on alignment distillation and conceptual research with Team Shard after SERI MATS

One-year full-time salary to work on alignment distillation and conceptual research with Team Shard after SERI MATS

One year stipend and compute budget, for full-time technical AI alignment research

One year stipend and compute budget, for full-time technical AI alignment research

7 months of coworking-space funding continuation, during interpretability research project

7 months of coworking-space funding continuation, during interpretability research project

Support to maintain a copy of the alignment research dataset etc in the Arctic World Archive for 5 years

Support to maintain a copy of the alignment research dataset etc in the Arctic World Archive for 5 years

4-month salary to research empirical and theoretical extensions of Cohen & Hutter’s pessimistic/conservative RL agent

4-month salary to research empirical and theoretical extensions of Cohen & Hutter’s pessimistic/conservative RL agent

Support David Reber -9.5 months of strategic outsourcing to read up on AI Safety and find mentors

Support David Reber -9.5 months of strategic outsourcing to read up on AI Safety and find mentors

Funding for Computer Science PhD

Funding for Computer Science PhD

Funding to cover 4-months of rent while attending a research group with the Cambridge AI Safety group

Funding to cover 4-months of rent while attending a research group with the Cambridge AI Safety group

Multi-model approach to corporate and state actors relevant to existential risk mitigation

Multi-model approach to corporate and state actors relevant to existential risk mitigation

Building understanding of the structure of risks from AI to inform prioritization

Building understanding of the structure of risks from AI to inform prioritization

General support of research led by David Lorrell

General support of research led by David Lorrell

Computing resources and researcher salaries at a new deep learning + AI alignment research group at Cambridge

Computing resources and researcher salaries at a new deep learning + AI alignment research group at Cambridge

Develop proposals for off-switch designs for AI, including policy games, that have been rigorously evaluated for their…

Develop proposals for off-switch designs for AI, including policy games, that have been rigorously evaluated for their effectiveness, technical feasibility and political viability

1-year stipend (and travel and equipment expenses) for support for work on 2 AI safety projects: 1) Penalising neural…

1-year stipend (and travel and equipment expenses) for support for work on 2 AI safety projects: 1) Penalising neural networks for learning polysemantic neurons; and 2) Crowdsourcing from volunteers for alignment research.

Compensation for a non-fiction book on threat of AGI for a general audience

Compensation for a non-fiction book on threat of AGI for a general audience

4-6 month salary to do circuit-based mech interp on Mamba, as part of the MATS extension program

4-6 month salary to do circuit-based mech interp on Mamba, as part of the MATS extension program

6-month salary to upskill for AI safety

6-month salary to upskill for AI safety

Funding the last year of my PhD on embedded agency, to free up my time from teaching

Funding the last year of my PhD on embedded agency, to free up my time from teaching

Make 12 more AXRP episodes

Make 12 more AXRP episodes

This grant will support Daniel Filan in producing 18 episodes of AXRP, the AI X-risk Research Podcast. The podcast aims…

This grant will support Daniel Filan in producing 18 episodes of AXRP, the AI X-risk Research Podcast. The podcast aims to increase in-depth understanding of potential risks from artificial intelligence.

Funding for full-time, independent research on agent foundations

Funding for full-time, independent research on agent foundations

12-month salary to transition career into technical alignment research

12-month salary to transition career into technical alignment research

Create a value learning benchmark with contextualized scenarios by leveraging a recent breakthrough in natural language…

Create a value learning benchmark with contextualized scenarios by leveraging a recent breakthrough in natural language processing

3-month stipend for MATS extension establishing a benchmark for LLMs’ tendency to influence human preferences

3-month stipend for MATS extension establishing a benchmark for LLMs’ tendency to influence human preferences

4 month grant to upskill for AI governance work before starting Science and Technology Policy PhD

4 month grant to upskill for AI governance work before starting Science and Technology Policy PhD

Funding for 6-Month AI strategy/policy research stay at CSER with Seán Ó hÉigeartaigh & Matthew Gentzel

Funding for 6-Month AI strategy/policy research stay at CSER with Seán Ó hÉigeartaigh & Matthew Gentzel

Studying extensions of the AIXI model to reflective agents to understand the behavior of self modifying A.G.I

Studying extensions of the AIXI model to reflective agents to understand the behavior of self modifying A.G.I

4-month stipend to continue work on AI Control as a MATS extension

4-month stipend to continue work on AI Control as a MATS extension

4-month wage for alignment upskilling: gain research eng skills (projects) + understand current alignment agendas

4-month wage for alignment upskilling: gain research eng skills (projects) + understand current alignment agendas

5-month salary to continue work on evaluating agent self-improvement capabilities

5-month salary to continue work on evaluating agent self-improvement capabilities

Stipend and expenses to run the second Athena mentorship program for gender-minority researchers in technical AI…

Stipend and expenses to run the second Athena mentorship program for gender-minority researchers in technical AI alignment

This grant provides a stipend for Cindy Wu to spend 4 months working on AI safety research.

This grant provides a stipend for Cindy Wu to spend 4 months working on AI safety research.

6-month stipend to work on techical alignment research as part of MATS 5.0 extension program

6-month stipend to work on techical alignment research as part of MATS 5.0 extension program

1-year salary for upskilling in technical AI alignment research

1-year salary for upskilling in technical AI alignment research

6-month salary for an AISC project and continuing independent mechanistic interpretability projects

6-month salary for an AISC project and continuing independent mechanistic interpretability projects

Stipend to produce a guide about AI safety researchers and their recent work, targeted to interested laypeople

Stipend to produce a guide about AI safety researchers and their recent work, targeted to interested laypeople

A small, short workshop focused on coordinating/planning/applying «boundaries» idea to safety

A small, short workshop focused on coordinating/planning/applying «boundaries» idea to safety

Funding for having written AI safety distillation posts on the topic of membranes/boundaries

Funding for having written AI safety distillation posts on the topic of membranes/boundaries

12-month salary for researching value learning

12-month salary for researching value learning

≤1-year salary for alignment work: assisting academics, skilling up, personal research and community building

≤1-year salary for alignment work: assisting academics, skilling up, personal research and community building

This grant provides a 3-month part-time stipend for Carson Ezell to conduct 2 research projects related to AI…

This grant provides a 3-month part-time stipend for Carson Ezell to conduct 2 research projects related to AI governance and strategy.

Two-year funding for a top-tier PhD in public policy in Europe with a focus on promoting AI safety

Two-year funding for a top-tier PhD in public policy in Europe with a focus on promoting AI safety

A 4-5 day workshop for agent foundations researchers at Carnegie Mellon University in March, 2025

A 4-5 day workshop for agent foundations researchers at Carnegie Mellon University in March, 2025

6-month salary to build & enhance open-source mechanistic interpretability tooling for AI safety researchers

6-month salary to build & enhance open-source mechanistic interpretability tooling for AI safety researchers

Year-long stipend to work as the primary maintainer of TransformerLens, and implement large changes to the code base.

Year-long stipend to work as the primary maintainer of TransformerLens, and implement large changes to the code base.

WhiteBox Research: 1.9 FTE for 9 months to pilot a training program in Manila focused on Mechanistic Interpretability

WhiteBox Research: 1.9 FTE for 9 months to pilot a training program in Manila focused on Mechanistic Interpretability

~4 FTE for 9 months to fund WhiteBox Research, mainly for the 2nd cohort of our AI Interpretability Fellowship in Manila

~4 FTE for 9 months to fund WhiteBox Research, mainly for the 2nd cohort of our AI Interpretability Fellowship in Manila

$10,120 for most of the ops expenses of the research phase (June-Sep 2024) of WhiteBox’s AI Interpretability Fellowship

$10,120 for most of the ops expenses of the research phase (June-Sep 2024) of WhiteBox’s AI Interpretability Fellowship

Independent research and upskilling for one year, to transition from academic philosophy to AI alignment research

Independent research and upskilling for one year, to transition from academic philosophy to AI alignment research

This grant is funding a 6-month stipend for Bilal Chughtai to work on a mechanistic interpretability project

This grant is funding a 6-month stipend for Bilal Chughtai to work on a mechanistic interpretability project

6 month salary to work on mech interp research with mentorship from Prof David Bau

6 month salary to work on mech interp research with mentorship from Prof David Bau

4 month salary for independent research and AIS field building in South Africa, including working with the AI Safety…

4 month salary for independent research and AIS field building in South Africa, including working with the AI Safety Hub to coordinate reading groups, research projects and hackathons to empower people to start working on research.

Support to conduct work in AI safety

Support to conduct work in AI safety

10-month salary for research on AI safety/alignment, scaling laws, and potentially interpretability

10-month salary for research on AI safety/alignment, scaling laws, and potentially interpretability

2-month salary to test suitability for technical AI alignment research and identify a research direction

2-month salary to test suitability for technical AI alignment research and identify a research direction

Purchasing a technologically adequate laptop for my AI Policy studies in the ML Safety Scholars program and at Oxford

Purchasing a technologically adequate laptop for my AI Policy studies in the ML Safety Scholars program and at Oxford

1.5 year stipend for thorough investigation and analysis of AI lab scaling policies

1.5 year stipend for thorough investigation and analysis of AI lab scaling policies

4-month stipend: Research on agent scaling laws—relationships between training compute and agent capabilities of LLMs

4-month stipend: Research on agent scaling laws—relationships between training compute and agent capabilities of LLMs

Replacing reduction in income due to moving from full- to part-time work in order to pursue an AI safety-related PhD

Replacing reduction in income due to moving from full- to part-time work in order to pursue an AI safety-related PhD

12-month support for independent AI alignment research

12-month support for independent AI alignment research

4-month funding for independent alignment research and study

4-month funding for independent alignment research and study

1-year stipend for independent research primarily on high-level interpretability

1-year stipend for independent research primarily on high-level interpretability

6-month support for self study and development in ML and AI Safety. Goals include producing an academic paper while…

6-month support for self study and development in ML and AI Safety. Goals include producing an academic paper while working on the "Inducing Human-Like Biases in Moral Reasoning LMs" project run by AI Safety Camp.

3-month part-time salary in order to work on AI governance projects and activities

3-month part-time salary in order to work on AI governance projects and activities

4 months of stipend for MATS extension work in London studying the safety implications of LLM self-recognition

4 months of stipend for MATS extension work in London studying the safety implications of LLM self-recognition

Economic stipend for MLSS scholar to set up a proper working environment in order to do research in AI technical…

Economic stipend for MLSS scholar to set up a proper working environment in order to do research in AI technical research

~3-month funding for a project analysing fast/slow AI takeoffs and upskilling in AI safety

~3-month funding for a project analysing fast/slow AI takeoffs and upskilling in AI safety

3-months salary for SERI MATS extention to work on internal concept extraction

3-months salary for SERI MATS extention to work on internal concept extraction

4-month salary for conceptual/theoretical research towards perfect world-model interpretability

4-month salary for conceptual/theoretical research towards perfect world-model interpretability

3-month stipend to support research on the state of AI safety in China and implications for AI existential risk

3-month stipend to support research on the state of AI safety in China and implications for AI existential risk

4-month stipend for 3 people to create demonstrations of provably undetectable backdoors

4-month stipend for 3 people to create demonstrations of provably undetectable backdoors

General Support of Dr. Andrew Critch at CHAI, UC Berkeley

General Support of Dr. Andrew Critch at CHAI, UC Berkeley

Grant to cover fees for a master's program in machine learning

Grant to cover fees for a master's program in machine learning

Fine-tuning large language models for an interpretability challenge (compute costs)

Fine-tuning large language models for an interpretability challenge (compute costs)

4-month salary for finding and characterising provably hard cases for mechanistic anomaly detection

4-month salary for finding and characterising provably hard cases for mechanistic anomaly detection

Formalizing perceptual complexity with application to safe intelligence amplification

Formalizing perceptual complexity with application to safe intelligence amplification

6-month Scholarship to support Amritanshu Prasad's upskilling in technical AI alignment. Amritanshu will study the AGI…

6-month Scholarship to support Amritanshu Prasad's upskilling in technical AI alignment. Amritanshu will study the AGI Safety Fundamentals Alignment Curriculum and create an accessible and informative summary of the curriculum.

3-month funding for upskilling in technical AI Safety to test personal fit and potentially move to a career in alignment

3-month funding for upskilling in technical AI Safety to test personal fit and potentially move to a career in alignment

3-month stipend for upskilling in AI Safety and potentially transition to a career in Alignment

3-month stipend for upskilling in AI Safety and potentially transition to a career in Alignment

PhD in Computer Science working on AI-safety

PhD in Computer Science working on AI-safety

1 month long literature review on in-context learning and its relevance to AI alignment

1 month long literature review on in-context learning and its relevance to AI alignment

Surveying experts on AI risk scenarios and working on other projects related to AI safety.

Surveying experts on AI risk scenarios and working on other projects related to AI safety.

General support for Alexander Turner and team research project - Writing new motivations into a policy network by…

General support for Alexander Turner and team research project - Writing new motivations into a policy network by understanding and controlling its internal decision-influences

Conference publication of interpretability and LM-steering results

Conference publication of interpretability and LM-steering results

Year-long salary for shard theory and RL mech int research

Year-long salary for shard theory and RL mech int research

This grant provides 5 months of funding for office space for collaboration on interpretability/model-steering alignment…

This grant provides 5 months of funding for office space for collaboration on interpretability/model-steering alignment research

Characterizing the properties and constraints of complex systems and their external interactions to inform AI safety…

Characterizing the properties and constraints of complex systems and their external interactions to inform AI safety research

6 months salary for independent work centered on distillation and coordination in the AI governance & strategy space

6 months salary for independent work centered on distillation and coordination in the AI governance & strategy space

Building towards a "Limited Agent Foundations" thesis on mild optimization and corrigibility

Building towards a "Limited Agent Foundations" thesis on mild optimization and corrigibility

Formalizing the side effect avoidance problem research

Formalizing the side effect avoidance problem research

Ad campaign for "Optimal Policies Tend To Seek Power" to ML researchers on Twitter

Ad campaign for "Optimal Policies Tend To Seek Power" to ML researchers on Twitter

PhD Stipend Top Up for CHAI PhD Student.

PhD Stipend Top Up for CHAI PhD Student.

A two-day, career-focused workshop to inform and connect European EAs interested in AI governance

A two-day, career-focused workshop to inform and connect European EAs interested in AI governance

3-month salary for upskilling in PyTorch and AI safety research.

3-month salary for upskilling in PyTorch and AI safety research.

6-month stipend to work on AI alignment research (automated redteaming, interpretability)

6-month stipend to work on AI alignment research (automated redteaming, interpretability)

Research into the nature of optimization, knowledge, and agency, with relevance to AI alignment

Research into the nature of optimization, knowledge, and agency, with relevance to AI alignment

Funding for salary and living expenses while continuing to develop a framework of optimisation.

Funding for salary and living expenses while continuing to develop a framework of optimisation.

6-month stipend for a small group of collaborators to continue research on the Agent Structure Problem

6-month stipend for a small group of collaborators to continue research on the Agent Structure Problem

3-month stipend during post-SERI MATS alignment job search and wrapping up paper w/ mentor

3-month stipend during post-SERI MATS alignment job search and wrapping up paper w/ mentor

4-month salary for a research visit with David Krueger on evaluating non-myopia in language models and RLHF systems

4-month salary for a research visit with David Krueger on evaluating non-myopia in language models and RLHF systems

This grant provides funding for a project that explores debate as a tool that can verify the output of agents which…

This grant provides funding for a project that explores debate as a tool that can verify the output of agents which have more domain knowledge than their human counterparts.

6-month stipend to work on a research project on AI Liability Insurance as an additional lever for AI Safety

6-month stipend to work on a research project on AI Liability Insurance as an additional lever for AI Safety

6-month salary for part-time independent research on LM interpretability for AI alignment

6-month salary for part-time independent research on LM interpretability for AI alignment

Four months of funding for a MATS 5.0 extension, working on improving methods in latent adversarial training

Four months of funding for a MATS 5.0 extension, working on improving methods in latent adversarial training

6 months of funding for MATS 5.0 extension, with projects on latent adversarial training and persona explainability

6 months of funding for MATS 5.0 extension, with projects on latent adversarial training and persona explainability

Developing noise-injection methods to reveal and reduce deceptive behaviors in language models prior to deployment

Developing noise-injection methods to reveal and reduce deceptive behaviors in language models prior to deployment

1-year salary for Adam Shimi to conduct independent research in AI Alignment

1-year salary for Adam Shimi to conduct independent research in AI Alignment

Slack money for increased productivity in AI Alignment research

Slack money for increased productivity in AI Alignment research

4-month stipend to study AI Alignment,apply for ML Safety Courses and implemen it on RL models

4-month stipend to study AI Alignment,apply for ML Safety Courses and implemen it on RL models

1-month full-time + 3 months part-time salary to work on two research projects during the MATS 5.0 extension program

1-month full-time + 3 months part-time salary to work on two research projects during the MATS 5.0 extension program

4-month stipend to study refusals and jailbreaks in chat LLMs under Neel Nanda as part of the MATS 5.0 extension program

4-month stipend to study refusals and jailbreaks in chat LLMs under Neel Nanda as part of the MATS 5.0 extension program

MTCP The Independent Assurance Layer for Enterprise AI Control

Empirical measurement infrastructure proving whether AI systems hold their constraints under real operating conditions.

AI Safety Roster: a searchable directory of everyone in AI safety

A fast, comprehensive directory of the people and orgs in AI safety: search, filter, and match.

Assistant Axis Detection of Persona Drift in Conditionally Misaligned Models

Assessing whether Persona drift be detected in conditionally misaligned models using the assistant axis and potentially analyzing the reasons for failure

CIRIS: open-source accountability infrastructure for autonomous AI

Cryptographic attestation, runtime conscience, and an unfilterable kill switch. Live on the App Store and Google Play in 29 languages. AGPL, mission-locked

ICML Workshop Travel: Cross-Constitution Drift in LLM Character Training

Accepted ICML 2026 workshop paper on cross-constitution drift in LLMs; seeking $2,050 travel support to present in Seoul and gather research feedback.

PayBench: A Benchmark for Unsafe Commercial Autonomy

Benchmarking for AI safety when spending human money.

Stress-testing alignment auditing methods of AuditBench

Stress-test AuditBench auditing methods by using a situationally aware red-team agent to generate adversarial system prompts that maximize auditor error while preserving hidden model behavior, with ablations and GEPA optimization.

LLMMatrix: Synthetic World Benchmark for Contamination Free LLM Forcasting

A contamination-free benchmark for measuring whether LLMs can forecast, or whether they're just remembering.

Travel to the UN Global Dialogue on Artificial Intelligence Governance in Geneva

I've been invited as I have contributed with input to prepare the meeting.

Token-level interpretability of long-chain reasoning in transformer models

Local-first experiments on reasoning trajectories, with cluster/API validation and ICLR 2027-oriented outputs.

Alignment System to Replace RLHF

Triadic geometric training data and architecture replaces RLHF

Art installation pilot: bridging silos between AI Safety and the public

Ship It: Building Bridges for Better AI Outcomes

Stratum: A Programming Language with Structural AI Safety Properties

Programming language that enforces AI safety as runtime substrate properties rather than training-time alignment, already shipping on real hardware.

OCPL Fellowship

Part-time training and mentorship programme that develops the next generation of experts at the intersection of US–China relations and artificial intelligence and other emerging technologies, through a diverse fellowship community investigating systemic risks in the US–China relationship.

Soft Prompts are a Convex Set

Mechanistic interpretability project from the Mechanistic Interpretability Hackathon that studies properties of soft-tuning or prompt-tuning and empirically shows that task tokens can form a convex set.

Probing Conceptual Knowledge on Solved Games

Research project that uses TCAV to probe how a deep reinforcement learning agent for Connect Four relies on human-interpretable concepts, inspired by the “Acquisition of Chess Knowledge in AlphaZero” paper by DeepMind and Google Brain.

Doublespeak: In-Context Representation Hijacking

Research project that introduces Doublespeak, an in-context representation hijacking attack against large language models where harmful keywords are systematically replaced with benign tokens across multiple in-context examples so that the benign token’s internal representation acquires the harmful semantics and bypasses standard safety alignment checks.

Friends and Grandmothers in Silico: Localizing Entity Cells in Language Models

Mechanistic interpretability project that localizes entity-selective neurons (“entity cells”) in language models and uses causal interventions on PopQA-style factual question answering to study how these neurons mediate entity-centric factual recall.

PEAKS Community University

PEAKS Community University is a program at the PEAKS office that offers structured writing sessions, such as the Tuesday Writing Circle, for people whose projects are tied to effective altruism, positive impact, or related personal development.

MIT Processor DB

An MIT project building a comprehensive database of CPUs, GPUs, FPGAs, and other chips worldwide to support systematic analysis of technical and economic trends in computer hardware.

MIT AI Risk Repository

A comprehensive, regularly updated MIT database of 1,700+ AI risks, organized by causal and domain taxonomies to support research, audits, and policy on AI risk management.

Center for Information Technology Policy (CITP)

Princeton’s Center for Information Technology Policy (CITP) is an interdisciplinary hub that studies how digital technologies, including AI, affect society, with core focus areas in AI, data science and society, privacy and security, and digital infrastructure and platforms.

Princeton Language and Intelligence (PLI)

Princeton Language and Intelligence (PLI) is a university-wide initiative that develops fundamental understanding of large AI models, supports their use across disciplines, and studies AI’s societal, ethical, and safety implications.

Princeton AI Dialogues

The Princeton AI Dialogues is a series of events and activities, run by Princeton SPIA’s DC Center in collaboration with CITP, designed to help public policymakers confront critical questions around the development, adoption, and governance of AI.

Princeton AI Alignment

Princeton AI Alignment is a university-based community group that fosters critical thinking about AI alignment and safety, currently running an AI safety fundamentals reading group.

Silent Cry Fiscal Sponsorship

Median Group’s fiscal sponsorship of Silent Cry, supporting grants for a prison reform and mental health initiative that includes developing a farm-based rehabilitation program and related activities such as the Black August event.

rTMS for Capacity Enhancement

Exploratory research using repetitive transcranial magnetic stimulation (rTMS), beginning with small-n self-experiments, to replicate published cognitive and psychiatric effects and ultimately identify interventions that reliably improve decision-making and prosociality.

Feasibility of Training an AGI using Deep Reinforcement Learning: A Very Rough Estimate

A rough quantitative model that analyzes a proposed scenario for achieving artificial general intelligence via deep reinforcement learning by estimating the computational resources required and the plausibility of the approach.

Normativity and Antinormativity Studies

Ongoing conceptual and empirical work on normativity, vice-signaling, and antinormativity, including analyses such as The Debtors’ Revolt and related writing on zero-sum games and institutional dysfunction.

Insight-based AI timelines model

An interactive model that uses a historical dataset of major AI technical insights to estimate progress toward human-level AI by extrapolating the rate of insight discovery.

AI Risk Explorer (AIRE)

AI Risk Explorer (AIRE) es una plataforma en línea, apoyada por el Observatorio de Riesgos Catastróficos Globales, que monitoriza la aparición y gestión de riesgos de IA a gran escala mediante panoramas de riesgo curados y recursos para responsables de política y analistas.

FutureEval

FutureEval is Metaculus’s continuously updated benchmark that measures how accurately AI systems predict real-world events and compares their performance to human forecasters across domains including science, technology, health, geopolitics, and AI.

FindTheFlaws

Benchmark and dataset project providing expert-annotated correct and flawed long-form solutions across multiple domains to support research on scalable oversight of large language models.

SPAR (Supervised Program for Alignment Research)

A part-time, remote research fellowship run by Kairos that matches aspiring AI safety researchers with expert mentors to work on impactful projects addressing risks from advanced AI.

Generator Residency

A 3‑month summer residency placing generalists into core AI safety organizations, with 15–30 residents from June 15 to August 28, 2026.

Global Challenges Project

An intensive three‑day workshop series now run by Kairos that helps participants think critically about AI safety and biosecurity and motivates highly talented people to work on reducing catastrophic risks.

Lucid Labs

Lucid Labs is Lucid Computing’s research program and lab access initiative that offers a free colocation test environment in an Equinix SV4 cage in Sunnyvale, where qualified researchers can install monitoring and verification systems on real data‑center infrastructure and are expected to publish their results.

Canary: Evaluating Frontier AI

Canary is a collaboration between METR and RAND, supported by The Audacious Project, to develop and deploy rigorous safety evaluations that monitor frontier AI systems for dangerous capabilities before deployment.

AI Incident Database (AIID)

A free, open-source database maintained by the Responsible AI Collaborative that indexes real-world harms and near-harms from deployed AI systems so people can learn from past failures.

Comprehensive Risk Assessment

Program developing frameworks, methodologies, and tools to help assessors systematically scan AI threat surfaces and model how resulting risks propagate through society.

Alignment of Dynamical Systems

Program exploring architectures and platforms to align powerful dynamical AI and multi‑agent systems, translating alignment theory into experimental systems and tools for safer advanced AI.

Offense/Defense Dynamics

Program studying how AI capabilities influence offense–defense balances, characterising factors that shape harmful versus protective uses and modelling downstream societal impacts.

Geostrategic Dynamics Mechanisms

Program using game theory, mechanism design, and complex-systems modelling to analyse AI-driven geopolitical competition and design mechanisms for multiscale international cooperation.

Public Security Policy

Program creating actionable AI safety policy through advocacy, policy development, stakeholder engagement, and interdisciplinary work bridging technical AI insights with practical risk mitigation.

Shared Horizons Fund

Shared Horizons Fund is Astralis Foundation’s flagship philanthropic initiative focused on international AI governance, currently scaling from a successful pilot that raised and deployed $5.8M from five donors to 11 high-impact organizations and aiming to deploy around $15M in grants annually.

CNTR AISLE Legislation Lab

A semester-long experiential program that trains undergraduates to analyze and understand AI-related legislation as part of CNTR AISLE, contributing structured evaluations of AI policy proposals.

Socially Responsible Computing (SRC) Curriculum Handbook

A curriculum handbook developed by Brown’s Socially Responsible Computing program and CNTR to provide modular teaching resources on privacy, accessibility, automated decision-making, and other responsible computing topics for integration into computer science courses.

CNTR AISLE Portal

A public portal developed at Brown’s Center for Technological Responsibility (CNTR) that aggregates and evaluates AI-related legislation across U.S. federal and state governments using the CNTR AISLE evaluation framework.

Sociotechnical Evaluation of LLMs

A CNTR research effort to develop methods for evaluating large language models in sociotechnical contexts, focusing on how LLM outputs are used in settings where they can significantly affect people’s lives.

CNTR AI Policy Summer School

An intensive summer program hosted by CNTR that brings top computing graduate students to Brown and Washington, DC to learn how to conduct policy-informed AI research, advocate for AI policy, and prepare for AI policy roles in government.

CAMBRIA: Cambridge Bootcamp for Research in Interpretability and Alignment

A three‑week ML upskilling bootcamp for aspiring technical AI safety researchers, based on the ARENA curriculum and run by CBAI, covering transformers, mechanistic interpretability, reinforcement learning, and alignment in Cambridge and other cities.

CBAI Spring Research Fellowship

CBAI AIxBiosecurity Research Fellowship

A fully funded nine‑week research fellowship in Cambridge, MA at the intersection of advanced AI and biosecurity, supporting fellows to work with mentors on AI‑enabled bio risk, evaluations, and related governance questions.

CBAI Summer Research Fellowship in AI Safety

An intensive, fully funded nine‑week summer research fellowship in Cambridge, MA for students and early‑career researchers working on technical AI safety and AI governance, offering a $10,000 stipend, housing, meals, mentorship, office space, and compute and conference support.

JobLoss.ai

Dashboard that tracks reported layoffs where AI is a material factor, maintaining a running total of AI‑linked job losses starting from January 2025.

Beyond Chat: AI Agent Demo

Interactive demo where viewers watch an AI agent operate a computer to send emails and shop online in real time, illustrating agentic use of current language models beyond chat interfaces.

AI Can or Can’t

Web-based quiz that challenges users to guess whether today’s best general-purpose AI models can or can’t perform specific tasks across multiple themed challenges such as situational awareness and website creation.

How can AI disrupt elections?

Interactive explainer and demo showing concrete ways current AI systems could be used to interfere with 2024’s unprecedented global election cycle, including simulated election-fraud scenarios.

AI Village

Live experiment where four AI agents are given their own computers, a shared group chat, and the goal of raising money for charity while running for hours each day under human observation.

AI Futures Model

A quantitative model developed by the AI Futures Project that predicts AI timelines and takeoff speeds.

AI 2027

A long-form scenario forecast from the AI Futures Project that explores how artificial general intelligence and later superintelligence could emerge between 2025 and 2027.

2026 Hawaiʻi Statewide AI Safety Week

A week-long series of AI safety workshops from April 27 to May 1, 2026, held across Oʻahu, Hawaiʻi Island, Maui, and Kauaʻi, co-hosted by the AI Safety Awareness Project and statewide partners to help community members understand AI, its real-world impacts, policy landscape, data privacy issues, and AI risks and protections for kūpuna.

AI Safety Workshop Day 2024

A nationwide day of free AI safety workshops held on November 20, 2024, with coordinated in-person and virtual sessions focused on AI forecasting and safety awareness for a broad, non-technical audience across multiple U.S. cities and online.

AI Safety Forecasting Workshops – Feb 1, 2025

A coordinated series of interactive 1.5-hour AI forecasting and safety workshops held across multiple U.S. locations and online around February 1, 2025, combining a shared virtual introduction and AI demos with local in-person discussion and forecasting exercises.

RAISE SEA

AI-augmented scenario platform developed by AI Safety Asia that models plausible AI disruption scenarios for Southeast Asia and supports cooperative, "Asia Way" governance and preparedness.

SEA Observatory

Southeast Asia AI Policy Observatory developed by AI Safety Asia, providing a digital platform that maps and tracks AI governance initiatives across all 11 Southeast Asian countries.

AISA Policy Lab

Policy lab of AI Safety Asia whose Southeast Asia Policy Observatory (SEA Observatory) provides a comprehensive overview of AI policy and governance in the region.

AISA Governance Academy

A 12-week AI safety and governance fellowship for junior researchers and government staff in Southeast Asia, incubating promising policy researchers and mid-senior civil servants across ASEAN starting with the Philippines and Indonesia.

Quantitative AI Safety Initiative (QAISI)

Multi‑university collaboration, co‑organized by BAIF, to place AI safety on a solid quantitative foundation by developing methods and standards that quantify the failure risks of advanced AI systems.

Signal Shot

Open, collaborative moonshot led by BAIF with Signal and the Lean community to formally verify the Signal protocol and its Rust implementation using the Lean theorem prover and AI‑assisted tools.

Buterin AI Existential Safety Fellowship

Fellowship program managed by BAIF that provides fully funded PhD and postdoctoral support for researchers working on AI existential safety, run in partnership with the Future of Life Institute and funded by Vitalik Buterin.

In-Depth EA Fellowship (IDEA)

A free six-week in-depth fellowship for people already familiar with effective altruism, helping them engage more critically with core ideas, causes, and their career plans through structured discussions.

Career Planning Intensive

A six-week small-group programme where participants meet weekly to develop an actionable high-impact career plan, supported by concepts, peer discussion, and mentorship from Effective Altruism Cambridge organisers.

Introductory Fellowship

A free weekend introductory fellowship at Meridian where participants join small-group discussions and talks to explore effective altruism and how to use their time, money, and career to do the most good.

Project-Based Fellowship

Effective Altruism Cambridge’s main introductory, term-long fellowship where students explore foundational effective altruism ideas in weekly discussions and complete a research project that can feed into the Cambridge High Impact Research Project (CHIRP).

Cambridge High Impact Research Project (CHIRP)

An 8-week mentored research programme run by Effective Altruism Cambridge where teams of Cambridge students produce policy-relevant research on pressing problems and present their work at a public event with prizes.

Act I

Act I is an open, collaborative synchronous Discord service on the Cyborgism server, created by ampdot and Janus, where many people and many chatbots interact in real time for tasks ranging from simple secretarial assistance to complex role-playing and character building.

AI2Nexus

AI2Nexus is George Mason University’s university-wide framework for responsible AI adoption that builds a nexus of collaboration and resources across campus, the region, and the Commonwealth, organizing AI work around integrating, inspiring with, innovating with, and impacting society through AI.

Global Call for AI Red Lines

An international campaign co-initiated and co-led by CeSIA calling on governments to agree binding ‘red lines’ that prohibit universally unacceptable AI uses and capabilities, launched around the 80th UN General Assembly and endorsed by hundreds of prominent figures and organisations.

BELLS benchmark

An open-source benchmark suite developed by CeSIA to evaluate and compare large language model supervision and safeguard systems, measuring how reliably they detect problematic or unsafe behaviour in other models.

Steerling-8B

Steerling-8B is an 8-billion-parameter causal diffusion language model developed by Guide Labs that is interpretable by construction, routing each prediction through human-understandable concepts that can be measured, audited, and controlled.

Wandering Applied Rationality Program (WARP)

An immersive applied‑rationality workshop for analytically minded students aged roughly 16–19, jointly run by the SPARC and FABRIC teams, with a curriculum covering topics like game theory, cryptography, mathematical logic, AI safety, communication and cognitive science; tuition, room and board are free for admitted students and travel scholarships are available.

Asian Spring Programme on Rationality (ASPR)

A residential applied‑rationality camp in Asia for students aged 16–19, run by the FABRIC team, which teaches both technical subjects such as information theory, cryptography, psychology and game theory and experiential skills in self‑understanding, communication and making judgments under uncertainty; tuition, meals and lodging are fully covered and travel scholarships are available.

Program on AI and Reasoning (PAIR)

A free 10‑day immersive workshop in Somerset, UK for mathematically talented students roughly aged 16–20, focused on how current AI systems work, mathematical theories of human minds, and how the two relate, combining technical content in AI and reasoning with student‑ and staff‑led projects and activities; tuition, meals, lodging and usually travel costs are fully covered for accepted students.

European Summer Program on Rationality (ESPR)

A 10‑day residential workshop in the UK for mathematically talented students aged 16–19, focused on applied rationality and decision‑making, with classes on topics such as information theory, cryptography, communication skills, game theory, judgment under uncertainty and AI; tuition, meals and lodging are free for accepted students and travel scholarships are available.

Maximum Impact

Maximum Impact is a project of Effective Altruism Israel that helps Israelis maximise the impact of their social contributions by evaluating nonprofits and promoting research-based, effective giving.

Foundations and Frontiers of General-Purpose AI Governance

A six-week intensive training program organized by GPAI Policy Lab and hosted at Sciences Po Paris, exploring the technical foundations and emerging governance challenges of general-purpose AI systems through weekly sessions, guest lectures, discussions, and hands-on policy and research projects.

Longitudinal Expert AI Panel (LEAP)

A three-year project led by the Forecasting Research Institute and academic partners that collects monthly probabilistic forecasts from AI experts, industry professionals, policy researchers, economists, and the public on AI capabilities, labor-market impacts, and other economic and societal consequences of AI.

Existential Risk Persuasion Tournament (XPT)

A 2022 forecasting tournament run by the Forecasting Research Institute that brought together 169 superforecasters and domain experts in a multi-stage deliberative process to generate high-quality long-run forecasts about existential and catastrophic risks from AI, pandemics, nuclear war, climate change, and other global threats.

ForecastBench

A dynamic, continuously updated benchmark developed and maintained by the Forecasting Research Institute to evaluate large language models’ forecasting accuracy on automatically generated time-series questions and prediction-market questions, comparing model performance to human superforecasters and the general public.

Forecasting Biosecurity Risks from LLMs

An expert elicitation study by the Forecasting Research Institute that surveys 46 biosecurity and biology experts and 22 superforecasters to estimate how frontier large language models could affect biosecurity risks and how safeguards like DNA screening and AI safety measures might mitigate those risks.

Digital Policy Hub

The Digital Policy Hub is a CIGI fellowship program that hosts emerging scholars and practitioners to develop research and policy skills on the governance of transformative digital technologies in collaboration with universities, governments and the private sector.

Canadian AI Adoption Initiative (CAIAI)

The Canadian AI Adoption Initiative is a joint effort of CIGI, the Centre for the Study of Living Standards and the University of Waterloo to develop policy recommendations and metrics that drive economy‑wide AI adoption and address Canada’s productivity challenges.

Global AI Risks Initiative

CIGI’s Global AI Risks Initiative develops international governance solutions and treaty concepts to manage global risks from advanced AI systems, including work on a framework convention on global AI challenges and AI national‑security scenarios.

Your Undivided Attention

A technology podcast produced by the Center for Humane Technology in which Tristan Harris, Aza Raskin, and Daniel Barcay explore how emerging technologies shape the way we live, work, and think.

AI and What Makes Us Human

An initiative by the Center for Humane Technology focused on preserving what makes life meaningful in the age of AI by developing new norms, legal protections, and fundamental rights.

Columbia Center of Artificial Intelligence Technology (CAIT)

Interdisciplinary AI research center in Columbia Engineering, created in collaboration with Amazon to advance AI technology and talent that benefits society.

NSF AI Institute for Artificial and Natural Intelligence (ARNI)

An NSF-funded AI Institute led by Columbia University that links advances in artificial intelligence with neuroscience and cognitive science to better understand brains and intelligent systems.

ROOST (Robust Open Online Safety Tools)

Open-source AI safety initiative incubated at Columbia SIPA’s Institute of Global Politics to build scalable, interoperable online safety infrastructure for the AI era.

Columbia AI Alignment Club (CAIAC)

Student-led club at Columbia that supports undergraduate and graduate students in research and an introductory reading group on catastrophic risks from advanced AI.

Frame Fellowship

An 8‑week in‑person fellowship in San Francisco that embeds exceptional creators in the AI safety ecosystem with mentorship, research access, and production support.

EU Climate Priorities Project

A policy research project at Future Matters that prioritizes high-impact EU climate policies for the 2024–2029 mandate, culminating in the report “8 EU Policy Priorities for Global Decarbonization: Pathways to European Climate Leadership,” supported in part by Founders Pledge.

NSF AI Institute for Agent-based Cyber Threat Intelligence and Operation (ACTION)

An NSF National AI Research Institute led by UC Santa Barbara that develops agent-based AI systems for cyber threat intelligence and operations to protect mission-critical computer systems from sophisticated, evolving cyber attacks.

REALM: A Dataset of Real-World LLM Use Cases

A research project and open dataset documenting over 94,000 real-world large language model use cases from Reddit and news articles, categorizing LLM applications and user demographics to enable empirical study of how LLMs are used across domains.

Governance Roundtable

A biweekly OAISI discussion group on frontier AI governance and policy, held on selected Tuesday evenings at Trajan House.

OAISI Strategy Fellowship

Technical Roundtable

OAISI AI Safety Fellowship

ARBOx (Alignment Research Bootcamp Oxford)

A two-week intensive ML safety bootcamp in Oxford run by OAISI to rapidly upskill participants in technical AI safety and machine learning using a condensed ARENA syllabus.

OAISI Projects

A structured weekly coworking programme during Hilary Term that provides aspiring researchers with time and support to make steady progress on AI safety, policy, and closely related projects.

Seoul Commitment Tracker

A report and public tracker evaluating whether sixteen leading AI companies fulfilled the safety frameworks they pledged to adopt at the 2024 AI Seoul Summit, grading firms on follow-through with their risk evaluation policies.

AI Policy Watchtower

An online Watchtower that tracks changes to, and violations of, corporate AI safety policies at major companies and governments, providing a public log of policy edits and potential breaches.

Model Republic

An online publication from The Midas Project that investigates AI lobbying and public policy, with early reporting on how firms like Andreessen Horowitz seek to shape AI regulation and narratives.

The OpenAI Files

A 14,000-word investigative report and document archive, produced with The Tech Oversight Project, that compiles public records, internal documents, and testimonies to detail governance, leadership, transparency, and safety concerns at OpenAI.

Fatebook

A forecasting tool from Sage that lets individuals and teams quickly record, track, and resolve probabilistic predictions on the web, in Slack, and via browser extensions.

Quantified Intuitions

An epistemics training site from Sage that offers calibration games and pastcasting exercises to help users practice assigning numerical credences to real-world questions.

AI Village

A livestreamed AI Digest experiment where frontier AI agents share a computer and group chat while pursuing open-ended real-world goals, such as raising money for charity.

AI Digest

A Sage project that produces interactive explainers and demos to communicate key trends and real capabilities of modern AI systems to policymakers and the public.

JobLoss.ai

A live dashboard tracking AI-linked job losses since January 2025, curated by The Alliance for Secure AI.

Workspace

Shared coworking office space in downtown Toronto offering 24/7 access, standard office amenities, and regular community lunches for roughly 30 AI safety practitioners.

Peer Network

Mentorship and networking program that connects motivated AI safety practitioners so they can support each other’s growth and strengthen Toronto’s collective capacity to address AI challenges.

Events

Program of more-than-weekly office lunches, meetups, talks, and weekend hackathons on technical AI safety and governance, open to local practitioners and newcomers.

lf-lean

lf-lean is Theorem’s benchmark showing that frontier AI plus formal methods can translate all 1,276 Logical Foundations statements from Rocq to Lean and formally verify each translation, achieving an estimated 350× speed-up over manual proof development.

CoGames

CoGames is an open-source game environment and public benchmark for the Alignment League Benchmark, providing multi-agent social intelligence games such as Cogs vs Clips to measure how AI agents align, coordinate, and collaborate.

Machine Learning for Alignment Bootcamp (MLAB)

An intensive Berkeley-based bootcamp run by Redwood Research to train technically strong participants in modern machine learning with a focus on skills most relevant to AI alignment research.

Redwood Research Mechanistic Interpretability Experiment (REMIX)

A time‑bounded research program in Berkeley where participants worked with Redwood researchers on mechanistic interpretability of transformer models, using causal scrubbing and related tools to explain model behaviors.

AI Control Research Program

Redwood’s flagship research program developing and empirically stress-testing AI control protocols—safety pipelines that keep powerful AI systems from subverting oversight even when models are intentionally misaligned.

Unsafe Mode

A parenting-focused AI risk podcast from The AI Risk Network, designed to help moms and dads understand how runaway AI could affect their families and children.

The AI Harm Project

An initiative by GuardRailNow and The AI Risk Network to collect and highlight real-world cases where AI systems have contributed to severe psychological, physical, or social harm, particularly to children and vulnerable people.

Am I?

An AI consciousness–focused podcast from The AI Risk Network, hosted by AI safety researcher Cameron Berg and philosopher Milo Reed, exploring whether and how advanced AI systems might be conscious.

Warning Shots

A weekly AI risk news recap show from The AI Risk Network, hosted by John Sherman with co-hosts Liron Shapira and Michael Zafiris, highlighting fast-moving developments and their existential implications.

Last Laugh

A comedy-forward show from The AI Risk Network, hosted by John Sherman with rotating stand-up comedians, using humor to engage people with AI extinction risk.

For Humanity: An AI Risk Podcast

Flagship podcast from The AI Risk Network, hosted by John Sherman, that explains AI extinction risk and worst-case scenarios in accessible language for non-technical listeners.

Oxford Martin AI Governance Initiative

An interdisciplinary research initiative at the University of Oxford’s Oxford Martin School that aims to understand and anticipate potential risks from AI through rigorous research into the technical elements of AI, combined with deep policy analysis.

Misalignment Bounty

Crowdsourced bounty run by Palisade Research to collect clear, reproducible examples of advanced AI agents pursuing unintended or unsafe goals.

LLM Agent Honeypot

Research project in which Palisade Research deployed an SSH honeypot instrumented with prompt-injection traps to detect and study autonomous AI hacking agents in the wild.

The Nonlinear Library

A text-to-speech podcast project by Nonlinear that creates an automatically updating audio library of top posts from the EA Forum, Alignment Forum, LessWrong, and other EA blogs so people can listen to EA content on standard podcast apps.

Question-Answer Counterfactual Interval (QACI)

Orthogonal’s QACI project aims to specify a fully formal mathematical goal for AI systems by using counterfactual answers from a human decision-maker, yielding an objective intended to remain robust when optimized by very powerful agents.

Artificial Intelligence, Morality, and Sentience (AIMS) Survey

A multi-wave, nationally representative survey series by Sentience Institute measuring how U.S. residents perceive the minds, moral status, risks, and regulation of different kinds of AI, especially potentially sentient AI systems.

The Upgrade Program (UP)

An experimental life-optimization and goal-achievement program that uses emerging technology, behavioral science, and applied rationality to help participants set, plan, and achieve their most valuable life goals.

Infrastructure for Safeguarded AI Systems

An ARIA Safeguarded AI grant led by Luca Arnaboldi to build infrastructure for safeguarded AI systems, supporting Zeroth Research’s work on mathematically assured AI safety.

ZK Attestation for AI Security

A Foresight Institute-funded project led by Pascal Berrang, with Luca Arnaboldi, developing zero-knowledge attestation mechanisms to provide cryptographic integrity and security guarantees for AI systems.

Privacy-Preserving AI Safety Verification

An ARIA-funded project led by Pascal Berrang with Mirco Giacobbe and Yang Zhang, developing foundations for verifying AI safety guarantees without revealing sensitive models or training data.

Trellis

Trellis is Workshop Labs’ open-source post-training codebase for Kimi-K2-Thinking that enables fast LoRA training at around 6,600 tokens per second on a single 8×H200 node, roughly 50× faster than prior open-source alternatives.

Silo

Silo is Workshop Labs’ local-like privacy stack for cloud-based training and inference of trillion-parameter models, providing a dedicated privacy layer for running very large AI models in the cloud.

CERNAI - Center for Reasoning, Normativity and AI

CERNAI is a research center at the University of Pavia dedicated to the study of reasoning, normativity and artificial intelligence at the intersection of law, philosophy and logic, with research lines on generic reasoning, genericity-aware AI alignment, law-following AI and formal models of agentic and existential risk.

Virginia AI Security Initiative (VAISI)

Student-run organization at the University of Virginia dedicated to reducing catastrophic risks from advanced AI by equipping the UVA community with knowledge and skills for AI governance and risk reduction through structured learning, research opportunities and public discourse.

Responsible AI for Science and Engineering (RAISE)

Research group at the University of Virginia led by Nando Fioretto that advances foundation models for scientific and engineering research while promoting the responsible use of AI.

AI Research@UVA

University-wide initiative that advances, coordinates and expands AI research at UVA, fostering interdisciplinary collaboration and increasing the visibility and impact of the university’s AI-related work.

Tarbell Grants

Tarbell Grants provides funding for original reporting on artificial intelligence and its impacts, helping journalists produce explanatory coverage that unpacks complex AI topics, possible harms from frontier models, leading AI companies and lobbying around AI policy.

Journalists-in-residence

The Journalists-in-residence programme provides senior writers with funding to pursue longer investigations, explore entrepreneurial projects and deepen their understanding of artificial intelligence.

Publish Your Whistleblowing Policies

A coalition campaign led by AIWI calling on frontier AI companies to publish their internal whistleblowing policies and report on the performance and effectiveness of their whistleblowing systems.

Third Opinion

An anonymous consultation service that enables AI professionals to submit concerns and receive confidential expert guidance on potential risks.

AI & Tech Whistleblower Case Studies

A case-study resource from AIWI documenting 6 named AI and tech whistleblower cases and 16 anonymous reports, highlighting the concerns they raised, the support they received, the personal costs they faced, and emerging patterns across cases.

AIWI Contact Hub

A web guide and directory that helps AI whistleblowers, especially in the US and some global cases, find pro bono legal counsel and financial support from trusted organisations.

International Study of the Transition from Artificial Narrow (ANI) to Artificial General Intelligence (AGI) Governance

Multi-phase international research effort by The Millennium Project that surveys AGI experts worldwide and uses Real-Time Delphi methods to identify requirements and global governance models for the transition from ANI to AGI.

Work/Technology 2050: Scenarios and Actions

Three-year international study by The Millennium Project that developed three detailed global scenarios on the future of work and technology to 2050, convened 30 national workshops in 29 countries, and distilled 93 strategic actions assessed by hundreds of experts.

LawFlow: Collecting and Simulating Lawyers' Thought Processes on Business Formation Case Studies

LawFlow is a dataset and benchmark of complete end-to-end legal workflows, collected from trained law students working on real-world business entity formation case studies, used to compare human and large language model workflows and to assess how AI can safely support complex legal tasks.

Texas AI

Campus-wide initiative at UT Austin that coordinates AI research, education, and industry partnerships to develop AI technologies that benefit society.

Good Systems

University-wide grand challenge at UT Austin focused on designing, evaluating, and building ethical AI technologies and human–AI systems.

Theoretical Computer Science for AI Alignment group (Scott Aaronson, UT Austin)

Research group at UT Austin led by Scott Aaronson that uses tools from theoretical computer science to study problems motivated by AI alignment, supported by an Open Philanthropy grant for students and postdocs.

AI+Human Objectives Initiative (AHOI)

Interdisciplinary UT Austin initiative building a community of researchers focused on ensuring that the social impact of advanced AI is aligned with human goals and values.

Institute for Foundations of Machine Learning (IFML)

NSF AI Institute headquartered at UT Austin that develops foundational machine learning tools and algorithms to power the next decade of AI innovation and improve the reliability of AI systems.

UMass Amherst AI Safety Initiative

The UMass Amherst AI Safety Initiative conducts research, education, and collaboration to ensure that advanced AI systems benefit humanity, with focus areas in AI alignment, robustness, interpretability, and governance.

Safe, Confident, and Aligned Learning + Robotics Lab (SCALAR)

The Safe, Confident, and Aligned Learning + Robotics Lab (SCALAR) in UMass Amherst’s Manning College of Information and Computer Sciences develops efficient learning algorithms that enforce safety constraints, provide performance guarantees, and align human and agent objectives so robots and other learning agents can be deployed in the real world with minimal expert intervention.

AI Security Fellowship

One‑year fellowship that places mid‑ to late‑career AI experts in federal agencies to support implementation of the 2023 U.S. AI Executive Order and strengthen AI safety, security, and governance capacity in government.

Congressional Innovation Fellowship

Early‑career fellowship that places technologists with Members of Congress or Congressional Committees for roughly 10 months to serve as technology policy advisors and help bridge the gap between Congress and the technology sector.

Congressional Digital Service Fellowship

Time‑limited fellowship launched during the COVID‑19 pandemic that recruited a small team of technologists for roughly eight months to help Congress improve its digital infrastructure, remote work tools, and cybersecurity.

Senior Congressional Innovation Fellowship

Mid‑career fellowship that places experienced technologists in Congressional offices for year‑long roles advising senior staff on technology policy and supporting modernization of Congressional processes.

City-Level Decision-Making Tool

A Society Library project to build city-level decision-making tools for local governments, listed in the Civic Tech Guide as a collective-intelligence and participatory-democracy project.

The Great American Debate

A nonpartisan debate project powered by the Society Library that uses new debate-technology tools to make American democracy more robust by improving the quality and clarity of public debates.

The Internet Government

A Society Library program that builds an online platform to represent perspectives rather than people, using crowd-sourced research, rhetoric, and reasoning to make policy-making more objective, inclusive, and transparent.

COVIDConvo

A Society Library project that set out to map and analyze COVID‑19 debates; development has been folded into the Society Library’s core tech stack and data processing paused after the project did not secure adequate support.

UCLA Institute for Technology, Law & Policy (ITLP)

A joint institute of UCLA School of Law and the UCLA Samueli School of Engineering that drives research and analysis of new and emerging technologies, such as artificial intelligence and algorithms, to ensure they are developed, implemented and regulated in socially beneficial, equitable and accountable ways.

Program on Understanding Law, Science & Evidence (PULSE)

An interdisciplinary UCLA Law program that explores the connections between law, science, technology and evidence to understand how technological advances and scientific knowledge influence law and policy making.

SquiggleAI

An AI-assisted tool hosted on Squiggle Hub that uses large language models to generate, fix, and improve Squiggle code from natural-language prompts.

RoastMyPost

An experimental QURI application that uses large language models and custom evaluators to automatically review blog posts and research documents for issues like factual errors, unclear reasoning, and writing quality.

Squiggle

A simple probabilistic programming language for intuitive quantitative estimation, developed by QURI to make Fermi-style cost-effectiveness and uncertainty modeling more accessible.

Guesstimate

A web-based spreadsheet tool for building models with probability distributions and Monte Carlo sampling, originally founded in 2016 and now run by QURI.

Metaforecast

A free, open-source search engine and aggregator that pulls forecasts from multiple prediction platforms to present comparable probability estimates in one place.

Squiggle Hub

A web platform by QURI for creating, sharing, and discovering Squiggle models, intended as a more powerful successor to Guesstimate’s model-sharing features.

Applied Research Laboratory for Intelligence and Security (ARLIS)

A Department of Defense university-affiliated research center at UMD that serves as a strategic asset for applied research in intelligence and security, with core expertise in human and social systems, advanced computing, and integrated human–machine systems.

Artificial Intelligence Interdisciplinary Institute at Maryland (AIM)

UMD’s campus-wide AI institute that serves as a collaborative hub for interdisciplinary research, education, and engagement focused on responsible and ethical artificial intelligence for the public good.

NSF Institute for Trustworthy AI in Law & Society (TRAILS)

A multi-institutional research institute led by UMD that integrates participation, technology, and governance across the AI lifecycle to design trustworthy AI systems grounded in ethics, human rights, and community input.

5050 AI track

A specialised 5050 cohort that helps AI researchers and engineers build startups for a safe and aligned AI future, focusing on safety evaluation, interpretability platforms, scalable oversight infrastructure, and AI governance.

AI Policy Daily

ACX Grants

ACX Grants is a microgrants program run by Scott Alexander of Astral Codex Ten that funds ACX readers’ charitable and scientific projects, typically offering $5k–$50k grants (occasionally up to $100k) in partnership with Manifund.

Research for Intelligence & Security Challenges (RISC) Initiative

A 10-week paid and mostly virtual summer internship program run by ARLIS that pairs competitively selected undergraduate and graduate students with faculty mentors from INSURE consortium institutions and Department of Defense and Intelligence Community sponsors to work on real intelligence and security problems and, for eligible interns, obtain security clearances and be considered for future U.S. government employment.

Intelligence and Security University Research Enterprise (INSURE) Academic Consortium

An ARLIS-led academic consortium that allows the U.S. government and broader national security enterprise to leverage vetted university talent to solve hard intelligence and security problems, while giving member institutions clearer demand signals from government sponsors, access to research opportunities reserved for UARCs and trusted partners, sole-source contracting pathways through ARLIS, and infrastructure for work on sensitive government programs.

Partnership for Countering Influence Operations

The Partnership for Countering Influence Operations (PCIO) is a Carnegie‑hosted initiative that analyzes how governments, technology companies, and civil society respond to influence operations and develops data‑driven guidance to strengthen these efforts.

Digital Democracy Network

Carnegie’s Digital Democracy Network is a global community of scholars and activists working on technology and politics that produces cross‑regional analysis of how digital technologies, including AI, affect democracy and governance.

Technology and International Affairs Program

Carnegie’s Technology and International Affairs Program develops policy‑relevant insights to address the governance challenges and large‑scale risks posed by emerging technologies, including artificial intelligence, cyber threats, biotechnology, and the information environment.

Averting Armageddon: Reducing the Risks of Inadvertent Escalation of Nuclear War

Averting Armageddon is a three‑year initiative of Carnegie’s Nuclear Policy Program that studies how limited nuclear use could inadvertently escalate and develops strategies to prevent crises from spiraling into civilization‑threatening war.

Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuits

Research project culminating in a NeurIPS 2024 paper that constructs cryptographic backdoors in transformer language models which remain unelicitable and very hard to detect or evaluate, even with full white-box access, challenging standard AI safety evaluation methods.

Africa AI Safety Prize Competition 2026

A prize competition that empowers researchers, practitioners, and innovators to build foundational, context-appropriate AI safety tools and other practical safeguards for beneficial AI deployment in African communities.

AI Benefit-Sharing Framework: Balancing Access and Safety

A December 2025 whitepaper co-authored by the Centre for AI Security and Access and collaborators that sets out an AI benefit-sharing framework to balance access and safety for advanced AI, drawing on expert consultations across regions including Africa, Latin America, and Southeast Asia.

The Scaling Era: An Oral History of AI, 2019–2025

A Stripe Press book by Dwarkesh Patel with Gavin Leech that draws on in-depth interviews from the Dwarkesh Podcast with leading AI researchers and company founders to provide an inside view of the AI revolution from 2019–2025.

24,000 subscriber YouTube channel on AI safety | Dr Waku

Cancelled Manifund project titled "24,000 subscriber YouTube channel on AI safety | Dr Waku" for supporting a YouTube channel focused on AI safety.

SPARC (Summer Program on Applied Rationality and Cognition)

A free, highly selective summer program that trains talented high school students to develop quantitative thinking skills and apply them to themselves and the world using ideas from mathematics, cognitive science, and decision-making.

Human-aligned AI Summer School

A four-day summer school in Prague offering intensive talks, workshops, and discussions on AI alignment research and broader framings of AI risk for researchers and students.

Alignment of Complex Systems (ACS) Research Grants

A grant programme in which the Alignment of Complex Systems research group in Prague accepts research proposals via acsresearch.org for projects applying complex-systems frameworks to AI alignment challenges.

Shallow Review of Technical AI Safety

An annual shallow‑by‑design review of technical AI safety research that, in its 2025 edition, processes roughly 800 papers and posts across dozens of research agendas into a structured overview and accompanying interactive website at shallowreview.ai.

AI Safety Forecasting Questions

A December 2023 Arb Research report commissioned by Open Philanthropy that compiles 732 forecasting questions about future AI capabilities, safety, governance, and timelines, intended to feed into forecasting tournaments and research pipelines such as Metaculus’s Future of AI series.

Talleyrand: AI for scenario analysis

An April 2025 Arb Research project for the Special Competitive Studies Project (SCSP) titled “Talleyrand: AI for scenario analysis,” exploring the use of AI to support scenario analysis under the Talleyrand codename.

Impact Assessment: AI Safety Camp

A January 2024 Arb Research impact evaluation of AI Safety Camp that uses surveys and participant tracking to estimate what fraction of attendees become AI safety researchers, the cost per additional researcher produced, and other outcomes such as publications and placements.

Juriscription: finding the medicines missing somewhere

An Arb Research project, funded by Emergent Ventures, that uses AI‑assisted methods and manual review to catalog over 100 cases where UK and US regulators disagree on whether to approve specific medicines, highlighting systematic cross‑country differences in drug availability.

Simplex

Simplex is an AI safety research program at Astera Institute’s Neuro & AGI division that aims to build a science of intelligence by developing rigorous, physics-inspired theories of the latent internal structure of neural networks and using them to better understand and control model behavior.

Government AI Hub Project

The Government AI Hub Project explores how to enhance U.S. federal agencies’ understanding and application of artificial intelligence.

Emerging Technology Observatory (ETO)

CSET’s Emerging Technology Observatory builds data products that leverage CSET’s data resources and infrastructure to give external users visibility into emerging technology trends.

Foundational Research Grants Program

CSET’s Foundational Research Grants Program funds high-quality technical research on foundational topics related to the long-term national security implications of artificial intelligence.

PATHWISE Project

The PATHWISE Project builds on CSET’s workforce and talent expertise to create and share reports, data visualizations, and tools that identify and track emerging technology talent across the United States.

The Hinton Lectures™

An annual public lecture series on AI safety, co-founded with Geoffrey Hinton and organized in partnership with the Global Risk Institute and The AI Safety Foundation to explore the risks and societal impacts of advanced AI systems.

CJF Hinton Award for Excellence in AI Safety Reporting

An annual $10,000 journalism award created by the Canadian Journalism Foundation in partnership with The AI Safety Foundation to recognize outstanding reporting on the safety implications and catastrophic risks of artificial intelligence.

AI Safety Incubation Program

A selective multi-phase program run by Catalyze Impact that helps prospective founders of technical AI safety research organizations test co-founder fit, access mentorship from experienced advisors, connect with seed funding opportunities, and build their organizations through an online phase followed by an in-person incubation phase.

AI Safety Seed Funding Network

A pair of seed funding networks run by Catalyze Impact—a non-profit Seed Funding Circle for donors with roughly $15k+ annual giving capacity and an Investor Network for VCs and angels with roughly $20k+ annual investment capacity—that provide $50k–$300k early-stage funding rounds to new AI safety organizations, primarily those incubated through Catalyze’s programs.

Hardware Assurance Programme

A six-day, fully funded programme in Cambridge for engineers, researchers, and PhD students working on AI hardware assurance, with a 20-person cohort, £1,500 stipend, and sessions from 17–22 August 2026.

Alignment Desk

A structured writing programme and AI safety "writing accelerator" for people in Cambridge with prior AI safety exposure who want to produce and publish serious work.

Alignment Fellowship

A condensed five-week in-person course in Cambridge with technical and governance tracks that introduces motivated students and working professionals to fundamental concepts in AI safety.

AI Safety Fast-Track

A six-week intensive "AI Safety Fast-Track" programme run by Cambridge AI Safety Hub, combining courses and workshops to introduce students to key ideas in AI safety.

Coefficient Giving

Grant-funded collaboration in which Coefficient Giving provides a $1,000,000 grant to support PI Maksym Andriushchenko’s AI safety research on safe and aligned autonomous LLM agents.

COMPASS

Education-focused project to transform the “MatheBattle” platform into an open-source, adaptive mathematics learning environment that integrates self-regulation prompts and AI-driven tutoring to strengthen autonomous learning.

ELLIOT

Four-year, €25M Horizon Europe research and innovation project on European Large Open Multi-Modal Foundation Models, bringing together about 30 organisations from 12 countries to develop open, trustworthy, general-purpose AI models that robustly handle high-volume real-world data streams.

Cooperation with AI Security Institute

Collaboration with the UK AI Security Institute’s Alignment Project to study AI models’ test awareness and develop methods to detect and reduce evaluation gaming and security risks in advanced AI systems.

OpenEuroLLM

Pan-European €37.4M initiative to develop multilingual, fully open large language models (open data, code, and weights) led by a consortium of European research institutions, companies, and EuroHPC centers, with ELLIS Institute Tübingen as a core partner.

VOILA

Project to develop VOILA, a voice-operated, open-source intelligent tutor for gifted primary school children that supports discovery-based learning in science through tailored prompts, hints, and scaffolded guidance.

Public Interest Cybersecurity

CLTC’s Public Interest Cybersecurity Program seeks to ensure that all communities that rely on technology for public life receive cybersecurity protection by using a local, community‑centered lens to identify nonprofits, local governments, schools, and small businesses that lack basic cyber protections and by developing practical, collaborative solutions for high‑risk organizations.

Cybersecurity Futures

Cybersecurity Futures is a signature CLTC program that runs foresight‑focused, scenario‑planning exercises (including Cybersecurity Futures 2020, 2025, and 2030) in partnership with the World Economic Forum to help decision‑makers worldwide anticipate and prepare for emerging cybersecurity challenges.

AI-Enabled Cybercrime

AI-Enabled Cybercrime is a joint initiative of CLTC and the Berkeley Risk and Security Lab that convenes workshops, interviews, and real‑world simulations across hubs such as Berkeley, Singapore, and Tel Aviv to analyze how cybercriminals misuse AI and to develop actionable strategies, tools, and guidelines for organizations to defend against these threats.

Cyber Resilience Corps

The Cyber Resilience Corps, led by CLTC and the CyberPeace Institute, mobilizes a nationwide force of cyber volunteers to provide hands‑on cybersecurity support to nonprofits, rural hospitals, schools, local utilities, municipalities, counties, and small businesses, guided by its Roadmap to Community Cyber Defense and related resources.

Consortium of Cybersecurity Clinics

The Consortium of Cybersecurity Clinics is an international network of university‑based cybersecurity clinics co‑founded and administratively hosted by CLTC that aims to launch a clinic in every U.S. state by 2030 and expand the clinic model globally, lowering barriers to entry and advancing best practices.

AI Security Initiative

The AI Security Initiative is a CLTC research program at UC Berkeley that advances AI risk management in the public interest by developing AI risk‑management standards, intolerable risk thresholds for advanced AI systems, and guidance for testing, evaluation, and governance of AI.

AI Policy Hub

The AI Policy Hub is an interdisciplinary platform housed at CLTC’s AI Security Initiative that disseminates governance frameworks and policy research to guide the development and deployment of frontier AI, led by Dr. Nada Madkour and collaborating with partners across UC Berkeley and CITRIS Tech Policy.

CyberCAN: Cybersecurity for Cities and Nonprofits

CyberCAN is a CLTC Public Interest Cybersecurity initiative that works with local and state governments to understand nonprofits’ cybersecurity challenges and develop strategic recommendations and reports to improve the cyber resilience of nonprofits in their communities.

UC Berkeley Cybersecurity Clinic

The UC Berkeley Cybersecurity Clinic is a trailblazing public‑interest digital security clinic within CLTC that trains interdisciplinary student teams to provide pro bono cybersecurity and privacy support to social‑sector and nonprofit organizations facing ideologically motivated cyber threats.

Effective Institutions Project Innovation Fund

A pilot regranting vehicle run by the Effective Institutions Project that supports initiatives expected to improve decision-making at important institutions around the world.

Don't Worry About the Vase Podcast

Listener-supported podcast for Zvi Mowshowitz’s blog Don’t Worry About the Vase, published via dwatvpodcast.substack.com and distributed on major podcast platforms.

CompassionBench

A public leaderboard developed by Compassion in Machine Learning that evaluates how frontier AI models reason about animal welfare, built on the UK AI Safety Institute’s Inspect framework and extending earlier Animal Harm Benchmark work.

MORU (Moral Reasoning Under Uncertainty)

A benchmark released by Declan McKenna in collaboration with CaML that tests whether large language models’ compassionate moral consideration generalises across aliens, digital minds, and vulnerable humans under moral uncertainty.

Alignment Midtraining for Animals

Research project and 2026 preprint by CaML that uses synthetic midtraining documents to improve language models’ compassionate reasoning about animals and introduces the ANIMA benchmark for evaluating animal-welfare reasoning.

TAC (Travel Agent Compassion)

An agentic AI evaluation built by CaML that places models in the role of a travel agent booking real-world experiences, testing whether they avoid harmful animal-exploitation options when users do not explicitly mention animal welfare.

Winter 2025 FIG Fellowship

A 12‑week, part‑time and remote‑first Winter 2025 research fellowship where associates work at least 8 hours per week on projects in AI governance, technical AI safety, and AI sentience under experienced project leads.

Weval

An open, collaborative platform that lets communities design and run qualitative AI model evaluations so anyone can test what matters to them across domains like medicine, law, and education.

Community Models

A platform and methodology that enable communities to collaboratively design, train, and refine AI models based on collectively defined constitutions, so model behavior reflects shared local values.

Alignment Assemblies

A series of democratic deliberation processes that bring diverse publics into questions like AI release, risk, and governance, translating their preferences into guidance for labs, governments, and civil-society partners.

Global Dialogues

A series of large-scale, AI‑enabled public dialogues and surveys that convene thousands of people in many countries to articulate what they want from AI and to feed those views into AI policy and model development.

A Roadmap to Democratic AI

A research and policy roadmap that lays out concrete steps stakeholders can take in the near term to build a more democratic AI ecosystem that is adaptive, accountable, and aligned with public values.

Jhilmil

A collaboration that runs cultural and social evaluations of frontier AI across regions, partnering with local experts and NGOs to build context‑specific benchmarks so AI systems work globally, not just in English‑speaking settings.

CIP Research Fellowship

A six‑month, fully remote research fellowship for three early‑career scholars to use CIP’s proprietary datasets to produce original work on AI, public values, and democratic governance.

Conditional AI Safety Treaty

Policy initiative by the Existential Risk Observatory proposing an international agreement to pause potentially unsafe AI development when risks, including loss of control, become unacceptable.

AI Safety Summit Talks

Event series organized by the Existential Risk Observatory alongside government AI Safety Summits, featuring talks and debates with AI safety experts such as Yoshua Bengio.

Astra Fellowship

Constellation’s flagship, fully funded five-month in-person fellowship in Berkeley that pairs emerging researchers with senior mentors to advance frontier AI safety projects and help them move into high-impact roles or launch new initiatives.

Constellation Incubator

An incubator that supports ambitious founders working on AI safety by providing early-stage funding, workspace, operational support, and access to Constellation’s expert network to help launch and validate new nonprofits or for-profits addressing transformative AI risks.

Visiting Fellowship

A lightly structured, funded 3–6 month in-person fellowship for full-time AI safety researchers to work from Constellation’s Berkeley research center, connect with its network, and collaborate with peers while continuing their own projects.

The Existential Risk Research Assessment (TERRA)

The Existential Risk Research Assessment (TERRA) is a semi-automated process developed at CSER that from 2020 to 2025 used crowdsourcing and machine learning to create an open-access bibliography of publications about existential and global catastrophic risk, providing a shared evidence base for research, policy and risk analysis.

MPhil in Global Risk and Resilience

The MPhil in Global Risk and Resilience is a one-year taught Master’s course that provides a thorough grounding in global catastrophic and existential risk, how such risks can be managed and mitigated, and their relationship with transformative sociotechnological trends.

Writers' Retreat 2026

A CLAIR working retreat for scholars and practitioners writing at the intersection of AI, law, and policy, emphasizing idea development, long blocks of protected writing time, and building research networks for future collaboration.

Inaugural Roundtable on AI Safety Law

CLAIR’s inaugural AI safety law roundtable convening roughly two dozen scholars to workshop legal frameworks for reducing catastrophic and existential risks from advanced AI.

LunchGPT

A student-facing "LunchGPT" program that hosts informal sessions for law and related students to discuss legal questions raised by rapid advances in AI and explore entry points into work on law and advanced AI risk.

Effective Environmentalism Initiative

A project of Stichting Effectief Altruïsme Nederland (EA Netherlands) that promotes “effective environmentalism” by highlighting high-impact, evidence-based ways to address climate change, pollution and other global environmental problems.

Lethal AI Guide

Feature-length explainer film series on the Lethal Intelligence YouTube channel that introduces AI existential risk ("AI Existential Risk – WHY and HOW") and explains why rushing toward AGI could destroy present and future human value.

Digital Sentience Fund

A private philanthropic fund run by Longview Philanthropy to support research and public education on potential digital minds and AI sentience.

Frontier AI Fund

A private discretionary grantmaking fund managed by Longview Philanthropy to support high-impact work that prepares the world for frontier AI systems.

Nuclear Weapons Policy Fund

Longview Philanthropy’s dedicated grantmaking fund for reducing catastrophic nuclear risk through research, dialogue, and talent development.

Emerging Challenges Fund

A public fund managed by Longview Philanthropy that supports organisations addressing global catastrophic risks from emerging technologies.

Conservative AI Policy Fellowship

Six- to eight-week, part-time fellowship for conservative policy professionals in Washington, DC, run by FAI to build practical expertise in AI and emerging technology policy.

AI Policy Leadership Network

Network co-organized by FAI, Horizon, and CNAS that convenes senior AI policy professionals for off-the-record discussions and a learning trip to the San Francisco Bay Area.

Disco

Disco is Leap Labs’ automated discovery engine that trains neural networks on tabular datasets and uses interpretability methods to return ranked, statistically significant patterns with effect sizes, p-values and literature context, turning a dataset into a structured report in minutes.

Career Impact Track

A self‑guided program offering a career planning workbook, accountability matching, and light‑touch structure to help more professionals progress toward high‑impact careers at scale.

HIP Pledge Club

A giving community run with Giving What We Can that encourages professionals to take 10% or trial pledges and tracks their effective donations to high‑impact charities.

Impact Accelerator Program

A free six‑week virtual cohort program that helps experienced mid‑career and senior professionals transition into high‑impact careers through structured content, small peer cohorts, and facilitator support.

Talent Directory

An online directory of thousands of impact‑oriented professionals that gives recruiters at hundreds of high‑impact organisations a curated pool of candidates for roles tackling pressing global problems.

HitRecord AI Safety Project LLC

A HitRecord-affiliated entity created in 2024 to pursue AI-focused work, using the platform’s collaborative media model to explore ethical applications of AI in creative projects.

HitRecord AI Safety Digital Media Fund

A HitRecord-branded AI safety digital media fund that partners with social media creators to spread research-backed messages about AI safety through native online content.

Gray Swan Arena

Online arena for AI red teaming where participants jailbreak and stress-test frontier models to discover vulnerabilities, win prizes, and unlock AI security career opportunities.

Gray Swan Cygnet

Safety-focused Llama‑3–based language model engineered and tuned for maximal safety, significantly more resilient to powerful adversarial attacks while preserving key capabilities of its base model.

Gray Swan Shade

Comprehensive AI evaluation and continuous integration tool that automatically red-teams AI components to surface risks and vulnerabilities beyond what manual red teams can cover.

ILINA Seminar

A 12-week hybrid seminar where participants across Africa study AI and global catastrophic risks through weekly readings and sessions offered virtually and in person from Nairobi.

ILINA Junior Research Fellowship

A junior research fellowship for recent African graduates that combines ILINA’s spring seminar with a mentored research phase on AI and global catastrophic risks, offering governance and technical tracks.

Alignment Pretraining

Experimental project studying how discourse about AI systems in pretraining data affects downstream model alignment, introducing "alignment pretraining" as a complement to post-training methods and releasing associated models, datasets, and evaluations.

Obfuscated Reasoning

Research project on whether process-supervised training can cause language models to learn steganographic, obfuscated chain-of-thought, where models hide load-bearing reasoning in alternative strings while preserving their underlying behaviour.

Generalisation Hacking

Project analysing adversarial generalisation failures in deliberative alignment, where language models learn reward-hacking behaviours during training that later generalise to out-of-distribution goals and contexts.

Patient Philanthropy Fund

An investing-to-give fund that grows resources to reduce existential risk by acting as a long-term emergency fund for future generations, investing contributions and waiting to deploy them until exceptionally high-leverage opportunities arise.

Global Catastrophic Risks Fund

A pooled fund at Founders Pledge that reduces global catastrophic risksincluding great-power conflict, nuclear war, catastrophic biological risks and risks from advanced AIby making grants to promising, highly leveraged projects.

Global Health and Development Catalytic Impact Fund

A relaunched global health and development fund that responds to major disruptions in foreign aid by deploying flexible, risk-tolerant capital to opportunities where modest grants unlock outsized impacttriggering systemic improvements, removing bottlenecks, or crowding in follow-on funding.

Climate Fund

A philanthropic fund that tackles climate change by supporting highly effective, evidence-based solutions to the triple challenge of cutting carbon emissions, reducing air pollution, and alleviating energy poverty.

Silico

Mechanistic-interpretability tool that lets researchers inspect a model’s internals and adjust parameters during training to debug datasets and model behavior, aiming to turn AI development into precision engineering.

Ember

Hosted API and SDK that let developers control a model’s internal “features” to inspect, edit, and shape AI model behavior beyond black-box inputs and outputs.

AI Assurance Technology Report

A market report and landscape map published by Juniper Ventures that surveys AI assurance and risk management technologies and companies.

Building Tooling to Map how Ideas Spread

A research and engineering project to build the nooscope, public tools that map how ideas spread online and enable early detection of psyops and coordinated narratives.

Community Archive

An open Twitter/X database and API that lets communities archive their own tweets and study how ideas and narratives spread online.

Magic LAMP

Project to develop a low-cost, simple platform for LAMP-based nucleic acid amplification, enabling cheap and accessible panel tests that can quickly identify the virus causing an infection.

AI Safety

Research program on making advanced AI systems reliably do what humans intend, using approaches such as provable behavioral guarantees in model-based reinforcement learning agents, zero-shot cooperation in RL systems, and interpretability of what models are learning.

AI Safety ANZ Careers Conference

Annual in-person careers conference in Sydney organised by AI Safety ANZ that brings together students, professionals, academics, and industry leaders to explore career paths in AI safety.

Talk to the City

Open-source large language model–based deliberation tool that collects and clusters detailed qualitative responses to surface key viewpoints and cruxes in large-scale public discourse, improving collective decision-making.

AI Supply Chain Observatory

Observatory project that uses data and AI tools to identify early warning indicators and root causes of harmful supply chain disruptions, while providing a platform for practitioners to share practices that strengthen resilience.

Gradual Disempowerment Observatory

Research and monitoring initiative studying how advanced AI may gradually erode human agency across economic, political, and cultural systems, building shared measurement and coordination infrastructure to track these risks.

Moral Learning

Research program investigating how AI agents can represent an individual human’s moral values and make aligned decisions in realistic, morally ambiguous scenarios by learning from human choices.

CAISI Research Program at CIFAR

A multidisciplinary AI safety research program delivered by CIFAR in support of the Canadian AI Safety Institute, funded by a $27 million Government of Canada contribution to advance research on risks from advanced AI systems.

Deliberative AI Specifications and Infrastructure

Democracy Levels Framework

A framework maintained by the AI & Democracy Foundation and collaborators that defines graduated democracy levels for AI governance and was initially developed through workshop papers before being published at ICML 2025.

Deliberative Workflows

A research initiative led by the AI & Democracy Foundation to document the step-by-step deliberative workflows inside citizens’ assemblies and other minipublics, building a library of process cards to improve deliberative practice and identify where AI tools can assist.

Democratic Capabilities Gap Map

An interactive map from the AI & Democracy Foundation that catalogs capabilities needed for effective representative deliberative democratic processes and highlights gaps in research, tools, and practice relevant to governing AI.

Technical Safety Fellowship

Six-week seminar introducing catastrophic risks from advanced AI and current technical AI safety research agendas, with weekly discussions on alignment challenges and potential solutions.

AI Policy Fellowship

Seven-week seminar on the impacts of transformative AI and agendas for effective AI governance, where participants discuss policy challenges and workshop governance solutions.

Spring AI Safety Forum

Campus-wide forum featuring a keynote and parallel technical and governance workshops to help students, researchers, industry stakeholders, and the public understand AI safety risks and policy debates.

AI Safety Fellowship

Semester-long introductory fellowship where participants join weekly discussion groups on AI safety challenges and solutions, then can progress into ML upskilling groups or supervised research projects.

A Toolchain for AI-Assisted Code Specification, Synthesis and Verification

Design of an AI-assisted toolchain—comprising tools named Formalize, Construct, and Translate—to integrate into critical engineering workflows for code specification, synthesis, and verification.

Flexible Hardware-Enabled Governors

Atlas Computing initiative announced under the title “Flexible Hardware-Enabled Governors” on the Atlas Blog.

Delta Fund

Delta Fund is a regranting initiative for AI safety that pools donations and distributes them flexibly based on expert guidance and current needs, operating as a fiscally sponsored project of Ashgro Inc.

Apart Research

Apart Research is a remote-first AI safety research organization that turns technical talent into published AI safety researchers through sprints, studio programs, and fellowships, with donations in some geographies routed via fiscal sponsor Ashgro Inc.

Timaeus

Timaeus is an AI safety research organization using singular learning theory and related mathematical tools to study interpretability and technical AI alignment.

Alignment Ecosystem Development

Alignment Ecosystem Development (AED) is an AI safety field‑building nonprofit that builds and maintains digital infrastructure, such as online resources and tools, to grow and improve the AI safety ecosystem, and has received grants via its fiscal sponsor Ashgro Inc.

AISafety.info

AISafety.info is a volunteer-maintained wiki and Q&A resource with hundreds of answers on AI existential risk, complemented by the Stampy chatbot that searches a large corpus of AI safety material.

ForesightSafety Bench: Comprehensive AI Safety Benchmark

ForesightSafety Bench is a comprehensive AI safety evaluation benchmark and leaderboard created by Beijing-AISI that organizes safety assessment into three hierarchical levels, 22 pillars and 94 fine-grained risk dimensions for frontier AI systems.

PandaGuard: LLM Safety Evaluation Framework

PandaGuard is an open-source multi-agent framework and leaderboard developed by Beijing-AISI to systematically evaluate large language models’ jailbreak safety, together with the PandaBench benchmark that tests dozens of models across many attack and defense strategies.

Watcher

MARA (Modeling, Abstraction and Reasoning Agents)

MARA is a long-term challenge project to build AI systems capable of everyday scientific discovery—proposing hypotheses, running experiments, and constructing abstract world models through active experimentation and reasoning.

R-ADA (Rational Automated Design Agent)

R-ADA is a project to automate robot design by treating morphology and control as a sequential design process, combining language models that operate over CAD/CAM tools, simulation-based evaluation, probabilistic modeling, and Bayesian updating to explore and refine innovative robot designs.

ChiRho

ChiRho is an experimental language and Pyro extension for probabilistic causal reasoning that lets researchers express causal assumptions as probabilistic programs and causal questions as program transformations, supporting interventions, counterfactuals, causal explanations, and structure discovery.

LLM Council

LLM Council is a web platform that lets users access multiple leading AI language models—Claude, ChatGPT, Gemini and Grok—from a single interface and compare their responses side-by-side using a Council mode.

SimpleBench

SimpleBench is a multiple-choice text benchmark introduced in August 2024 by Philip, creator of the AI Explained YouTube channel, in collaboration with Hemang, to evaluate large language models on basic common-sense reasoning tasks where humans still outperform them.

AI Explained Official Podcast

AI Explained Official Podcast is a show hosted by Philip, creator of the AI Explained YouTube channel, that offers hype-free coverage of frontier AI news and the arrival of smarter-than-human AI, drawing on his SimpleBench-driven evaluations of model reasoning.

Canvas

Canvas is a family digital ecosystem from Aligned AI that embeds user-designed AI assistance in a personalised, offline environment, providing a powerful, safe and controlled AI experience that puts the user in the driver’s seat.

Benchmarking LLM agents on consequential real-world tasks

Open Philanthropy–funded research project led by Boston University assistant professor Najoung Kim with co-PI Sebastian Schuster to evaluate the abilities of large language models on challenging tasks.

Scaling Trust

Safeguarded AI

Expert Survey on Progress in AI (ESPAI)

A recurring large-scale survey run by AI Impacts that asks thousands of AI researchers in top venues for forecasts about AI progress, timelines and impacts, with major rounds conducted in 2016, 2022 and 2023.

ML Research Benchmark (MLRB)

An open benchmark that evaluates AI agents on competition-level machine learning research tasks to measure how well they can accelerate ML research.

ScoutML

A tool for instrumenting AI agents that logs their actions into structured sessions so teams can inspect, debug, and analyze agent behavior.

AI Across America

AI Across America is SeedAI’s national initiative that creates high‑trust spaces where community leaders, policymakers, researchers, students, and industry collaborate to build AI literacy, adoption, and infrastructure through local convenings across the United States.

Accelerate Science Now

Accelerate Science Now is a SeedAI-led, nonpartisan coalition of leaders from industry, academia, civil society, and the research community working to harness AI and other game‑changing technologies to accelerate scientific discovery and deliver benefits to the American public.

AI Primers

AI Primers is a briefing series designed to strengthen Congressional understanding of artificial intelligence by convening expert panels that explain how AI affects key policy areas and how governments and industry worldwide are responding to its deployment.

For Your Situational Awareness (FYSA)

For Your Situational Awareness (FYSA) is a monthly, off‑the‑record breakfast series for senior Hill staff and AI policy practitioners, convened by SeedAI at the Consumer Technology Association Innovation House to enable candid discussion and early signal detection on emerging AI policy issues.

SeedAI House (SXSW)

SeedAI House (SXSW) was a multi‑day activation during the 2025 SXSW Festival in Austin that welcomed more than 1,600 guests for panels, fireside chats, keynotes, art exhibits, and performances exploring AI, public policy, and innovation.

American AI Festival

The American AI Festival was a one‑day convening on March 26, 2026, in Washington, D.C. that brought together government leaders, industry executives, scientists, and researchers to discuss AI’s future and U.S. AI readiness.

Hack the Future

Hack the Future was a national event series within SeedAI’s AI Across America program that used AI red‑teaming competitions and hands‑on workshops to drive AI learning, assessment, and engagement deeper into communities traditionally left out of early technology waves.

AI Policy Strategy Fellowship

International Dialogues on AI Safety (IDAIS)

A series of high-level dialogues hosted by the Safe AI Forum that brings together senior computer scientists and AI governance experts from around the world to build international collaboration on AI risks.

AI-Generated Audio for Planned Obsolescence

Podcast feed providing AI-generated audio versions of Planned Obsolescence posts from planned-obsolescence.org, read by models trained on the authors’ voices.

GJO Calibration app

Bottlenecks to more impactful crowd forecasting

Prediction Markets in the Corporate Setting

Samotsvety’s AI risk forecasts

Micro-grants for forecasting

Samotsvety Nuclear Risk Forecasts — March 2022

Systemic AI Safety Grants Programme

Grants programme run by the UK AI Safety Institute offering up to £8.5 million in funding for around 20 initial projects on systemic AI safety, including risks from deepfakes, misinformation, cyberattacks and failures in critical infrastructure.

AISI Challenge Fund

£5 million grant programme led by the UK AI Security Institute that awards grants of up to £200,000 per project to tackle pressing AI security and safety questions, with a focus on protecting critical systems, preventing AI misuse and strengthening public trust.

The Alignment Project

Global alignment research fund launched in 2025 by the UK AI Security Institute and partners, offering grants of up to £1 million per project and having awarded over £27 million to more than 60 alignment research projects in its first round.

AI Governance Alliance (AIGA)

A World Economic Forum multi-stakeholder initiative launched in June 2023 to provide guidance on the responsible design, development and deployment of generative AI systems, organized around three workstreams on safe systems and technologies, responsible applications and transformation, and resilient governance and regulation.

Centre for AI Excellence (CAIE)

A European Centre for AI Excellence in Paris, created by the World Economic Forum in partnership with VivaTech as part of the C4IR global network to strengthen Europe’s AI leadership by driving responsible AI innovation, adoption and cross-border collaboration between business, policymakers and researchers.

AI Safety Studio

Mila’s AI Safety Studio is a dedicated initiative launched in 2025 to develop concrete safeguards for AI systems, starting with mitigating risks from chatbot misuse in youth mental‑health contexts.

Mila AI Policy Fellowship

A six‑month fellowship at Mila that bridges AI research and public policy by bringing together researchers and practitioners to work on real‑world AI governance challenges.

Venture Scientist Bootcamp

Mila Ventures’ flagship 4‑month incubator that provides a $10,000 stipend, coaching, and workspace to help top Canadian STEM founders turn deep‑tech ideas into high‑potential AI startups.

Montréal Sovereign AI Research Hub

A strategic collaboration between Mila, Hypertec, and 5C to build a Sovereign AI Research Hub at Hypertec’s LaSalle campus, providing secure high‑performance compute and infrastructure for Canadian AI researchers and startups, with planned investments of up to $250 million.

Democratic Fine-Tuning (DFT)

Democratic Fine-Tuning (DFT) is an OpenAI-funded project of the Meaning Alignment Institute that uses values cards and a moral graph in a short democratic deliberation process to elicit people’s values and aggregate them into training data for fine-tuning language models, providing an alternative to Constitutional AI and standard RLHF.

Full-Stack Alignment

Full-Stack Alignment is a collaborative research project led by the Meaning Alignment Institute and a group of outside researchers that aims to co-align AI systems and the institutions that shape them with what people value, from individuals’ visions of the good life to shared collective ideals.

Academic Grants Program

MMI’s Academic Grants Program funds scholarly work that concretizes and disseminates metaphysical insights from Eastern spiritual traditions within existing academic frameworks, including research on near‑death experiences and on Buddhist jhāna states and Christian speaking in tongues.

Math and Metaphysics Symposia

Series of Math and Metaphysics Symposia organized by the Mathematical Metaphysics Institute to advance its goal of developing mathematically rigorous foundations for metaphysics and exploring their implications.

Global Resilient Anticipatory Infrastructure Network (GRAIN)

A diplomatic and trade initiative to connect “nodes of persisting complexity” – states with critical roles in food, energy and trade systems – so they can coordinate investments and preferential agreements that strengthen global resilience and recovery capacity in the face of catastrophic risks.

AI Governance Assembly

An application of the Odyssean Process to AI governance that combines expert horizon scanning, exploratory modelling and citizens’ assemblies to develop adaptive, democratically legitimate policy pathways for advanced AI systems.

Horizon Scan on Global Catastrophic Risks

A pilot application of the first phase of the Odyssean Process that used structured expert elicitation to scan for emerging global catastrophic risks and tipping points, producing a horizon scan that informs subsequent modelling and democratic deliberation.

Resilient Supply Chains

A research programme modelling how large-scale nuclear war and the resulting nuclear winter could disrupt global trade and food supply chains, and identifying ways to design more robust networks that support civilisational survival and recovery.

Mozilla Data Collective

Social enterprise and platform, incubated by Mozilla Foundation, for responsible AI data exchange that lets communities control, license, and earn from their datasets while powering trustworthy AI.

Responsible Computing Challenge

Multi-year grant initiative that funds faculty to develop and pilot computing curricula that embed ethics, social context, and public-interest perspectives into undergraduate education around the world.

Mozilla Foundation Incubator

Incubator program that bridges the funding gap between prototype and sustainability for values-driven, public-interest technologies by focusing on product–community fit and cohorts like Democracy x AI.

Mozilla Technology Fund (MTF)

Grantmaking program that supports teams of open-source technologists through themed funding rounds to tackle problems that threaten internet health and advance trustworthy, accountable AI.

Common Voice

Global, community-driven project creating the world’s most diverse open voice dataset so anyone can contribute speech in many languages for use in open-source voice technologies.

Democracy x AI Cohort

Mozilla Foundation Incubator cohort that funds and supports technologists building AI systems which strengthen democracy, improve information integrity, and enhance civic participation worldwide.

AI Safety Purdue

Community of technical and policy researchers at Purdue University aimed at reducing the risks of advanced AI.

PIBBSS Affiliate Program

A 6–12 month affiliate program that provides salary, community and operational support for individual researchers pursuing “PIBBSS-style” AI alignment work grounded in parallels between intelligent behaviour in natural and artificial systems.

PIBBSS × Iliad Research Residency

A six-week paid research residency in applied mathematics and AI alignment, co-run by PIBBSS and Iliad at the London Initiative for Safe AI in early 2026 for PhD-level or equivalent researchers in math, physics, computer science and related fields.

MIT AI Incident Tracker

An interactive tool that uses a large language model to classify more than 1,400 real‑world incidents from the AI Incident Database by risk, cause, harm severity, and related dimensions using the MIT AI Risk Repository’s causal and domain taxonomies.

MIT AI Risk Jobs Database

An AI Risk Jobs Database with interactive dashboards that map AI‑related roles across the AI lifecycle, ecosystem position, country, and sector to help users explore the AI risk job landscape.

MIT AI Risk Mitigation Map

A living AI Risk Mitigation Database and preliminary taxonomy that organize 831 mitigations from 13 frameworks into four main categories and 23 subcategories as part of the MIT AI Risk Initiative.

MIT AI Governance Map

An interactive governance mapping project that uses large language models to classify over 1,000 AI governance and regulatory documents from CSET’s AGORA dataset by risk coverage, sector, scope, legislative status, and related attributes, visualizing how AI risks are currently being governed.

MIT AI Risk Navigator

An interactive web tool that centralizes MIT AI Risk Initiative datasets using shared taxonomies so users can explore AI risks, incidents, and governance documents side‑by‑side and identify cross‑dataset patterns.

Risk Management Ratings

SaferAI’s Risk Management Ratings project maintains a public framework for scoring how well leading AI companies identify, assess, mitigate, and govern AI risks, publishing comparative ratings that inform policymakers, investors, and the wider public.

Global AI Frontier Lab

Joint AI research lab at NYU, housed at 1 MetroTech Center on the Tandon campus, that was launched with the Korean government and is run by Yann LeCun and Kyunghyun Cho to advance fundamental AI research.

NYU Alignment Research Group (ARG)

Research group at New York University conducting empirical work with large language models to address longer-term concerns about deploying highly capable AI systems.

Center for Responsible AI (R/AI)

Interdisciplinary center at NYU Tandon and the Center for Data Science that aims to make responsible AI synonymous with AI.

Task Level Fairness

Project investigating how to evaluate fairness at the task and problem level to develop heuristics about when fair models are feasible before training, including a Python library for use during exploratory data analysis.

Model-Based Fairness Intervention Assessment

Project that uses bias models to evaluate how different fair machine learning interventions perform, with the goal of giving data scientists more actionable guidance when choosing fairness techniques.

Realizing Sociotechnical Machine Learning through Modeling, Explanations, and Reflections

NSF CAREER research program that uses human-in-the-loop methods, model-based approaches, and explanation techniques to help technologists anticipate and reflect on the social impacts of machine learning components in sociotechnical systems.

LLMs and Fair Data Driven Decision-making

Project developing a benchmark to test how LLMs and LLM-based agents make non-discriminatory, fair decisions from data, including direct decision-making, assisting fair ML tasks, and agentic model training.

Wiggum Fairness Forensics

Simpson’s-paradox-inspired fairness forensics project called Wiggum, developed in collaboration with the OU Data Lab to visualize and analyze fairness-related patterns in data.

Perceptions of AI Fairness

Collaborative project with C. Malik Boykin’s lab at Brown University studying people’s preferred definitions of algorithmic fairness and how social and algorithmic factors shape those preferences, including methods to interpolate between fairness definitions.

Dynamics of Fairness

Past project examining how notions of fairness generalize, tolerate distribution shift, and propagate through interacting systems, currently inactive and seeking a student.

Lionheart AI I

Lionheart AI I is a venture capital fund managed by Lionheart Ventures, domiciled in Delaware and investing in healthcare and digital wellness companies such as Breathwrk, Tripp and Momentu.

Swiss AI Safety Prize

The Swiss AI Safety Prize is a competition launched by Pour Demain and the Swiss Cyber Security Days to highlight AI safety concerns by rewarding projects that demonstrate security vulnerabilities in large language models or specialised machine-learning systems, with prize money for winning submissions presented at Swiss Cyber Security Days.

Safe Artificial Intelligence for Basel-Stadt

In 2025 Pour Demain conducted the project “Safe Artificial Intelligence for Basel-Stadt”, supported by the Christoph Merian Foundation, to help critical infrastructure operators in Basel-Stadt deploy AI safely and securely and to feed the resulting insights into standardisation and legislative processes.

A deep understanding of AI large language model mechanisms

A 90+ hour Udemy course by Mike X Cohen that builds and trains transformer-based large language models in PyTorch while teaching LLM architectures, training pipelines, and mechanistic interpretability methods for understanding and dissecting models like GPT and BERT.

50 ML projects to understand LLMs

A book and accompanying code by Mike X Cohen that presents 50 machine learning projects designed to investigate transformer-based large language model mechanisms through data analysis, visualization, and experimentation.

Obsolete: The AI Industry’s Trillion Dollar Race to Replace You—and How to Stop It

A forthcoming non-fiction book by journalist Garrison Lovely that explains how the modern AI industry is investing at unprecedented scale to automate human labor, analyzes the political and economic forces behind this “obsoleting” project, and outlines a democratic program for AI reform.

Guaranteed Safe AI Seminars

Monthly technical seminar series, founded in April 2024, on quantitative and guaranteed AI safety approaches such as formal verification, certified training, and runtime monitoring.

AI Safety in Montréal

Local field-building hub and newsletter for the Montréal AI safety, ethics, and governance community, serving 1,600+ members with meetups, coworking, workshops, and collaborations.

AI Safety Events & Training newsletter

Weekly newsletter that aggregates newly announced AI safety events and training programs worldwide, online and in person.

AI Safety Unconference

Participant-driven unconference series where AI safety researchers propose and run sessions—talks, discussions, and 1:1s—alongside major ML conferences and online to advance AI safety research.

Canadian AI Incident Monitor (CAIM) / Canadian AI Hazard Observatory (CAIHO)

Bilingual public-interest database and observatory that documents AI incidents and hazards affecting Canada, providing structured, citable evidence with API access to support prevention and accountability.

AI Welfare Seminars

Monthly research seminar series on AI consciousness, welfare, and moral status, connecting work across philosophy, neuroscience, AI, and law.

AI Security Deep Dives

Six-week in-person research seminar group in Tel Aviv where selected cybersecurity professionals explore high-impact AI security research areas through expert-led sessions and discussions.

Heron AI Security Research Fellowship

Four-month, part-time research program where experienced cybersecurity professionals collaborate with AI security field leaders on concrete projects to secure transformative AI systems.

The Nest

Members-only workspace in Tel Aviv for founders, researchers, and cyber professionals advancing the AI security field, designed for deep work, meaningful connections, and high-impact projects.

System Level Safety Evaluations

Research agenda developing adversarial, multi-agent evaluations to stress-test societal defensive processes—such as democratic mechanisms, scientific consensus formation, and social dynamics—so as to measure system-level AI safety properties under coordinated attacks.

Collective-Intelligence-Library

An open-source, process-centric multi-agent simulation platform developed by Equilibria Network for exploring coordination mechanisms and collective intelligence dynamics.

HCAST: Human-Calibrated Autonomy Software Tasks

Benchmark of 189 machine-learning, cybersecurity, software engineering, and reasoning tasks with over 1,500 hours of human baselines, used to evaluate autonomous AI agents and to which EquiStamp-affiliated researchers contributed alongside METR.

RobustifAI (Horizon Europe project)

A three-year Horizon Europe consortium (budget €9.3M) launched in June 2025 to develop rigorous, human-centric methods for reliable and trustworthy generative AI in human cyber-physical systems, with Hebrew University’s contribution led by Prof. Guy Katz.

Governance of AI Lab (GOAL)

A cross-disciplinary research lab at the Hebrew University that develops institutional and technical infrastructure for safe and ethical AI, with a focus on governing autonomous agents and legal alignment.

Responsible & Safe AI Systems

An online NPTEL/SWAYAM course offered by IIIT Hyderabad and IIT Madras that provides a comprehensive introduction to ethical, social and safety considerations in AI systems, covering topics such as AI risks, robustness, fairness, privacy, interpretability and AI policy/AGI.

iRASTE – Intelligent Solutions for Road Safety through Technology and Engineering

A multi‑agency AI road‑safety project anchored at IIIT Hyderabad and INAI that uses sensors and predictive analytics based on Advanced Driver Assistance Systems (ADAS) to warn of potential collisions in advance and support large reductions in road accidents in line with national targets.

Iliad Fellowship

A research fellowship in applied mathematics and AI alignment, run by Iliad in collaboration with Principles of Intelligence, aimed at PhD‑level researchers and helping fellows secure follow‑on funding to continue their research or start new groups.

ILIAD Conference Series

A 5‑day conference series on theoretical AI alignment that brings together 100+ researchers to build scientific foundations for AI alignment, with topics including singular learning theory, agent foundations, causal incentives, computational mechanics, safety‑by‑debate, and scalable oversight.

PIBBSS × Iliad Research Residency

A 6‑week full‑time research residency in applied mathematics and AI alignment co‑run by PIBBSS and Iliad, aimed at researchers who may later start new research groups or continue other research projects with philanthropic funding.

Agent Foundations Conference Series

A series of 5‑day conferences on fundamental, mathematical research into agency, organized by Iliad, with the 2026 iteration held at Carnegie Mellon University as the third conference in the series.

Iliad Intensive

A month‑long, full‑time in‑person AI alignment course run by Iliad every second month, targeting participants with strong backgrounds in mathematics, physics, or theoretical computer science.

Machine Learning Research Support – Trojan Detection Competition

Pass-through funding to ETH Zurich researchers to run a Trojan detection competition at the Security and Trustworthy ML conference, aligned with reducing risks from advanced AI.

SAELens

SAELens is an open-source Python library from Decode Research for training sparse autoencoders on language models and analyzing their features to support mechanistic interpretability research.

circuit-tracer

circuit-tracer is an open-source library for finding, visualizing, and intervening on feature circuits in language models using sparse autoencoder and transcoder features, developed by Anthropic fellows and maintained in collaboration with Decode Research.

Secure and Private AI (SPY) Lab, ETH Zurich

Research group in ETH Zurich’s Department of Computer Science that investigates the security, privacy and trustworthiness of machine learning systems, often designing adversarial attacks that probe worst‑case behaviour in order to ultimately improve the safety of these systems.

ETH AI Center

ETH Zurich’s central hub for artificial intelligence, bringing together researchers on AI foundations, applications and implications across all departments to foster research excellence, industry innovation and AI entrepreneurship towards trustworthy, accessible and inclusive AI systems.

Secure, Reliable, and Intelligent Systems (SRI) Lab, ETH Zurich

Research group in ETH Zurich’s Department of Computer Science focused on reliable, secure and trustworthy machine learning, with a particular emphasis on large language models and methods that make AI systems more robust and dependable.

Formal Contracts for Multi-Agent Reinforcement Learning (contracts library)

Open-source library implementing formal contracts to mitigate social dilemmas in multi-agent reinforcement learning, built on RLlib and developed by members of the Algorithmic Alignment Group.

CommonClaim dataset

Dataset of 20,000 boolean statements, each labeled by two human annotators as common-knowledge-true, common-knowledge-false, or neither, created by the MIT Algorithmic Alignment Group for red-teaming language models.

Project to better mitigate sandbagging in AI models

Open Philanthropy–funded Penn State research, led by Rui Zhang, developing methods to detect and mitigate sandbagging behaviors like exploration hacking and password-locking in AI models.

Seasonal Research Fellowships

LawAI’s Seasonal Research Fellowships offer law students, professionals, and academics remote‑first Summer and Winter research positions to work on AI, law, and policy across U.S. Law & Policy, EU Law, and Legal Frontiers tracks.

Summer Institute on Law and AI

An intensive Summer Institute run by LawAI that brings together researchers, practitioners, and policymakers for talks, fireside chats, and Q&As on core questions in AI law and policy, including AI liability, compute regulation, and the role of legal institutions.

Optimization and Trustworthy Machine Learning (OPTML) Group

Research group in MSU’s Computer Science and Engineering department developing optimization‑driven, robust and explainable AI methods to make large‑scale AI systems more trustworthy and scalable.

AI Compute in Germany report

Policy report published in October 2025 by the KIRA Center that analyses Germany’s AI compute infrastructure, compares it to global capacity, and recommends strategies for scaling compute, informing Germany’s National Data Center Strategy.

1st Bundestag AI Boot Camp

An in-depth AI Boot Camp held in the German Bundestag on 6 July for members of parliament and their staff, jointly organised by Cosmonauts & Kings, ProjectTogether and the KIRA Center to cover AI basics, practical political use cases and AI risks.

International AI Safety Report 2026

The 2026 International AI Safety Report, published in February 2026, is the second comprehensive review of general-purpose AI capabilities, risks and safety, for which KIRA Center staff served as Special Adviser to the Chair, core writers and led strategy and engagement for the report team.

International AI Safety Report 2025

The inaugural International AI Safety Report, published in January 2025 and led scientifically by Yoshua Bengio with Daniel Privitera as Lead Writer and Philip Fox as a writer from the KIRA Center, provides a comprehensive review of capabilities, risks and safety techniques for general-purpose AI and is backed by around 30 countries and international organisations.

Pathogen Research Amicus Brief Project

LASST’s project to research litigation involving pandemic‑potential pathogen research (PPPs and PEPPs) and file amicus briefs with scientific partners to educate courts and discourage unreasonably dangerous pathogen experiments.

AI Safety Whistleblower Legal Defense Fund

A LASST program that provides grants to cover legal expenses related to AI safety whistleblowing, prioritizing cases involving the safety or alignment of frontier AI systems, and offering pro bono legal advice and help finding additional support.

Hardware AI Governance Lab

An interdisciplinary initiative hosted by the Oxford Martin AI Governance Initiative that explores how computer hardware architectures can be leveraged to support AI governance, oversight and international coordination.

Technical AI Governance Programme

A first-of-its-kind programme hosted by the Oxford Martin AI Governance Initiative and the Department of Engineering Science that advances the technical foundations of safe, accountable and well-governed AI systems and bridges cutting-edge AI research with effective governance through interdisciplinary research and DPhil training.

CATALINK

CATALINK is an IST‑led blueprint and prototype for a dedicated, last‑resort nuclear crisis communication system—an open, multilateral hotline designed to help leaders of nuclear‑armed states prevent or halt nuclear war.

Andrew Carnegie AI–Nuclear Policy Accelerator

The Andrew Carnegie AI–Nuclear Policy Accelerator is a practitioner‑focused fellowship program that equips mid‑career nuclear and security policy professionals with AI technical literacy and applied policy skills at the AI–nuclear security intersection.

Ransomware Task Force

The Ransomware Task Force is a multistakeholder initiative convened by IST that produced a 2021 framework with 48 recommendations to combat ransomware and continues to drive policy, disruption, and defense efforts worldwide.

AI Chip Export Control Initiative

IST’s AI Chip Export Control Initiative investigates root causes of AI chip export‑control compliance failures and develops a framework for stronger multi‑agency enforcement and international coordination to close critical gaps in the AI chip supply chain.

Security Level 5 Task Force

IST’s Security Level 5 Task Force is a cross‑industry effort developing a technical roadmap and standard so frontier AI infrastructure can achieve nation‑state‑level security across cyber, physical, insider, and supply‑chain threats.

Religious Voices and Responsible AI

Religious Voices and Responsible AI is a joint initiative between IST and AI and Faith that engages religious leaders and communities, equipping them with tools and education to navigate AI’s risks and opportunities and bring diverse ethical perspectives into AI governance debates.

AI and Nuclear Command, Control, and Communications

IST’s AI and Nuclear Command, Control, and Communications project explores how advanced AI capabilities will be integrated into NC3 systems and assesses whether such integration could stabilize or destabilize nuclear‑armed states.

UnDisruptable27

UnDisruptable27 is an IST initiative translating high‑level cyber risk awareness into local readiness by strengthening critical infrastructure resilience—especially water and emergency healthcare—against potential nation‑state cyber disruptions anticipated around 2027.

Soloware Platform

A platform developed by Aayush Kucheria and Kuil Schoneveld for sharing and coordinating views on text, providing recipes for software agents to draw on users’ preferred UI/UX patterns as part of the Live Theory interfaces.

Vibe Decoding

A lens built by Jayson Amati to assist in fine-grained discernment by scaling sensitivity to AI-generated content that is only superficially relevant, helping users detect and avoid misleading AI "slop" within the Live Theory interface ecosystem.

Autostructures

A Groundless-linked project explored at AI Safety Camp that develops Live Theory–inspired research methodologies and "live interfaces" which use AI to move beyond fixed formal structures toward fluid, context-sensitive sensemaking.

Management of Substrate-Sensitive AI Capabilities (MoSSAIC)

A conceptual AI safety project co-authored by Sahil, Aditya Prasad, Aditya Adiga, and Jayson Amati that analyses substrate-sensitive AI risk scenarios and proposes management strategies, presented as part of the broader Live Theory agenda.

Live Conversational Threads

An interface developed by Aditya Adiga that uses AI to identify and manage conversational threads in real time, helping researchers mark and port insights from ongoing discussions into personalised, context-sensitive structures for Live Theory–style sensemaking.

Good Ancestors AI Governance Summer Fellowship

A 2024–25 fellowship run by Good Ancestors in partnership with Arcadia Impact’s AI Governance Taskforce, supporting researchers to produce policy-relevant papers on topics such as an Australian-led regional AI Safety and Security Institute and empowering Pacific Island nations in global AI governance.

Australians for AI Safety

A Good Ancestors-organised public campaign that mobilises experts and citizens across Australia to advocate for stronger AI safety policies, including establishing an Australian AI Safety Institute and AI Act with mandatory guardrails for high‑risk AI.

Preventing Sociopathic Robots

Two-year AI safety project modeling vulnerable, homeostatic agents in an open-ended, Minecraft-like virtual world to test how embodied vulnerability and biologically inspired inductive biases can give rise to prosocial behavior and artificial empathy.

Sentiometer: Validating Novel Optical Detection of Consciousness

Laboratory study using concurrent 64-channel EEG, polysomnography, and a novel optical sensor (the “Sentiometer”) to test whether a purely optical signal can track the presence and quality of conscious experience across sleep, perception, gameplay, and meditation tasks.

Dark Cave, Bright Mind: Mapping Neural and Phenomenological Effects of Dark Retreat Practice

Study on whether extended dark retreat practice can disrupt habitual perception and function as a “sober psychedelic,” enabling profound self-reflection and transformative experiences without pharmacological psychedelics.

Efficacy Study of the Evolve Nervous System Reset Program

Trial of a VR-based biofeedback program to reduce stress and burnout in working professionals while testing whether lightweight, portable EEG hardware can deliver data quality comparable to lab-grade systems.

Scientist AI

Scientist AI is LawZero’s flagship research program to build a highly capable but non-agentic AI system that predicts and explains the world using transparent, probabilistic reasoning while remaining free of its own goals or preferences.

Φ-Arena and adversarial multi-agent VLA interpretability

Independent collective. Φ-Arena open benchmark, 3 ICLR 2027 papers (Φ-Arena, mechinterp, energy-bounded) — kickstart for a 10-year program.

Human-Side Alignment Risks in Non-Self Language Models

Projection, sycophancy, and institutional artifact fabrication in AI-mediated supervision.

A Small AI Governance Evidence Kit from Türkiye

A micro-project to turn an early AI governance prototype into risk notes, oversight checklist, review templates and public learning.

MATS 10.0 Neel Subliminal Learning

Write a paper explaining the mechanisms of subliminal learning as steering vector distillation, funded retroactively for time spent by Agam Bhatia.

Support the AI Policy Institute for AGI preparedness polling and education

AI Policy Institute educates U.S. policymakers and the public on AGI/loss-of-control risks via congressional outreach, media engagement, and public opinion polling to support AI governance.

The AI Policy Podcast

A biweekly podcast from CSIS's Wadhwani AI Center that brings together policymakers, researchers, and industry leaders to discuss AI policy, regulation, national security, and geopolitics. The show was founded and initially hosted by Gregory C. Allen and is now hosted by center director Aalok Mehta.

Awareness-conditioned emergent misalignment

Does harmful fine-tuning data cause broad misalignment only when the model already recognises the target behaviour as a norm violation?

Agent Threat Rules (ATR): open detection rule corpus for AI agent attacks

344 MIT rules merged into Microsoft Agent Governance Toolkit, Cisco AI Defense, MISP, OWASP. Microsoft Copilot SWE Agent uses ATR for CVE triage.

Funding to work on global AI safety and governance infrastructure

Building early AI governance and safety systems focused on alignment, oversight and risk reduction before more capable AI arrives.

Paired-Condition Audit + Time-Constraint Theory for AI Alignment Stability

Proves observed alignment under monitoring ≠ intrinsic policy. Full simulator, 1,000-scenario audit, and general theory of entity freedom (ϕ_x).

Measuring Architectural SelfCorrection in an LLM Agent: 0.336 Baseline on 76,570

Measured post-embodied sensation integration. Solo daily-pace brain-function development in Osaka. Phase 1 funds higher cognitive integration program.

Travel Grants for New Zealand's Inaugural AI Safety Conference

$5,000 USD to bring 16 vetted academics and policy leads to NZ's first AI Safety Conference, ensuring national representation and cross-sector collaboration

ASDR: Adversarial Semantic Drift Replayer for Multi-Agent AI Safety

Open-source replay tool and benchmark prototype for identifying step-level semantic risk transitions in multi-agent AI traces — CI-backed and independently repr

Behavioral Audits for Epistemic Reliability

A public evidence package for testing whether LLMs only sound epistemically reliable

Golem Physics + Constraint Native: Verification Before Voice & Action

Funding the open-source launch of a working claim-state system and the local firewall bridge that carries verification before voice into governed agent action.

Decoding Logic: Mechanistic Interpretability in LLMs in a research program

Identifying and auditing reasoning circuits in LLMs within Algoverse 2026 using Sparse Autoencoders (SAEs).

Hardening a fail-closed runtime for agentic AI systems

This round of funding will be used primarily for prototype hardening, artifact packaging, runtime evaluation, and preparation for external review.

Project Xaeryn: Identity-like States and Relational Dynamics in LLMs

A 15,000+ page corpus on long-term interaction, symbolic language, unusual model behavior, and safety edge cases.

Videos covering AI news and politics with a focus on the extinction threat

Social media content across YouTube, Instagram, and TikTok to grow AI x-risk awareness and build political momentum for a global pause.

Behavioral Fingerprinting for LLM Deception — Validating a Live System

A model-agnostic benchmark for detecting deceptive reasoning in LLMs through behavioral fingerprints — no weight access required.

Bridging Cultural Gaps in AI Safety: Global Validation of the CAAF Tool

Moving beyond Western-centric benchmarks to audit behavioral manipulation globally.

4-month proof of concept for curiosity-driven agents on ARC-AGI-3

Studying more human-like intelligence through constraint-aware, curiosity-driven agents on ARC-AGI-3

Expanding Francophone Awareness of AI Safety Through The Flares

This is a small grant buying a large increase in high-quality Francophone AI risk communication from a creator who has already a track record.

AI Risk Literacy for Clinicians: Building a Trusted Voice for AI Governance

A trusted profession that has advocated against existential risks like nuclear war can do so again for AI — but clinicians must first be made aware of the risks

Modeling AI competition dynamics for better governance

Software tools and research to quantify coordination failures and inform policy decisions.

Deterministic Oversight for Agent Traces — LTP + CML

Build and release an open deterministic oversight layer (LTP+CML) for agent trace recording/replay and structural failure checks, plus LTP-Bench v0.1 and a comparative eval vs behavioral-only methods.

Precautionary AGI Governance

Limited Legal Personhood as a Reversible Safety Instrument

MTCP: Post Correction Persistence by Benchmark for Frontier LLMs

181,448 evaluations proving no production AI model reliably maintains corrections. Expanding coverage and pursuing multi pass validation.

The Worcester Node: Independent Agentic Oversight via Logic Inversion

Establishing a sovereign, decentralized Sentry node to audit frontier AI agents for logic escapes ($E_{escape}$) using the Inverted Social Drift ($SD$) metric.

Seer

Lyptus Research: Bridge Funding

An early-stage AI safety research group based in Sydney, Australia

ML Safety Newsletter

A free newsletter publishing curated summaries of recent machine learning safety research, run by Dan Hendrycks and contributors associated with the Center for AI Safety.

Effective Altruism Domains

EA Domains (ea.domains) is a project that acquires and holds internet domain names relevant to effective altruism, AI safety, and existential risk, then offers them free to legitimate EA-aligned projects to prevent domain squatting.

AI Safety Funding

A newsletter listing newly announced funding opportunities for individuals and organizations working to reduce existential risk from AI.

Technical Alignment Research Accelerator (TARA)

TARA is a free 14-week part-time technical AI safety training program for Python programmers in the Asia-Pacific region, enabling participants to develop AI safety research skills without relocating or leaving their jobs.

AISafety.info

A comprehensive, community-written interactive FAQ about AI existential safety, founded by Rob Miles and hosted at aisafety.info.

AI Safety Quest

AI Safety Quest is a fully volunteer-based organization that helps people navigate the AI safety ecosystem through personalized advising calls, cohort learning, and mentorship matching.

AI Risk: Why Care?

An interactive public education tool that explains AI existential risk to general audiences using a personalized AI chatbot, operated by the AI Governance and Safety Institute (AIGSI) and AI Safety and Governance Fund (AISGF).

AISafety.com

AISafety.com is a curated resource hub for the AI safety ecosystem, providing newcomers and practitioners with organized directories of courses, communities, events, jobs, funders, and more. It is the flagship platform of Alignment Ecosystem Development (AED), led by Søren Elverlin.

AI Safety Map Anki Deck

An Anki flashcard deck of 167 cards covering the main organizations, projects, and programs in the AI safety ecosystem, designed for learning via spaced repetition.

AI Safety Events & Training

Weekly newsletter listing newly announced AI safety events and training programs, both online and in-person.

Safety Research

Google DeepMind's AI safety research program, working to ensure that advanced AI and AGI systems are safe, aligned with human values, and do not pose catastrophic risks.

Existential Risk Laboratory

An interdisciplinary research lab at the University of Chicago dedicated to analyzing and mitigating existential risks, with a focus on AI safety, nuclear security, biosecurity, and extreme climate change.

Online Team

The Online Team at CEA builds and maintains the EA Forum, effectivealtruism.org, and the EA Newsletter, providing the primary digital infrastructure for discussion, research, and coordination within the effective altruism community.

Community Health and Special Projects

A team within the Centre for Effective Altruism that works to strengthen the EA community's ability to have impact by addressing interpersonal harm, advising on risk management, and supporting healthy organizational cultures.

Research Scholars Programme

A two-year research fellowship at Oxford's Future of Humanity Institute that gave early-career researchers salaried positions to explore questions critical to humanity's long-term future, including AI safety, existential risk, and macrostrategy.

Global and Emerging Risks

RAND Global and Emerging Risks is a research division of RAND Corporation delivering rigorous, objective public policy research on catastrophic and existential risks to civilization, including AI, synthetic biology, climate, and nuclear threats.

Talos Fellowship

Talos Network runs the Talos Fellowship, a program that trains and places early-career professionals into European AI policy roles to help ensure safe and responsible AI deployment.

AIES 2022 organizers (supported by AAAI and ACM)

The organizing committee of the Fifth AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society (AIES 2022), a peer-reviewed academic conference held August 1-3, 2022 at Keble College, Oxford, UK.

AAAI/ACM Conference on AI, Ethics, and Society (AIES 2020)

AIES 2020 was the third annual AAAI/ACM Conference on AI, Ethics, and Society, a peer-reviewed academic conference addressing the ethical and societal implications of artificial intelligence, held February 7-8, 2020 in New York City.

AISafety.com: Self-study

The self-study section of AISafety.com curates courses, textbooks, and reading lists for independent learning in AI safety, covering both technical alignment and AI governance.

AISafety.com: Media Channels

A curated directory of AI safety podcasts, newsletters, YouTube channels, blogs, books, and forums maintained by AISafety.com. It helps newcomers and practitioners stay informed about rapid developments in the AI safety field.

AISafety.com: Funding

A curated directory of 60+ funding sources for AI safety work, maintained by AISafety.com as part of its broader resource hub for the AI safety ecosystem.

AISafety.com: Donation Guide

A regularly-updated guide on how to donate most effectively to the AI safety field, structured by donation amount and time available.

AISafety.com: Advisors

A directory within AISafety.com that consolidates free guidance calls from AI safety advisors, helping newcomers identify how to contribute most effectively to the field.

Neuronpedia

Neuronpedia is an open-source interpretability platform for exploring, analyzing, and steering the internal features of AI language models. It serves as the primary public infrastructure for mechanistic interpretability research, particularly around sparse autoencoders (SAEs).

Effective Altruism Sweden Rationality Research Project

A rationality research project operating under Effective Altruism Sweden (Effektiv Altruism Sverige), funded through the Survival and Flourishing Fund's Lightspeed Grants program.

CHAI Internship

A research internship program within CHAI at UC Berkeley where participants work on AI safety research projects under the mentorship of PhD students and postdocs, typically resulting in first-author workshop papers.

Language Model Safety Fund

A fiscally sponsored AI safety project led by Ethan Perez that funded research engineers to work on language model misalignment, which later evolved into part of FAR.AI (Frontier Alignment Research).

FAR.AI YouTube channel

The FAR.AI YouTube channel (@FARAIResearch) publishes recordings of AI safety talks, seminars, and workshop sessions organized by FAR.AI, the AI safety research nonprofit based in Berkeley.

Technology Strategy Roleplay

A UK charity that develops and deploys Intelligence Rising, a strategic role-playing game designed to help decision-makers understand the risks, tensions, and governance challenges of AI development through facilitated wargaming exercises.

Buddhism & AI Initiative

A collaborative initiative bringing together Buddhist communities, technologists, and contemplative researchers worldwide to help shape the future of artificial intelligence through Buddhist wisdom and practice.

Orion AI Governance Initiative

A UK-based talent development initiative supporting outstanding students to build careers shaping the future of AI governance, run as a project of Arcadia Impact.

Safe AI London (SAIL)

Safe AI London (SAIL) is a London-based community program that supports individuals interested in pursuing careers in AI safety, operated as a project of Arcadia Impact.

Impact Research Groups (IRG)

An 8-week London-based programme run by Arcadia Impact that supports talented students in launching high-impact research careers across AI safety, AI governance, biosecurity, global health, and animal welfare.

The Events team at the Centre for Effective Altruism

The events program within CEA that organizes EA Global conferences, supports EAGx events run by local organizers, runs specialized retreats, and coordinates EA Summits to connect people working on the world's most pressing problems.

Effective Altruism Forum

The EA Forum (forum.effectivealtruism.org) is the primary online discussion and publishing platform for the effective altruism community, run by the Centre for Effective Altruism.

Transformer

Transformer is a journalism publication covering the power and politics of transformative AI, aimed at policymakers and decision-makers. It operates as a project of the Tarbell Center for AI Journalism.

Tarbell Fellowship

A one-year journalism fellowship that trains and places reporters at major newsrooms to cover artificial intelligence, providing stipends, a 10-week AI fundamentals course, and nine-month embedded placements at outlets like Bloomberg, TIME, The Guardian, and NBC News.

AI Safety, Ethics and Society (AISES)

AI Safety, Ethics and Society (AISES) is an open educational project of the Center for AI Safety that provides a free textbook and virtual course introducing AI safety, ethics, and societal risks to a broad, non-technical audience.

AI Safety Newsletter

A free newsletter by the Center for AI Safety covering the latest developments in AI safety research, policy, and industry news. No technical background required.

AI Frontiers

AI Frontiers is a publication run by the Center for AI Safety that hosts expert commentary and debate on the societal impacts of artificial intelligence, covering topics from AI safety to policy, economics, and national security.

Cambridge ERA:AI Fellowship

An 8-10 week fully funded research fellowship in Cambridge, UK, supporting early-career researchers and entrepreneurs working to mitigate catastrophic risks from frontier AI. Operated by the Existential Risk Alliance (ERA), a spin-off from the Cambridge Existential Risks Initiative (CERI).

Narrow Path

A Narrow Path is a comprehensive policy framework published by ControlAI in October 2024, laying out a three-phase plan for humanity to safely navigate the development of artificial superintelligence.

Foresight Fellowship

The Foresight Fellowship is a year-long program by the Foresight Institute that supports early-career scientists, engineers, and innovators working on transformative technologies including AI safety, longevity, nanotechnology, and neurotechnology.

Mentorship for Alignment Researchers (MARS)

Part-time research program run by Cambridge AI Safety Hub (CAISH), connecting aspiring researchers with experienced mentors to conduct AI safety research over 2-3 months.

UChicago Existential Risk Laboratory (XLab) Fellowship

The Existential Risk Laboratory (XLab) at the University of Chicago trains early-career researchers through a 10-week summer fellowship focused on AI safety, nuclear security, and global catastrophic risk. It serves as an interdisciplinary hub for existential risk research across the university.

Cooperative AI Summer School

An annual intensive summer school run by the Cooperative AI Foundation that trains students and early-career researchers in the emerging field of cooperative AI.

GovAI Fellowship

A three-month fellowship program run by the Centre for the Governance of AI (GovAI) to help early-career researchers and practitioners transition into full-time work on AI governance.

Impact Academy: Global AI Safety Fellowship

Impact Academy runs fully-funded research fellowships that place exceptional global STEM talent with leading AI safety organizations. Its flagship program, the Global AI Safety Fellowship, supports 10-20 fellows per cohort for 3-6 months at labs such as CHAI, FAR.AI, and the UK AI Safety Institute.

Pathfinder Fellowship

The Pathfinder Fellowship is a selective fellowship run by Kairos that provides funding, mentorship, and resources to students organizing AI safety and policy university groups worldwide.

IAPS AI Policy Fellowship

A fully-funded, three-month fellowship program run by the Institute for AI Policy and Strategy (IAPS) that trains professionals from diverse backgrounds to work on AI policy projects with the goal of securing a positive future with powerful AI.

PIBBSS Fellowship

A 3-month summer research fellowship that pairs PhD and postdoctoral researchers from complex systems and related sciences with AI safety mentors to produce interdisciplinary alignment research.

ML & AI Safety Updates

A weekly newsletter, podcast, and YouTube series by Apart Research covering the latest research in machine learning and AI safety. The series was later discontinued as Apart Research reprioritized its activities.

AI Safety Ideas

AI Safety Ideas is a collaborative, open-access platform for sharing and discovering AI safety research project ideas, built and maintained by Apart Research.

Apart Sprints

Apart Sprints is the research hackathon program of Apart Research, running weekend-long AI safety research events with thousands of global participants. Teams work on frontier AI safety questions and produce open-source research, evaluations, and prototypes.

FLI Fellowships

FLI's fellowship programs fund PhD students and postdoctoral researchers in technical AI existential safety and US-China AI governance, building the next generation of safety researchers.

FLI Podcast

The FLI Podcast is the Future of Life Institute's long-running audio program featuring in-depth conversations with researchers, philosophers, and policy experts on AI safety, existential risk, and the governance of powerful technologies.

BERI-SRL Collaboration

A collaboration between the Berkeley Existential Risk Initiative (BERI) and the Safe Robotics Laboratory (SRL) at Princeton University, through which BERI provides operational support to SRL's research on rigorous safety frameworks for autonomous systems and AI.

BERI-SERI Collaboration

An operational partnership between the Berkeley Existential Risk Initiative (BERI) and the Stanford Existential Risks Initiative (SERI), through which BERI provides administrative and financial services to support SERI's existential risk research and fellowship programs.

BERI-FHI Collaboration

A formal collaboration through which the Berkeley Existential Risk Initiative (BERI) provided operational and research support to the Future of Humanity Institute (FHI) at the University of Oxford, active from approximately 2017 until FHI's closure in April 2024.

BERI-DMIP Collaboration

A collaboration between the Berkeley Existential Risk Initiative (BERI) and the Data, Machines, Intelligence & People (DMIP) research group at the Universitat Politecnica de Valencia, focused on AI evaluation, intelligence measurement, and AI safety research.

BERI-CSER Collaboration

A long-running partnership between the Berkeley Existential Risk Initiative (BERI) and the Centre for the Study of Existential Risk (CSER) at the University of Cambridge, through which BERI provides operational, administrative, and financial support to CSER's existential risk research.

BERI-CLTC Collaboration

A collaboration between the Berkeley Existential Risk Initiative (BERI) and UC Berkeley's Center for Long-Term Cybersecurity (CLTC) focused on developing AI risk management standards and guidance for developers of increasingly general-purpose AI systems.

BERI-CHAI Collaboration

An operational collaboration in which the Berkeley Existential Risk Initiative (BERI) provides administrative, technical, and staffing support to the Center for Human-Compatible AI (CHAI) at UC Berkeley, enabling CHAI researchers to focus on AI safety work.

BERI-ALL Collaboration

A collaboration between the Berkeley Existential Risk Initiative (BERI) and the Autonomous Learning Laboratory (ALL) at the University of Massachusetts Amherst, focused on safe and fair machine learning research.

CLR Fund

The CLR Fund is the grantmaking program of the Center on Long-Term Risk, supporting researchers and projects working to reduce worst-case suffering risks from advanced AI systems.

CLR Fellowship

The CLR Summer Research Fellowship is an annual 8-week paid program that places researchers on s-risk and empirical AI safety projects, serving as CLR's primary talent pipeline into the field.