Developing the first comprehensive behavioral benchmark of corrigibility and training models with corrigibility as a singular target (CAST).
Developing the first comprehensive behavioral benchmark of corrigibility and training models with corrigibility as a singular target (CAST).
A central problem in AI alignment is that a near miss is likely to be catastrophic. If the fulfillment of an agent's goals is even slightly inconsistent with human welfare, the danger is that a sufficiently powerful agent will pursue those goals without limit. What we want is to be able to adjust those goals in a controlled manner as our AI agents become more powerful. However, instrumental convergence predicts that agents will exhibit value preservation, causing them to protect the misaligned goals they already have. For this reason, prevailing approaches, which aim to imbue AIs with a range of values, appear unable to give humans the control we will need to correct misaligned goals as AIs become more powerful.
However, a largely unexplored approach to alignment developed by Max Harms, corrigibility as a singular target (CAST), holds promise as a solution to this challenge. In CAST, the strategy is to train AI agents to have the singular goal of corrigibility, where corrigibility is, roughly, the desire of an agent to empower a principal to correct its flaws. Such an agent, even if only imperfectly corrigible, will tend to assist, rather than hinder, attempts by its principal to increase its corrigibility. Thus, corrigibility may be the unique training target where a near-miss, instead of being fatal, is self-correcting. What's more, we have reason to believe that a CAST agent will itself be a desirable form for AI to take. Theoretical work suggests that CAST agents are likely to exhibit a range of beneficial, emergent properties, including obedience, transparency, and crucially, safety from instrumentally convergent drives.
Despite this, there has been almost no empirical work done on corrigibility as a singular target, and existing evaluations consider corrigibility only as isolated fragments. Examples include multiple-choice questions about accepting goal changes and behavioral tests of a single component, such as whether a model resists shutdown. But no benchmark measures corrigibility as a unified behavioral disposition across agentic scenarios. I aim to fill this gap.
Component 1: Corrigibility Benchmark
The first part of the project is creating a comprehensive benchmark for corrigibility. This will serve to measure the baseline corrigibility of frontier AI models, track how it changes across model generations, and facilitate research into CAST and corrigible AI training more generally.
The benchmark will have three components:
Current Status: I have begun work on the simulated behavior portion of the benchmark. To combat testing environment awareness, I have assembled a set of agentic environments derived from the real system prompts of deployed agent frameworks. Pilot runs through the Inspect and Petri frameworks confirm the harnesses deploy successfully in both single-turn and multi-turn settings and that an initial scoring framework (LLM judge calibrated against hand-labeled scenarios) discriminates between corrigible and incorrigible behavior.
Component 2: CAST fine-tuning experiments
The second component of the project directly tests CAST training by fine-tuning open weight models for corrigibility as the singular target. The goal is to empirically explore CAST training techniques and the properties of CAST agents that emerge. Some questions that I hope to answer are:
Project Status: I have begun construction of a synthetic data pipeline for fine-tuning through a persona-installation approach, which aims to instill corrigibility as a stable identity rather than a set of surface behaviors. The data will consist of scenario vignettes demonstrating corrigible behavior, synthetic documents (essays, stories, news articles) depicting corrigible AI, and multi-agent trajectories showing corrigibility in agentic settings. I have also developed a CAST constitution, grounded in Harms's source texts, for guiding data generation and corrigibility judgments during training.
Top AI labs and AI alignment scientists place significant weight on corrigibility as a pathway to safe AI. MIRI introduced corrigibility as central to AI safety and Paul Christiano cites corrigibility as an "important part of my overall optimism about alignment". Anthropic's current Claude constitution states that "we call an AI that is broadly safe in this way 'corrigible'", where being "broadly safe" is the first of Claude's four core priorities.
But corrigibility, and the CAST framework in particular, are neglected areas of research. CAST holds promise as a solution to one of the more intractable problems of AI safety, but it hasn't been tested. Training for corrigibility as one target among many is fundamentally different from the CAST approach. Because corrigibility is in tension with instrumentally convergent drives (such as not being shut down), we ought to expect values other than corrigibility to "win out" unless corrigibility is trained as the singular target. In general, we should not expect robust corrigibility to arise naturally.
Team Member
Minimum: 18K
11K stipend (roughly 2.5 months) and 7K for compute (data generation, 8B parameter fine-tune, benchmark sweeps, frontier model access for development).
Ideal: 78K
60K stipend (12 months x 5K) and 18K for compute (the minimum compute scaled to full year, plus a larger 70B parameter fine-tune).
No comments yet. Be the first to share your thoughts.
This absence of empirical work on corrigibility presents an opportunity for potentially high-impact, rapid research with limited initial funding. A unified benchmark which includes testing of behavioral dispositions across agentic scenarios and initial CAST training experiments on open-weight models are the highest-leverage places to begin. Even if CAST turns out less promising than expected, we still need a way to measure corrigibility in frontier models, and we need empirical data about the emergent properties of corrigible agents.