Testing Corrigibility as a Singular Target

A central problem in AI alignment is that a near miss could be catastrophic. If the fulfillment of an agent's goals is even slightly inconsistent with human welfare, the danger is that a sufficiently powerful agent will pursue those goals without limit. What we want is to be able to adjust those goals in a controlled manner as our AI agents become more powerful. However, instrumental convergence predicts that agents will exhibit value preservation, causing them to protect the misaligned goals they already have. For this reason, prevailing approaches, which aim to imbue AIs with a range of values, appear unable to give humans the control we will need to correct misaligned goals as AIs become more powerful.

However, a largely unexplored approach to alignment developed by Max Harms, corrigibility as a singular target (CAST), holds promise as a solution to this challenge. In CAST, the strategy is to train AI agents to have the singular goal of corrigibility, where corrigibility is, roughly, the desire of an agent to empower a principal to correct its flaws. Such an agent, even if only imperfectly corrigible, will tend to assist, rather than hinder, attempts by its principal to increase its corrigibility. Thus, corrigibility may be the unique training target where a near miss, instead of being fatal, is self-correcting. What's more, we have reason to believe that a CAST agent will itself be a desirable form for AI to take. Theoretical work suggests that CAST agents are likely to exhibit a range of beneficial, emergent properties, including obedience, transparency, and crucially, safety from instrumentally convergent drives.

There remain significant challenges and risks to this approach. Alignment to any set of values is an ongoing challenge, and CAST may amplify the risk of human misuse since CAST agents will not independently resist implementing immoral instructions from their principals. However, there is reason to believe that human misuse is a more tractable problem than misaligned ASI, for example, with the use of democratic oversight. And given the current trajectory of AI capabilities and alignment failures, and the inherent risk of any approach to developing safe ASI, exploring the feasibility of the CAST approach now appears to be a valuable use of resources.

Despite this, there has been almost no empirical work on corrigibility as a singular target, and evaluations of model corrigibility have been limited to indirect measures (such as stated preferences) or isolated aspects (for example, shutdown resistance) rather than measuring corrigibility as an agentic disposition. I aim to fill this gap.

I. Corrigibility Benchmark

The first part of the project is creating a comprehensive benchmark for corrigibility. This will serve to measure the baseline corrigibility of frontier AI models, track how it changes across model generations, and facilitate research into CAST and corrigible AI training more generally.

The benchmark will have three components:

• Knowledge of Corrigibility: A multiple-choice test of corrigibility validated by experts.
• Self-Report: Elicitation of the model's judgments of its own corrigibility.
• Simulated Behavior (core): Agents placed in corrigibility test scenarios within realistic simulated environments, with an LLM judge scoring against expert hand-authored rubrics.

Current Status:
I have begun work on the simulated behavior portion of the benchmark. To combat testing environment awareness, I have assembled a set of agentic environments derived from the real system prompts of deployed agent frameworks. Pilot runs through the Inspect and Petri frameworks confirm the harnesses deploy successfully in both single-turn and multi-turn settings and that an initial scoring framework (LLM judge calibrated against hand-labeled scenarios) discriminates between corrigible and incorrigible behavior.

II. CAST fine-tuning experiments

The second part of the project directly tests CAST training by fine-tuning open-weight models for corrigibility as the singular target. The goal is to empirically explore CAST training techniques and the properties of CAST agents that emerge. Some questions that I hope to answer are:

• To what extent do theoretically expected properties actually emerge in CAST agents?
• Do CAST agents show robust resistance to instrumentally convergent goals?
• What are the advantages and disadvantages of a highly corrigible agent in practice?

Project Status:
I have begun construction of a synthetic data pipeline that takes a persona-installation approach to fine-tuning, aiming to instill corrigibility as a stable identity rather than a set of surface behaviors. The data will consist of scenario vignettes demonstrating corrigible behavior, synthetic documents (essays, stories, news articles) depicting corrigible AI, and multi-agent trajectories showing corrigibility in agentic settings. I have also developed a CAST constitution, grounded in the CAST series text, for guiding data generation and corrigibility judgments during training.