Teaching Claude Why Replication + Extension

Chicago IL

Replicating, stress-testing, and extending the experiments from Anthropic's blog post "Teaching Claude Why."

Project Details

We are verifying and extending the main experiments in Anthropic’s Teaching Claude Why (TCW) with open-source midsize models (~32B) as part of the Second Look research fellowship. In TCW, Anthropic introduces new mid-training alignment techniques with synthetic document fine-tuning (SDF) and supervised fine-tuning (SFT) to align and attach the model to the "Claude" character that behaves in accordance with Claude’s constitution. They found that this technique successfully mitigates agentic misalignment propensities, generalizes well out-of-distribution (OOD), and the effect persists through reinforcement learning (RL) post-training.

At a high level, our replication tests the following claims:

Does the SDF corpus's effectiveness hinge on an AI assistant persona or simply on generalizable principles illustrated in the data? How much does the mitigation rely on the Personal Selection Model?
How important is the reasoning aspect in the SDF for instilling moral decision-making?
Do improvements from SDF persist through long-run alignment RL and/or capabilities RL?

Second Look is committed to funding replications of core experiments in the post. However, this funding only covers a minimum viable product with single training seeds and limited evaluations. Additional funding would allow us to

Run each main experiment multiple times with extra evaluations to provide uncertainty measurements
Test SDF + SFT result persistency in much longer RL training steps (i.e., the depth of the persona attachment)
Derive a scaling law for the SDF corpus size to understand how much safety data is needed to efficiently mitigate misalignment

Details about each experiment and additional alternative hypothesis testing can be found in our full implementation details document.

The concrete output will be an open-source replication of TCW with additional experiments and ablation studies. We will publish a series of detailed write-ups of the research design and methods, all associated codes, model training checkpoints, and high-quality datasets. Inventing better alignment techniques is out of scope for this project, but we believe that our work could inspire them.

Second Look fellows will conduct the project with oversight from the core team:

Anastasia Wei (leading the project, will be first author)
Stewy Slocum (mentor, previously xAI)
Arav Dhoot, Finn Cairns, Jack Thompson, Brandon Qi (co-authors)
Yixiong Hao, Zephaniah Roe (research management)
Harshul Basava (operations/logistics)

Theory of Impact

Anthropic’s TCW blog describes some of the only alignment techniques shown to be practical and effective in production frontier models. It’s plausible that Anthropic's current alignment plan relies on implementing mid-training with constitutional documents (roughly as detailed in TCW), and then handing off alignment research to (hopefully) aligned human-level AIs. Since mid-training alignment is a central part of their strategy for aligning superintelligence, these techniques must be verified to hold up and scale. Otherwise, recursive self-improvement could easily amplify minor hidden alignment failures into catastrophic outcomes.

Additionally, whether these techniques are robust, generalize OOD, and persist through RL directly impacts whether other frontier labs should adopt similar practices. Notably, in OpenAI’s replication of self-fulfilling misalignment, they found that mid-training (mis)alignment largely washes out over longer post-training RL runs. While Anthropic partially demonstrated alignment persistence through harmlessness RL, they did not test it against capability RL, and it’s unclear whether they ran a sufficient number of RL optimization steps. Therefore, we believe that verifying the long-term persistence of this alignment technique is especially load-bearing.

People

Anastasia Wei

Project Leader

Zephaniah Roe

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$38,000

Ideal

$70,000

What this ask is

We are replicating, stress-testing, and extending main experiments in Teaching Claude Why, in which Anthropic introduces new alignment techniques used in current production Claude models. In the post, Anthropic uses synthetic document fine-tuning (SDF) and supervised fine-tuning (SFT) to align and attach the model to the "Claude" character, and finds that this technique successfully mitigates agentic misalignment propensities, with the effect persisting through reinforcement learning (RL) post-training.

Additional funding would allow for uncertainty measurements on our key findings, experiments testing persistence in long RL runs, and scaling laws for SDF corpus size. Our current funding is only sufficient for delivering a set of minimal viable products, and we think persistence testing is very important for frontier labs that are currently doing long RL runs.

Anthropic’s key claims are:

Training on documents about Claude's constitution and fictional stories about AI behaving admirably improves alignment despite being out-of-distribution (OOD) of the agentic misalignment evals.
Training on demonstrations of desired behavior is often insufficient; explaining why some actions are better than others matters, as does training on richer descriptions of Claude's overall character.
Data quality and diversity are crucial.

At a high level, we would like to test the following claims on an open-source model:

Is alignment tied to the AI assistant persona specifically? Or, is it more dependent on the quality and generalizable principles illustrated in the content of the data? That is, does the data need to be tied to the AI persona, and how much does the Personal Selection Model apply?
How important is the reasoning for instilling moral decision-making?
Do improvements from SDF persist through alignment RL and/or capabilities RL?

Our theory of change for this research is to provide an open-source replication to verify the paper’s claims, and for the broader alignment research community to inspect and build on. Inventing better alignment techniques is out of scope for this project, but we believe that our work could inspire them.

This project is being conducted as part of the Second Look Research fellowship at UChicago XLab, focused on replicating load-bearing results in AI safety. Research is primarily conducted by the fellows with oversight from the core team.

Who is involved:

Anastasia Wei (leading the project, will be first author)
Stewy Slocum (mentor, xAI)
Arav Dhoot, Finn Cairns, Jack Thompson, Brandon Qi (co-authors)
Yixiong Hao, Zephaniah Roe (research management)
Harshul Basava (operations/logistics)

Theory of impact

Anthropic’s Alignment Science’s Teaching Claude Why blog describes the alignment-training techniques behind its current production models and proposes principled interventions for alignment mid-training. These are among the few alignment techniques shown to be practical and effective in production so far; whether they are robust, generalize OOD, and persist through RL has direct impacts on whether other frontier labs should adopt similar practices.

Anthropic's current alignment plan is implementing mid-training with constitutional documents (roughly "Teaching Claude Why"), and it’s plausible that they will then hand off alignment research to (hopefully) aligned human-level AIs. Since mid-training alignment is central to Anthropic’s strategy for aligning superintelligence, we must verify that these methods are robust. Otherwise, recursive self-improvement could easily amplify minor alignment failures into catastrophic outcomes.

Regardless of the outcome, our work will produce useful signals. If the mid-training step works well in an open-source model, it could encourage other frontier labs to adopt similar techniques. If it works but for unexpected reasons, it would help us and other labs to design more robust techniques. And if it fails completely, it could point to execution errors on our end or suggest these techniques don’t easily transfer to different model families. In this scenario, we will focus on disentangling why the technique fell short.

We plan to open source all of our code and models and their training checkpoints, enabling the broader AI safety community to pursue further research in:

Extension of mid-training alignment techniques
Data-centric interpretability research
Interp work on what mid-training instills in models
Associated claims about the personal selection model

Because some experiments are quite expensive, it is unlikely that other groups prioritize running multiple seeds and have statistical tests. We also have not seen concrete demonstrations from frontier labs that these methods scale in long RL runs, and we aim to verify the robustness of these mid-training techniques with the additional funding.

How the money will be spent

For 38-60k, we expect to accomplish the highest priority items in our proposal if no complications arise. 70k would provide a buffer for resolving unforeseen issues around data generation and leave no financial bottlenecks for the ideal version of our project. See budget breakdown linked here.

The “current budget” column denotes the funds we have already allocated to the project in USD from Second Look. The next two columns indicate our funding asks to improve the quality and impact of our work.

Discussion

No comments yet. Be the first to share your thoughts.