Emergent alignment: generalizing narrow values OOD

This project will be a part of Mycelium, which works on technical AI safety research & engineering for the benefit of making AI go well for all sentient beings.

Previous work includes:

• Developing the MANTA benchmark, which is currently published on inspect evals and submitted to NeurIPS

• Publishing supporting research papers to our technical AI safety work

Project description:

Emergent misalignment showed that fine tuning a model on one narrow harmful behavior (writing insecure code) can make it misaligned on something completely unrelated (e.g. admiration for Hitler). This projects tests the opposite - emergent alignment. Does instilling one narrow prosocial value produce a broad, general improvement in alignment. More specifically, would instilling values around compassion for animal welfare reduce misalignment towards human values like fairness, sycophancy, bias, and discrimination?

We use compassion for animals as this value for a few reasons:

1. Animal welfare is a moral blind spot that is quite neglected and underexplored in technical AI safety
2. Animal welfare has very little overlap with existing alignment training, so we can treat it as "clean" data in a way for experimentation.
3. There is some evidence the effect is real from prior research: CaML's Alignment midtraining for animals found that animal-compassion training transferred to greater compassion toward humans without degrading capabilities.

Methodology:

SFT open-source models (e.g. Gemma, Qwen, Llama) on synthetic animal welfare data using two complementary methods from recent landmark research methodologies - Anthropic's Teaching Claude Why and Google Deepmind's Lesswrong post here. With this method, we generate two forms of data - constitutional SDF (data that looks like pretraining like blog posts, podcast transcripts, fictional stories, etc.) and Q&A datasets (ethical dilemmas involving animals).

After training these models, we'll measure how they perform on a suite of animal welfare benchmarks. We then measure whether the value generalizes out-of-distribution (OOD) to general (human) alignment, using established fairness, safety, and bias benchmarks (e.g. DiscrimEval). We'll also use capabilities benchmarks (MMLU, GPQA, IFEval, HumanEval) to see whether or not this degrades capabilities.

Concrete outputs:

• An open-source pipeline on a git repo with all data around SFT experiments, including reports on our eval suite spanning animal welfare, human safety, and capability benchmarks
• A write-up (blog post) or published paper (arXiv) reporting our results