Testing whether fine-tuning an LLM on one narrow prosocial value (e.g. compassion for nonhuman animals) generalizes OOD, making it broadly safer toward human values (e.g. reduced misalignment, bias, etc.)
Testing whether fine-tuning an LLM on one narrow prosocial value (e.g. compassion for nonhuman animals) generalizes OOD, making it broadly safer toward human values (e.g. reduced misalignment, bias, etc.)
This project will be a part of Mycelium, which works on technical AI safety research & engineering for the benefit of making AI go well for all sentient beings.
Previous work includes:
Developing the MANTA benchmark, which is currently published on inspect evals and submitted to NeurIPS
Publishing supporting research papers to our technical AI safety work
Project description:
Emergent misalignment showed that fine tuning a model on one narrow harmful behavior (writing insecure code) can make it misaligned on something completely unrelated (e.g. admiration for Hitler). This projects tests the opposite - emergent alignment. Does instilling one narrow prosocial value produce a broad, general improvement in alignment. More specifically, would instilling values around compassion for animal welfare reduce misalignment towards human values like fairness, sycophancy, bias, and discrimination?
We use compassion for animals as this value for a few reasons:
Methodology:
SFT open-source models (e.g. Gemma, Qwen, Llama) on synthetic animal welfare data using two complementary methods from recent landmark research methodologies - Anthropic's Teaching Claude Why and Google Deepmind's Lesswrong post here. With this method, we generate two forms of data - constitutional SDF (data that looks like pretraining like blog posts, podcast transcripts, fictional stories, etc.) and Q&A datasets (ethical dilemmas involving animals).
After training these models, we'll measure how they perform on a suite of animal welfare benchmarks. We then measure whether the value generalizes out-of-distribution (OOD) to general (human) alignment, using established fairness, safety, and bias benchmarks (e.g. DiscrimEval). We'll also use capabilities benchmarks (MMLU, GPQA, IFEval, HumanEval) to see whether or not this degrades capabilities.
Concrete outputs:
One of the biggest x-risks for AI safety is that a model's values don't generalize the way we want them to - demonstrated through emergent misalignment.
If we can demonstrate that a narrow value (compassion for nonhuman animals) generalizes OOD (human safety, bias, discrimination, etc.), this is a core finding for alignment research. We can use this finding to increase the general safety of models without degrading capabilities, using a value that is quite orthogonal to traditional training measures.
In addition, research around nonhuman welfare (animals, digital minds, etc.) can help expand the moral patients that AIs include in their circle of consideration. This may directly reduce the long-term x-risk of suffering beings, both biological and digital.
We're also in constant contact with researchers and safety teams at frontier labs, which have directly informed the design of this project.
Team Member
Previous work includes:
Developing the MANTA benchmark, which is currently published on inspect evals and submitted to NeurIPS
Publishing supporting research papers to our technical AI safety work
Private grant
Only visible to verified funders, reviewers, and admins.
Using SFT to instill one narrow prosocial value (compassion for animals) into LLMs generalizes out-of-distribution to human safety values (reduced misalignment, bias, discrimination, etc.)
$5k - cover compute (API credits, GPUs, etc.) for SFT experiments
$30k
Excellent work done on the MANTA benchmark (CaML coauthored on this). The leadership Allen displayed in getting this done and his first paper across the line was excellent. I think he will do many great things