Surrogate base model for Mechanistic Interpretability

docs.google.com

Creating a reference model for mechanistic interpretability without assuming that at auditing time we have a safe model to compare the suspicious model against.

Endorsed by

Ryan

Gavin

Creating a reference model for mechanistic interpretability without assuming that at auditing time we have a safe model to compare the suspicious model against.

Endorsed by

Ryan

Gavin

Project Details

Context

Activation Oracles (AOs) aim to uncover harmful behaviours from AI models, and they are currently tested on model organisms, i.e., otherwise-safe models fine-tuned to exhibit unwanted behaviours. Recently, I showed that AOs don’t reliably uncover unwanted behaviors if we remove the assumption of having a safe model to train the AO on. This assumption is unrealistic for two reasons: (i) it is not obvious when, during an actual training, an unwanted behavior arises, (ii) undesired behaviors do not arise in isolation. For example, some behaviors emerge during pretraining, making it especially hard to extract a model without unwanted behaviour.
However fragile, the assumption of having a safe model at disposal is not unique to AOs. For instance, activation difference steering (ADL), patchscopes, SAEs all rely on the availability of a pre-malign-initialized model A.

Henceforth, let A be a pre-malign-initialized safe model, and B be a malign model trained from A that exhibits an unwanted behaviour X.

Project Summary

The project investigates whether it is possible to create a model C to use as a safe alternative to model A: starting from B, apply broad concept unlearning using a safe and controlled dataset to remove X. With broad concept unlearning I mean a training phase that aims to remove X by broadly compromising the model’s capabilities in a controlled way. Then use C in place of A for auditing purposes. For example, train an activation oracle from C, or compute the activation differences between B and C for steering or ADL. I will refer to C also as the surrogate base model henceforth.

Challenges and possible failures
The main challenge is calibrating the concept unlearning: removing B’s behaviour X while avoiding catastrophic forgetting. The latter is particularly important: erasing all B’s capabilities could make X’s activations buried among the activations of other erased features and behaviours.

The setting seems easier than normal training: without the objective of improving capabilities, we can tune the training methodology (such as datasets, techniques and parameters) to reduce behaviour X while still avoiding catastrophic forgetting.

The most likely failure mode is X being too hard to isolate. Beyond catastrophic forgetting (failure 1), the unwanted behaviour X may be particularly hidden (e.g. it only activates in a very niche context), making it hard to unlearn specifically (failure 2).

Experimental plan
The experimental setting is split into phases and aims to detect failure modes early.

In the first phase, we make two simplifications to ground the proposed technique in an easier setting:

The experiments will be run on ad-hoc MOs, with narrow behaviours (such as the ones studied here). These models are built in a way which makes them arguably easier to audit than real unwanted behaviour in LLMs.
If the surrogate base model setting fails with these MOs, I will likely stop the project and publish the failures found.

Use targeted concept unlearning interventions, that is, concept unlearning on specific target contexts to directly address the behaviour X of the model. While in a real scenario the unwanted behaviour will be unknown (otherwise there is no need for auditing), here the aim is to understand how much information about the unwanted behaviour is needed to isolate and remove it.

If it is possible to create a surrogate base model with these simplifications, the project can move to the second phase where we try to relax them. I have less confidence in the following experiments, but I expect a better understanding and better scoping abilities after working through the first phase.

Simplification 1 makes the setting significantly far from being real: current MOs haveunrealistically salient activations that are easy to isolate. This phase can test against the censorship of Chinese models (e.g., DeepSeek), or explore known undesired behaviours such as sycophancy or refusal/over-refusal.

Simplification 2 is more realistic. Even if in a real setting we are unaware of behaviour X, I believe iterative refinement can narrow down the context and uncover behaviour X. That being said, I aim to progressively relax simplification 2 to characterise how much narrowing is needed to actually uncover the behaviour.

Theory of Impact

Using unsafe reference would make many mechanistic interpretability (MI) techniques blind by construction. More concerningly, these techniques would produce plausible results that do not show the potential unsafe audited models’ behaviours.

This silent failure mode would expose us to the risk of deploying a model that is not really safe.

Finding a surrogate reference would instead allow techniques to compare it with the audited models, addressing all those settings where it is not certain to have access to a safe ancestor of the model audited. This would significantly increase the number of auditing settings where MI would be effective in uncovering unsafe behaviours.

This project is a preliminary study to understand whether it is possible to create a more realistic setting for these techniques, either suggesting a new area of research or informing about a real weakness in current MI settings.

People

Raffaello Fornasiere

Team Member

Grants Received

Add grant

LASR Labs extension phase grant

from Coefficient Giving

$250,000

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$50,000

Ideal

$96,000

What this ask is

Context

Henceforth, let A be a pre-malign-initialized safe model, and B be a malign model trained from A that exhibits an unwanted behaviour X.

Project Summary

Experimental plan
The experimental setting is split into phases and aims to detect failure modes early.

In the first phase, we make two simplifications to ground the proposed technique in an easier setting:

The experiments will be run on ad-hoc MOs, with narrow behaviours (such as the ones studied here). These models are built in a way which makes them arguably easier to audit than real unwanted behaviour in LLMs.
If the surrogate base model setting fails with these MOs, I will likely stop the project and publish the failures found.

Use targeted concept unlearning interventions, that is, concept unlearning on specific target contexts to directly address the behaviour X of the model. While in a real scenario the unwanted behaviour will be unknown (otherwise there is no need for auditing), here the aim is to understand how much information about the unwanted behaviour is needed to isolate and remove it.

How the money will be spent

The requested amount covers research, living and workplace expenses, and I provide an estimate in the following spreadsheet. Many of the estimates are based on a budget estimation of a previous grant funding request that was accepted by Coefficient Giving.

Surrogate Base Model Budget

I would like to highlight the importance of working at the LISA workspace, which is where I am currently working. This office offers natural feedback loops, since residents often engage with each other’s work. It also allows me to easily meet potential mentors and collaborators. This has been happening for the last 7 months with talented fellows from AI Safety programs and other researchers who work here.

The minimum amount ($50k) would cover roughly half of the overall project expenses. If funded at the minimum, I will probably need to reduce living and research expenses, by developing part of the project outside of London and without working at LISA, reducing the amount of feedback and support received.

Discussion

No comments yet. Be the first to share your thoughts.