Surrogate base model for Mechanistic Interpretability

Context

Activation Oracles (AOs) aim to uncover harmful behaviours from AI models, and they are currently tested on model organisms, i.e., otherwise-safe models fine-tuned to exhibit unwanted behaviours. Recently, I showed that AOs don’t reliably uncover unwanted behaviors if we remove the assumption of having a safe model to train the AO on. This assumption is unrealistic for two reasons: (i) it is not obvious when, during an actual training, an unwanted behavior arises, (ii) undesired behaviors do not arise in isolation.  For example, some behaviors emerge during pretraining, making it especially hard to extract a model without unwanted behaviour.
However fragile, the assumption of having a safe model at disposal is not unique to AOs. For instance, activation difference steering (ADL), patchscopes, SAEs all rely on the availability of a pre-malign-initialized model A.

Henceforth, let A be a pre-malign-initialized safe model, and B be a malign model trained from A that exhibits an unwanted behaviour X.

Project Summary

The project investigates whether it is possible to create a model C to use as a safe alternative to model A: starting from B, apply broad concept unlearning using a safe and controlled dataset to remove X. With broad concept unlearning I mean a training phase that aims to remove X by broadly compromising the model’s capabilities in a controlled way. Then use C in place of A for auditing purposes. For example, train an activation oracle from C, or compute the activation differences between B and C for steering or ADL. I will refer to C also as the surrogate base model henceforth.

Challenges and possible failures
The main challenge is calibrating the concept unlearning: removing B’s behaviour X while avoiding catastrophic forgetting. The latter is particularly important: erasing all B’s capabilities could make X’s activations buried among the activations of other erased features and behaviours.

The setting seems easier than normal training: without the objective of improving capabilities, we can tune the training methodology (such as datasets, techniques and parameters) to reduce behaviour X while still avoiding catastrophic forgetting.

The most likely failure mode is X being too hard to isolate. Beyond catastrophic forgetting (failure 1), the unwanted behaviour X may be particularly hidden (e.g. it only activates in a very niche context), making it hard to unlearn specifically (failure 2).

Experimental plan
The experimental setting is split into phases and aims to detect failure modes early.

In the first phase, we make two simplifications to ground the proposed technique in an easier setting:

1. The experiments will be run on ad-hoc MOs, with narrow behaviours (such as the ones studied here). These models are built in a way which makes them arguably easier to audit than real unwanted behaviour in LLMs. 
   If the surrogate base model setting fails with these MOs, I will likely stop the project and publish the failures found.

1. Use targeted concept unlearning interventions, that is, concept unlearning on specific target contexts to directly address the behaviour X of the model. While in a real scenario the unwanted behaviour will be unknown (otherwise there is no need for auditing), here the aim is to understand how much information about the unwanted behaviour is needed to isolate and remove it.

If it is possible to create a surrogate base model with these simplifications, the project can move to the second phase where we try to relax them. I have less confidence in the following experiments, but I expect a better understanding and better scoping abilities after working through the first phase.

Simplification 1 makes the setting significantly far from being real: current MOs haveunrealistically salient activations  that are easy to isolate. This phase can test against the censorship of Chinese models (e.g., DeepSeek), or explore known undesired behaviours such as sycophancy or refusal/over-refusal.

Simplification 2 is more realistic. Even if in a real setting we are unaware of behaviour X, I believe iterative refinement can narrow down the context and uncover behaviour X. That being said, I aim to progressively relax simplification 2 to characterise how much narrowing is needed to actually uncover the behaviour.