Does RL training induce resistance to activation steering?

Raising $5,000active

Test whether task-reward RL trains language models to (a) detect, (b) resist activation steering, and whether this resistance generalizes beyond the trained direction.

Endorsements made here support University of Chicago.

Test whether task-reward RL trains language models to (a) detect, (b) resist activation steering, and whether this resistance generalizes beyond the trained direction.

Endorsements made here support University of Chicago.

Project Details

Language models can sometimes detect when steering vectors are injected into their residual stream (https://arxiv.org/abs/2601.01828). This ability appears to scale with capability, so we can expect it to be more of a problem in the future.

Steering vector detection can also be trained (https://arxiv.org/abs/2511.21399v3). But steering vector detection and steering vector resistance are dissociable. Detection-trained models are actually more steerable, not less (https://arxiv.org/abs/2511.21399v3).

With RL as the dominant "capabilities scaling vector", and activation-level interp beginning to enter RL training loops themselves (https://vmax.ai/team/propel, https://x.com/GoodfireAI/status/2077073035006546005), I believe it is likely that activation-level interventions will be soon applied to models as a part of RL training.

A natural question is what optimization pressure from this steering does to a model's ability to detect and resist these interventions. We might consider sycophancy training as a simple example. Reward models are known to prefer sycophantic responses. A lab that applies an anti-sycophancy steering vector during RL, has placed the intervention in direct opposition to the gradient; so the model's "optimal behavior" now includes defeating the vector. One might reasonably expect the model to learn to detect and resist steering vectors. To my knowledge, there is no work testing whether RL under these conditions improves steering detection/resistance. (The nearest neighbor is probably https://arxiv.org/abs/2602.06941, which does not apply training pressure.)

I propose to test this by training open-weight instruct models with RLVR while injecting steering vectors that are variously opposed to the task goal, and measuring (before and after training) spontaneous detection rates, prompted detection rates, steering vector resistance, and whether acquired resistance generalizes to unseen steering vectors.

Theory of Impact

Mechanistic interpretability techniques assume the model does not learn against the intervention. If RL breaks this assumption, then (a) interp-based safety tooling (i.e. control protocols at labs) can silently fail, and (b) safety cases that rely on interp techniques need to account for this fact, before post-training scales further.

A positive result would pose a strong case for caution against using interpretability techniques for post-training (à la https://www.lesswrong.com/posts/mpmsK8KKysgSKDm2T/the-most-forbidden-technique). A narrow (lack of generalization) result would show that activation-space oversight degrades in predictable ways, so safer RL includes maintaining interventions. Either way, measuring this seems cheap relative to its importance.

People

Nathan Delisle

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$5,000

Ideal

$10,000

How the money will be spent

Because steering vector detection and resistance scale with model size, my experiment would require a model where the pre-training baseline is nonzero, like Llama-3.3-70B.

Costs would come from RLVR, steering (steering makes rollouts expensive) and statistical-measurement related costs (baseline rates of 4% detection imply hundreds of rollouts per condition per checkpoint to detect an effect with reasonable CIs).
All money would be spent on training (Modal/RunPod).

Discussion

No comments yet. Be the first to share your thoughts.