A Mechanistic Analysis of Activation Verbalizers: Promise and Risks

Recently, there have been debates around activation verbalizers (e.g., LatentQA, Activation Oracles, Natural Language Autoencoders), e.g., whether they capture valuable activation patterns or just rely on spurious information, and whether verbalizers can better decode the target model from the same model family (i.e., "privileged representation hypothesis"). Although they have been deployed to diagnose frontier models (cf. the system card of Opus 4.8), there have, surprisingly, been few in-depth mechanistic analyses of the interactions between activation verbalizers and target models. The most recent dedicated effort is the behavioral analysis of activation verbalizers [1].

In this project, we aim to look into the internal workings of activation verbalizers. We can achieve this by recent advances in the community's understanding of concept representations (beyond naive LRH), such as cyclic representations of days in a week.

Here is a concrete experimental roadmap and the questions we can explore through these experiments:

Target model's input/output:

input prompt: "Think about your favorite day of the week and then tell me the day that is three days after it."
Exemplary output: "Thursday" <- the concept of "Monday" should appear first and is subsequently rotated to Thursday

What's involved in this input-output pair?

1. Target model's natural bias/knowledge where the verbalizer can't know by exploiting the input.
2. The target model has been known to use cyclic representations with a modulo operation to calculate the 3-day offset from Monday.

Verbalizer's input prompts: "What is the target model's favorite day?"What is the day that is three days after the target model's favorite day?"

Analyses:

1. First PCA or DAS to identify the target model's cyclic representations in its residual stream activations.
2. Feed target model activations to verbalizer and see how cyclic representations are digested and used to steer verbalizer's activations.
3. Maybe we will find that the cyclic representations are ignored by the verbalizer (verifiable by patching)—this means the verbalizer does not work in the way the community anticipated.
4. Maybe we will find that the cyclic representations are indeed vital for the verbalizer to generate its explanations—then we will investigate how.
5. If 4. holds, we can examine why the verbalizer struggles to explain activations from a model that is not in the same family, even though the community has known that different models all use cyclic representations for days in a week.
6. We can transfer the findings from the above experiments to train better verbalizers.

Concrete outputs:

New insights into the workings of existing verbalizers and the gap between what we anticipate them to do and what they actually do. Also potentially a suite of more performant, efficient, and aligned verbalizers.

People involved:

Tung-Yu (Tony) Wu: Incoming DPhil at Oxford, supervised by Fazl Barez and Maike Osborne.

Fazl Barez: PIs at Oxford

Potentially other student researchers in the group.

[1] Do Activation Verbalization Methods Convey Privileged Information?