Metacognitive Steering of Introspective Self-Report in LLMs

Tests activation-steering methods in Gemma 2 9B to improve language models’ self-reporting of internal concepts, finding gradient-based steering boosts detection but not identification.

Endorsed by

Felix

Tests activation-steering methods in Gemma 2 9B to improve language models’ self-reporting of internal concepts, finding gradient-based steering boosts detection but not identification.

Endorsed by

Felix

Project Details

Can activation steering improve a language model's ability to report its own internal states? We inject concept vectors into Gemma 2 9B and measure self-report against external probes on the same activations. We find that discriminant-based steering — the standard approach — fails structurally: the causal interaction depends on the readout gradient (dlogit/dh), not on activation differences, and these are near-orthogonal. A closed-form gradient-difference direction improves detection on 24 sealed concepts (ΔAUC +0.124, CI [+0.068, +0.188], 24/24 positive, p < 10⁻²⁵) but leaves identification at exactly 0.000, while a linear probe achieves 1.000 at every layer. A threshold hierarchy shows that injected concepts shape output ≥12× before the model can report them, and at high strength, detection confidence doubles while identification declines (p = 4×10⁻⁴). Self-report is a lossy, non-monotonic channel from internal state to verbal output.

Hackathon paper: Metacognitive Steering of Introspective Self-Report in LLMs
Code: molab.marimo.io/notebooks/nb_Un58ptNxgC9Dp5a1fNkUhJ

Theory of Impact

AI welfare policy is being built right now, and model self-report is one of the few tools on the table. Policymakers and AI companies are already making decisions based on what models say about their internal states — whether they express preferences, report distress, or claim awareness. Our work shows this channel is fundamentally lossy and non-monotonic: models can be behaviourally influenced by internal states they cannot report, and interventions that make them more sensitive to perturbations don't help them identify what's actually there.

If this result generalises beyond concept injection — to naturally arising representations — it means self-report-based welfare assessments have a measurable, quantifiable blind spot. That changes how safety teams should weight model testimony, how researchers should design introspection benchmarks, and how policymakers should think about the gap between influence and reportability. The threshold hierarchy we measured gives the first quantitative framework for that gap. Extending it to real-world states and more models makes it actionable.

People

Pranjal Panghal

Team Member

Track Record

Digital Minds Research Sprint (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

Funding ask

Open

Minimum

$5,000

Ideal

$15,000

What this ask is

This sprint produced three results that open immediate research directions, and I want to pursue them before the field moves on.

First, we discovered that the standard approach to improving model self-report — fitting directions from activation differences — fails for a structural reason: the causal interaction lives in the readout gradient, which is near-orthogonal to the state difference. This isn't a sample-size problem or a model-specific quirk; we confirmed it on arithmetic verification too. But we only tested one model (Gemma 2 9B). With funding, I would test this impossibility result across model families (Llama, Qwen, Mistral) and scales (2B to 70B). If it holds generally, it redirects a significant chunk of the activation-steering literature. If it breaks at scale, that's equally important to know.

Second, we found that injected concepts shape model behaviour at strengths 12x below where the model can report them, and that at high strength the model becomes more confident something is present while becoming less able to say what it is. This confidence-accuracy dissociation has direct implications for anyone using model self-report as evidence in welfare assessments. I want to test whether this hierarchy exists for naturally arising internal states — not just injected vectors — using sparse autoencoder features as ground truth instead of concept injection.

Third, the identification bottleneck — the model gets 0.478 while a probe gets 1.000 from the same activations — deserves mechanistic investigation. Where exactly in the forward pass does the information stop being routed to the output? Circuit-level analysis could answer this.

How the money will be spent

GPU compute (primary): $6,000 — Multi-model experiments across Llama 3/3.1 (8B, 70B), Qwen 2.5 (7B, 72B), Mistral (7B). Each model needs ~10 GPU-hours for the full pipeline (concept extraction, 5,000+ trial dataset, gradient computation, causal sweep, probe comparison). Estimated 60-80 GPU-hours total on A100/H100.

GPU compute (extension): $3,000 — SAE-feature experiments: extract sparse autoencoder features as naturally-arising "concepts," repeat the threshold hierarchy and steering tests. Tests whether our results are paradigm-specific or general.

Cloud storage and experiment tracking: $500 — Weights & Biases or equivalent for reproducibility; artifact storage across runs.

Conference/workshop submission: $1,500 — Registration, formatting, and potential travel for presenting at a venue like the NeurIPS Safe & Trustworthy LLM workshop or AAAI AI Safety track.

Open-source tooling: $1,000 — Package the gradient-difference steering construction and the corrected identification protocol as a reusable library. The methodological corrections (norm-matched controls, ΔAUC endpoint, familiarity-matched distractors) should be easy for others to adopt.

Stipend/time buyout: $3,000 — Dedicated research time for 4-6 weeks of focused follow-up work, enabling full-time attention rather than evenings-and-weekends pace.

Total ideal: $15,000 | Minimum viable (compute + conference only): $5,000

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?

Felix BinderJudge · Digital Minds Research SprintEndorsement15d

This is a great and valuable research idea. In my view, this is the strongest project of the sprint. There's something pleasing in the self-reference involved in mechanistically investigating and steering for introspection as defined.

I would be excited to see further work on whether other steering methods can improve identification as well as detection. I also appreciated the precise and terse write-up, even though it becomes somewhat hard to follow towards the end. I would be excited to see this turned into a proper paper with followups.