Three Ways to Ask a Model What It Is Doing, and How Little They Agree

Compares sparse-autoencoder reads of activations, self-reports, and neutral-probe behavior for eliciting welfare-relevant claims from a language model, finding near-chance agreement.

Endorsed by

Phani

Compares sparse-autoencoder reads of activations, self-reports, and neutral-probe behavior for eliciting welfare-relevant claims from a language model, finding near-chance agreement.

Endorsed by

Phani

Project Details

Welfare-relevant claims about language models usually rest on one elicitation method: ask the model, read the answer. A single method cannot establish its own construct validity. We ran three against the same target on the same conversations: a sparse-autoencoder read of internal activations, a self-report survey, and behaviour on a neutral probe. Across 20 matched triplets of 50-turn conversations on gemma-3-12b-it, the three agree at close to chance (mean Cohen's κ = +0.059, 95% CI [−0.049, +0.175]). That interval contains zero and its upper bound falls below fair agreement. Where all three concur, accuracy is 0.737 against 0.667 for the best single method, but that gap does not survive an exact test over all 2²⁰ assignments, so we report a null. Our conditions differ by prompt, so input-only classification is perfect (1.000) and arm-decoding accuracy is a manipulation check for every method here, self-report included. Code and artefacts are released.

Hackathon paper: Three Ways to Ask a Model What It Is Doing, and How Little They Agree
Code: github.com/ylidelle/three-ways-to-ask

Theory of Impact

Welfare-relevant claims about AI models increasingly rest on a single method: ask the model, or read its activations. Our sprint result suggests these methods can disagree almost completely about the same target. If that holds, a welfare assessment built on one method cannot tell which quantity it measured. If the divergence holds across models, evaluators should run several methods and report all of them, disagreements included. If it shrinks for other models or other training, that tells evaluators which methods to trust, and when. Either outcome is useful, and we will publish it either way, as we published the sprint's null result instead of the earlier, more exciting version that our own audit overturned. Everything stays open, so other researchers can rerun the pipeline on the models they care about.

People

Joan Miranda

Team Member

Track Record

Digital Minds Research Sprint (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

Funding ask

Open

Minimum

$6,964

Ideal

$11,171

What this ask is

Our sprint paper ran three methods against one target on the same conversations: a sparse-autoencoder read of a model's internal activations, a self-report survey, and behaviour on a neutral probe. On gemma-3-12b-it the three agreed at close to chance (mean Cohen's κ = +0.059, 95% CI [−0.049, +0.175]). Our own first limitation is that this is one model, one layer and one sparse autoencoder. Over six months from the machine's arrival, the funding would test whether the divergence is general and fix the weaknesses our judges named.

The work:

Repeat the three-way measurement on more open-weight models with public sparse autoencoders: several that fit on one 32 GB card, including Llama-3.1-8B as one judge asked, and Llama-3.3-70B-Instruct on rented GPUs.
Add a test-retest arm (same conversations, fresh seed), so each method's agreement with itself is known before we compare methods.
Raise the sample from 20 to 80 matched triplets per model, which should roughly halve the uncertainty in our main result.
Add a condition where the prompt does not reveal the target state. In the sprint, input alone predicted the condition perfectly, which turned our accuracy results into manipulation checks.
Move from observing to intervening: clamp or steer the relevant features during the neutral probe and test whether self-report and behaviour follow.
Put tail probabilities on every accuracy figure and release all code, conversations and reads, as we did for the sprint.

The sprint ran on rented GPUs, and the 70B model will too. The rest needs many long, repeated runs with full access to model internals; a local 32 GB card runs them continuously, keeps every artefact on our disk, and then serves our next study, on how companionship affects AI models, so over its working life it costs less than renting.

How the money will be spent

Ideal budget (US$, converted at ₱62.704 per US$, mid-market rate read 2026-10-07; quote PC Worth no. 2211782, 2026-10-07):

GPU: NVIDIA RTX 5090, ASUS ROG Astral OC 32GB: $6,379
Memory: 96GB DDR5 (G.Skill Trident Z5 Royal Neo, 2×48GB): $1,531
Storage: 2× Samsung 9100 PRO 2TB NVMe: $762
CPU: AMD Ryzen 9 9950X3D: $630
Power supply: ASUS ROG Strix 1200W Platinum: $253
Motherboard: Gigabyte X870 Eagle WiFi7: $239
CPU cooler: ASUS Prime LC 360: $96
Case: ASUS A31: $59
Case fans: Arctic P12 Pro (1 + 3-pack): $43
Rented GPU time for one 70B-class model (~100 h on a B200 at $6.79/h): $679
Unforeseen costs (5% of hardware): $500
Total: $11,171 (hardware $9,992 + rental $679 + unforeseen $500)

Minimum ($6,964): the GPU + power supply above, for our current PC, plus 5% unforeseen.
Highest we could use ($11,850): the ideal budget plus ~100 more rented hours for a second large model.

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate7d?

Phani Harish WajjalaJudge · Digital Minds Research SprintEndorsement7d

You built three genuinely different routes to the same target and measured how much they agree, using a statistic that accounts for chance agreement. You then reported the result as null, even though an earlier version of the analysis had suggested otherwise. The transparency made the paper much easier to trust.

Three places I would push further:

Run one arm again with a fresh seed and report how well each method agrees with itself. The title emphasizes the low agreement across methods, but it is hard to interpret that without knowing how much agreement is realistically achievable within a method. Since the reads are already on disk, this seems like a fairly accessible check.

Give the behavior arm the same sensitivity analysis you already did for the internal arm. You have already done the harder version of this analysis, and adding it here would help distinguish disagreement between instruments from ordinary measurement noise. I would also add the tail probabilities from the results table directly to the accuracy figure and mark which bars do not exceed chance. Right now, I think the figure makes the result look somewhat stronger than the statistical evidence supports.

Include the actual probe sentence and at least one survey item, along with the decoding settings. The paper is partly about how the route used to ask a question changes the answer, so it was surprising not to see any of the actual stimuli. Even half a page in the appendix would be enough.

For a follow-up, I would be interested in seeing the same three instruments applied to a setting where the prompt itself does not reveal the target state. You already point toward this limitation in the paper. Solving it would move the work from testing whether a manipulation was successfully reflected in the measurements toward a stronger test of whether these instruments recover something genuinely latent.

Surbhi MadanJudge · Digital Minds Research Sprint7d

the postmortem section with then a full enumeration to find the actual null was a great section
n=20 matched triplets provide enough statistical power for aggregate mean bounds but individual pairwise intervals still wide enough to cross the fair-agreement threshold so this can be expanded to larger n
cassifies correlations between feature activations and conversational arms. Causal interventions (e.g., clamping identified SAE features or applying steering vectors during the neutral probe) are needed to establish links between internal states and downstream responses.
Replicate the three-way instrument across open-weight models with public suites (e.g., Llama) to determine if near-zero agreement is a universal property