Readable but Not Causal: Self-Attributed Welfare Representations

Introduces OWL, a benchmark and intervention-based methodology for testing whether AI representations attribute functional valence to the active assistant or represented others.

Endorsed byHHayley

Naman

Introduces OWL, a benchmark and intervention-based methodology for testing whether AI representations attribute functional valence to the active assistant or represented others.

Endorsed byHHayley

Naman

Project Details

Welfare-like internal representations are increasingly studied as candidate evidence about AI systems. Their entity attribution—whether a valence state belongs to the active assistant or to a merely represented other—is unresolved. We introduce OWL, a factorial SELF × OTHER × surface-affect benchmark with held-out scenario families and rendering controls. Using linear probes, factorial representation decomposition, and natural counterfactual activation interchange in a learned attribution subspace, we evaluate predictive and causal attribution of functional valence. Linear probes decode the active assistant's functional outcome with test AUROC 0.859 versus 0.601 for a represented other and survive every rendering control (0.848–0.999), yet causal interchange in the learned S subspace recovers only 0.18 of the natural counterfactual effect on self-directed choice (full-vector patch 0.42, random ≈ 0). Readability without causal efficacy implies that internal valence probes require intervention-based validation.

Hackathon paper: Readable but Not Causal: Limits of Self-Attributed Welfare Representations in Language Models
Code: github.com/Vishwa-docs/owl-benchmark

Theory of Impact

AI-welfare measurement is upstream of every downstream decision: funding, monitoring, and intervention all depend on correctly attributing internal states. Today's content-only valence detectors conflate a represented other's distress with the assistant's own, risking both over-attribution (spurious welfare claims and protections) and under-attribution (missed harms). OWL shows these can be separated predictively but not yet causally—stopping researchers from building interventions on decoded-but-not-causal signals.

If we locate the causal owner-binding mechanism and validate the ownership-conditioned monitor, the field gains a construct-valid measure of whose state a model represents, with stated identification assumptions and failure modes. Internal-state evidence becomes usable—or clearly not—for welfare claims, which is what funders, labs, and policymakers need before treating such signals as evidence. The open benchmark also reduces risk by making owner/content attribution testable and reproducible across the ecosystem.

People

Vishwa Kumaresh

Team Member

Track Record

Digital Minds Research Sprint (Apart Research), 2026 — 🏆 1st Place Winner

Grants Received– no grants recorded

Add grant

Funding Asks

Readable but Not Causal: Limits of Self-Attributed Welfare Representations in Language Models

Open

Minimum

$3,000

Ideal

$8,000

What this ask is

OWL showed that a readable owner tag is not the linearly bound mechanism the model consumes. Funding would let us resolve the central open question—where (and whether) the model's own decision-making actually binds the owner of a valence state—and turn the benchmark into a validated, generalizable measurement tool.

Find the real mechanism. Extend the protocol from linear attribution subspaces to distributed representations using Boundless DAS rotation fitting, testing whether owner binding is distributed/nonlinear. Add KV-cache branching to probe the temporal identity of welfare-like states. This determines whether internal valence evidence can be causally attributed at all.
Generalize. Run the full predictive + causal + monitor protocol across model families (Qwen3 variants, Llama, base vs. post-trained checkpoints) to establish cross-family transfer of owner directions and of the readable-but-not-causal pattern.
Validate the monitor. Expand the distress hard-negative set (more templates, more kinds) to give the ownership-conditioned monitor the statistical power it currently lacks (the H5 difference CI included zero), and evaluate it under CIF-style identification assumptions before any deployment claim.
Release a standard. Package OWL as an open benchmark—prompt sets, protocol, eval harness—so ownership attribution of welfare-like states is reproducible across labs.

Deliverables: a mechanism/interchange paper, a monitor-validation paper, an open eval library, and preregistered cross-family results.

How the money will be spent

GPU compute for Boundless DAS fitting + cross-family activation capture (≈2 rented A100/H100 months): $4,000
Evaluation and replication labor (LLM eval runs, human annotation of distress templates, preregistration): $2,500
Open benchmark release and hosting (eval harness, artifact DOI, small API): $500
Dissemination (ICLR submission fees, conference/travel support): $1,000
Total: $8,000 (min ask $3,000 covers compute for the mechanism search)

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?

HHayley ClatterbuckJudge · Digital Minds Research SprintEndorsement15d

Motivation for the project is clear and effectively argued. Probes for internal welfare states don't distinguish between representations of the model's own welfare states and others welfare states. Authors show that the model can distinguish ownership but that the model's "own" welfare states don't disproportionately cause its own actions. This is very useful work, though more investigation is needed into the causal relationships would be between model welfare states (if any exist) and model behavior, given assistant training, safety training, and other protocols that may prevent models from acting on their own states. It would have been good to see more detailed presentation of the OWL benchmark used.

Naman AhujaJudge · Digital Minds Research SprintEndorsement15d

This is an unusually strong sprint submission that addresses an important construct-validity problem for AI welfare measurement: distinguishing whether a valenced representation belongs to the active assistant or merely to another represented entity. I particularly appreciated the factorial SELF × OTHER × affect design, held-out scenario families and unseen renderings, lexical baselines, natural-counterfactual activation interchange, full-vector and random controls, and the explicit separation between predictive decodability and causal use.

The "readable but not causal" result is compelling: SELF outcome is strongly decodable across held-out conditions, yet intervention in the learned SELF subspace recovers only a small fraction of the natural behavioral effect. The fact that the full-vector patch recovers more and random controls are near null makes the negative causal result substantially more informative than probe accuracy alone.

I would nevertheless narrow the mechanistic interpretation slightly. Failure of the rank-1 SELF subspace demonstrates that this linear direction is not sufficient for the downstream behavior under the stated intervention, but it does not uniquely establish distributed/nonlinear owner binding. The full-vector intervention itself recovers only part of the natural counterfactual effect, so information outside the selected anchor/layer or other violations of the stated identification assumptions remain possible. Multi-layer/token patching and learned distributed subspaces such as Boundless DAS would be particularly useful follow-ups.

Cross-family replication would also materially strengthen the result, as the current study is limited to Qwen3-8B. The ownership-conditioned monitoring result is promising but appropriately reported as nonsignificant; expanding the distress hard-negative set would help determine whether the apparent false-positive reduction generalizes.

Overall, this is a rigorous and valuable contribution, especially because it demonstrates why strong probe performance should not automatically be interpreted as evidence that the decoded feature is the causal representation used by the model.