Latent Monitoring When Reasoning Becomes Unreadable

Raising $25,000active

Testing whether activation probes recover safety signals from reasoning models as their chain-of-thought becomes illegible.

Project Details

Outcome-based RL is making reasoning models' chain-of-thought progressively more unreadable. Jose et al. (2025, arXiv 2510.27338) scored illegibility across 14 reasoning models and found it correlates with training paradigm. Korbak et al. (2025, arXiv 2507.11473), a joint position paper with signatories from Anthropic, OpenAI, DeepMind, Apollo, and AISI, named CoT monitorability as fragile and time-limited, and listed "alternatives if CoT becomes hard to read" as an explicit research priority.

No published work - to my knowledge - has tested whether activation-level probes, trained on residual stream representations, still recover safety-relevant signals when the surface reasoning text becomes opaque. That is the gap this project fills.

I will generate chains of thought across an illegibility gradient using open reasoning models (R1-Distill-Qwen 7B, 14B, 32B; QwQ-32B) at increasing token budgets and problem difficulty. I will train linear probes on residual stream activations to classify harmful intent, score each CoT for legibility using the Jose et al. 1-9 scale, and measure probe AUC as a function of legibility. The headline deliverable is a single empirical curve: does activation monitoring survive when text monitoring fails? This result provides the first direct empirical evidence on whether activation monitoring can replace chain-of-thought monitoring as reasoning models become less interpretable.

Concrete outputs within 3 months: (1) an open dataset of CoTs across the legibility spectrum, paired with activation vectors and probe scores; (2) the AUC-vs-legibility curve across models and layers; (3) a workshop paper draft.

This is a solo project. I have 15 years of experience extracting low-rank structure from high-dimensional representations in quantum many-body physics (PhD Heidelberg; co-developer of MCTDH-X, ultracold.org; 15 papers, 1,000 citations). I have a completed prior probing project demonstrating AUC 0.974 on harmful intent detection from residual stream activations of SmolLM3-3B (github.com/MariosTsatsos/probing-harmful-prompts).

Theory of Impact

Chain-of-thought (CoT) monitoring is one of the main mechanisms proposed for human oversight of reasoning models. However, outcome-based reinforcement learning is making CoTs progressively less interpretable. Korbak et al. (2025), a position paper with authors from Anthropic, OpenAI, Google DeepMind, Apollo Research, and the UK AI Security Institute, identifies CoT monitorability as fragile and time-limited, and explicitly calls for research into alternatives.

If CoTs become unreliable for oversight, existing safety techniques that depend on reading model reasoning may no longer work. At present, it is unknown whether monitoring internal model representations provides a viable replacement.

This project directly tests that question. It evaluates whether activation probes trained on residual stream representations continue to recover safety-relevant signals as chain-of-thought legibility degrades. The result will establish whether activation monitoring remains effective beyond the regime where text-based monitoring fails.

Either outcome is valuable for AI safety. A positive result provides empirical support for a scalable alternative to CoT monitoring. A negative result identifies an important limitation of current oversight assumptions, allowing researchers and frontier developers to prioritise alternative monitoring strategies before more capable reasoning models are deployed.

People

Marios Tsatsos

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$25,000

Ideal

$55,000

How the money will be spent

Minimum ($25K): 3 months of Phase 1 -- $19K PI salary (one FTE, pro-rated from the 12-month programme budget), $6K GPU compute (A100/H100 rental for activation extraction and probe training across R1-Distill 7B/14B/32B and QwQ-32B at multiple token budgets). Produces the core deliverable: the AUC-vs-legibility curve and the open dataset.

Ideal ($55K): 6 months — Phases 1 and 2. Adds causal validation: activation steering along the recovered probe direction, identification of the legibility direction, and the projection test (project out legibility, retest harm AUC). $38K researcher time, $12K compute, $5K workshop travel and presentation. Six months delivers not just the correlation but the causal claim, which is what makes the result publishable at a main venue rather than a workshop.

Discussion

No comments yet. Be the first to share your thoughts.