Testing whether activation probes recover safety signals from reasoning models as their chain-of-thought becomes illegible.
Testing whether activation probes recover safety signals from reasoning models as their chain-of-thought becomes illegible.
Project Details
Updated 07/14/26 · Provided via application · VerifiedOutcome-based RL is making reasoning models' chain-of-thought progressively more unreadable. Jose et al. (2025, arXiv 2510.27338) scored illegibility across 14 reasoning models and found it correlates with training paradigm. Korbak et al. (2025, arXiv 2507.11473), a joint position paper with signatories from Anthropic, OpenAI, DeepMind, Apollo, and AISI, named CoT monitorability as fragile and time-limited, and listed "alternatives if CoT becomes hard to read" as an explicit research priority.
No published work - to my knowledge - has tested whether activation-level probes, trained on residual stream representations, still recover safety-relevant signals when the surface reasoning text becomes opaque. That is the gap this project fills.
I will generate chains of thought across an illegibility gradient using open reasoning models (R1-Distill-Qwen 7B, 14B, 32B; QwQ-32B) at increasing token budgets and problem difficulty. I will train linear probes on residual stream activations to classify harmful intent, score each CoT for legibility using the Jose et al. 1-9 scale, and measure probe AUC as a function of legibility. The headline deliverable is a single empirical curve: does activation monitoring survive when text monitoring fails? This result provides the first direct empirical evidence on whether activation monitoring can replace chain-of-thought monitoring as reasoning models become less interpretable.
Concrete outputs within 3 months: (1) an open dataset of CoTs across the legibility spectrum, paired with activation vectors and probe scores; (2) the AUC-vs-legibility curve across models and layers; (3) a workshop paper draft.
This is a solo project. I have 15 years of experience extracting low-rank structure from high-dimensional representations in quantum many-body physics (PhD Heidelberg; co-developer of MCTDH-X, ultracold.org; 15 papers, 1,000 citations). I have a completed prior probing project demonstrating AUC 0.974 on harmful intent detection from residual stream activations of SmolLM3-3B (github.com/MariosTsatsos/probing-harmful-prompts).
Theory of Impact
Updated 07/21/26 · By grantmaking.aiChain-of-thought (CoT) monitoring is one of the main mechanisms proposed for human oversight of reasoning models. However, outcome-based reinforcement learning is making CoTs progressively less interpretable. Korbak et al. (2025), a position paper with authors from Anthropic, OpenAI, Google DeepMind, Apollo Research, and the UK AI Security Institute, identifies CoT monitorability as fragile and time-limited, and explicitly calls for research into alternatives.
If CoTs become unreliable for oversight, existing safety techniques that depend on reading model reasoning may no longer work. At present, it is unknown whether monitoring internal model representations provides a viable replacement.
This project directly tests that question. It evaluates whether activation probes trained on residual stream representations continue to recover safety-relevant signals as chain-of-thought legibility degrades. The result will establish whether activation monitoring remains effective beyond the regime where text-based monitoring fails.
Either outcome is valuable for AI safety. A positive result provides empirical support for a scalable alternative to CoT monitoring. A negative result identifies an important limitation of current oversight assumptions, allowing researchers and frontier developers to prioritise alternative monitoring strategies before more capable reasoning models are deployed.
People
Updated 07/21/26 · By grantmaking.aiTeam Member
Hello @Marios Tsatsos.
I found your project worth to endorse. Because the question you rise here is the one the big labs named as a priority and left unanswered: does activation monitoring survive when text monitoring fails? - it is inspiring to me how do you see the answer to that question. The discipline of your project is impressive too.
Good luck.
Thank you @Katja Gorlinski for the endorsement.
It was a good feeling to endorse the strong project of yours :)
I'd be genuinely grateful for any support to my project in return, especially your critique.