Investigating Chain-of-Thought Faithfulness in AI Reasoning

Raising $5,000active

Probe an open-weight model’s activations under biasing/cue conditions to test whether chain-of-thought explanations match internal reasoning, releasing paper, code, and datasets.

Endorsed by

Ryan+1

Probe an open-weight model’s activations under biasing/cue conditions to test whether chain-of-thought explanations match internal reasoning, releasing paper, code, and datasets.

Endorsed by

Ryan+1

Project Details

Recent developments in AI have led to Chain-of-Thought (CoT) reasoning becoming increasingly leaned on as an oversight signal. But does it faithfully reflect the model’s actual reasoning? Research from Turpin et. al and Chen et. al suggests that corrupting CoT or introducing cues can affect a model’s answer without being verbalized. I intend to combine activation probing with biasing an open-weight model to cross-check CoT and internal activations.

I’ll be working independently with the help of my mentor in Non-Trivial (selective research fellowship), expert feedback (connected with Martin Tutek already), and fellow peers. As an output, I expect a detailed paper, code repo, and released datasets so my results are reproducible. Any outcome, whether positive or negative, would be insightful in AI alignment: if we can trust CoT, we can more comfortably proceed with scaling AI models; if not, we can further investigate how to ensure CoT remains faithful by extending my results from varied task types.

I’m a student at Carnegie Mellon, so I know how important AI safety is. Moving forward in mech interp, I’ll be using knowledge gained from Leaf's Dilemmas and Dangers in AI course as well as Non-Trivial's fellowship program to guide my thinking.

Theory of Impact

Leading AI models can produce CoT reasoning as visible evidence of their reasoning. While this appears to be useful and reliable, what if this reasoning is generated after the model has already decided (i.e., post-hoc rationalization) rather than a genuine causal account of how it reached its answer? If that’s true, then a lot of oversight proposals that rely on reading AI reasoning would need to be revisited.

If we can ensure CoT reasoning is faithful, then we could see into the black box of AI reasoning. As long as reasoning stays faithful as AI models grow, alignment efforts would be aided by looking at CoT to ensure alignment. Current developments are already shifting towards CoT as a reliable source of oversight, but is this true? If my results are positive, then we can focus more on supporting CoT as an alignment tool; if results are negative, we could focus more efforts on other areas of AI alignment, like evals or scalable oversight.

People

Daniel Zhang

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$5,000

Ideal

$10,000

How the money will be spent

Flexible spending depending on scale and rigor. Money will cover both API costs (for inference, used for all three approaches) as well as GPU usage (mostly for training activation probes on internal weights). Minimum amount to cover preliminary trials, while increases in funding would correlate directly with larger models and more trials. Minimum budget:

3k for API (one frontier, mid-tier, and open model, ran across basic reasoning tasks and cue types)
1k for GPU (~400 hours of H100, to train linear probes on open ~8B Qwen model)
1k for data storage, ~1.5 month stipend, contingency

Ideal budget:

6k for API (multiple frontier, one mid-tier, multiple open models, ran across reasoning tasks and all cue types from Turpin/Chen)
2k for GPU (~800 hours of H100, activation probing across several open models
2k for data storage, ~3 month stipend, contingency

Discussion

Sheikh Abdur Raheem AliEndorsement5d

Practically all of the Frontier Labs are betting on COT monitoring for safety and so faithfulness is an extremely important property to understand. However, 2K for a 3 month stipend seems to be on the lower end for this kind of work. I would encourage making that higher to enable a sustainable commitment.