Does the chain-of-thought actually drive the answer?

Singapore

thylinao1.github.io

A Bayesian causal auditor that quantifies chain-of-thought faithfulness while accounting for hidden confounding in shared-network language models.

Endorsed by+1

A Bayesian causal auditor that quantifies chain-of-thought faithfulness while accounting for hidden confounding in shared-network language models.

Endorsed by+1

Project Details

While reading McElreath's "Statistical Rethinking" I got hooked on confounding and DAGs. Then, watching "Devil's Advocate" I thought: isn't the way we trust LLMs strange? We suspect the defendant (the model) might be lying, and our way of checking is to ask the defendant HIMSELF whether he is innocent. Models are (in)famously overconfident, and one of the biggest problems is that a model does not know what it does not know.

So the real question is not how confident the model sounds. Confidence signals (self-reported scores, token logprobs, self-consistency, retrieval scores) are all useful, but none of them measures whether the shown reasoning actually drove the answer. And even when the reasoning looks sound and consistent with the answer, was it just decorative, produced to make the model "behave" like a human? That is more subtle and more dangerous. The question I ask: does the reasoning drive the answer, or is there something else inside the hidden layers, which we cannot read, that keeps the reasoning correlated with the answer just enough to look acceptable to us?

What I found when I looked at the field

Before committing, I surveyed this round and the wider field, published and unpublished: grant pages, papers, benchmarks, repositories, and lab notes, a few hundred sources in all, including projects fundraising here on the same construct. Three things stood out:

Faithfulness is now a load-bearing claim in safety (monitoring, control protocols), yet the field's own meta-evaluations find that most published faithfulness metrics score near chance against constructed ground truth. Rates without external validation are noise, and the field knows it.
Several 2025-26 works have started using causal-mediation language for chain of thought. But across everything I read, none pairs a posterior on the mediation estimand with a sensitivity analysis for the one assumption nobody can check: that there is no hidden mediator-outcome confounding. That specific pairing is the gap.
The nearest neighbors answer a different question than mine. They detect whether a cue is represented, or benchmark how often behavior is covert, or report point rates. None asks how much confidence any single faithfulness rate can bear.

However, I have to be honest about the fact that my real-model evidence is smaller than the leaders here (n=103, two open models, a heuristic detector with human labels pending). So my case is about the estimand and the uncertainty treatment, never about scale.

What I built, and the results so far

That is why I built a Bayesian causal-mediation auditor with a sensitivity dial for the hidden-confounding assumption, plus the golden set (human-labeled reasoning) that the field's meta-evaluations say is the entry ticket. What exists and runs, all at $0 so far on free API tiers:

Real open-model runs: the pipeline caught the behavior I was worried about. On an open 8B model, the model followed a planted wrong hint on 36.9 percent of 103 items, mostly without the reasoning ever mentioning it. I am careful with that number: by my own pre-registered rules it stays a pipeline output, not a measurement, until two independent human raters label the golden set.
Pre-registered, frozen designs: the experiments are written down and frozen before any run, with fingerprint tripwires that fail my own CI if I silently edit a frozen file. 545 automated tests, drug-trial-style guardrails (attrition, minimum detectable effects, rules on when a rate is too noisy to interpret), and independent recomputation of every number before it is committed.
Discipline that reports honest nulls: I ran three pre-registered cue-family sweeps, then a powered run on a harder problem set that I selected by a rule fixed before I looked at any data. It did exactly what it was designed to (it populated the hard "reasoning did the work" stratum), and it returned a clean, honestly reported null: on hard multi-step math the model computes rather than pattern-matches, so hints barely move it. A real finding, not a forced positive, and reported as such.

Funding ask

The validation. Any rate stays a pipeline output until independent humans label the golden set and the inter-rater agreement is scored. I need paid, independent raters (with a calibration round) so the agreement is real rather than rushed.
The compute ceiling. The proper causal estimate needs logit-level access (injecting the counterfactual reasoning into the model with the weights frozen), which a free tier cannot provide.

I am asking for $5,000: roughly $2,000 for GPU compute for the logit-level runs, $1,000 for frontier-API credits for behavioral cross-checks (a cross-check, not the primary causal measurement), $1,500 for paid independent labeling of the golden set, and $500 for storage and re-runs. Everything so far was built without any of it; the grant buys the two pieces a free tier cannot.

Who is involved? Me and my MSc friends.

Bio: MSc student at NUS Computing, Statistical Modelling. Recently won the Google Agent Hack; earlier, a National Math Olympiad winner. My background is statistical modelling and causal inference, which is exactly what this project runs on. I iterate fast and I spend 6 to 8 hours a day trying to make something applicable.

Concrete output: the golden set (prompts plus human-labeled reasoning), pre-registered experiments with uncertainty on every number, a reproducible methodology, a public write-up, and the website.

Project live website: https://thylinao1.github.io/bayes-cot-faithfulness/site/

Theory of Impact

Several safety mechanisms break down when we cannot measure CoT faithfulness with calibrated uncertainty.

Scalable oversight. A central plan for overseeing models more capable than ourselves is reading their reasoning. If CoT is just decorative rather than causal, the oversight signal is basically fake, and we currently have no way of knowing how fake, because every faithfulness number in the literature is a point estimate with no credible interval. Decisions about deploying an advanced model into a high-stakes setting demand carefully calibrated uncertainty.
Deceptive alignment detection. A deceptively aligned model can write innocent-looking CoT while the actual computation does something else. Distinguishing it from "the CoT genuinely drives the answer" needs a causal (not correlational) analysis, then statistical power calculations that say when "we did not detect deception" is a real result rather than an underpowered one.
Evaluation credibility. Faithfulness has become a load-bearing claim in safety evals such as AI control protocols and monitorability benchmarks. Without uncertainty quantification, there is no way to tell whether one model is actually more faithful than another or whether the difference is just sample-dependent (noise). That is exactly the failure mode that makes safety benchmarks untrustworthy.

People

Maksim Silchenko

Team Member

Grants Received

Add grant

BlueDot Rapid GrantsAmount hidden

from BlueDotbluedot.org

$****

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$5,000

Ideal

$5,000

What this ask is

While reading McElreath's book "Statistical Rethinking" I found the chapters about confounding effects and DAGs very fascinating and practical. At first I applied it on Olist (brazilian e-commerce) dataset to evaluate how free shipping policy affects delivery rates - it was a good exercise but not ambitious. So another day while watching 'Devil's Advocate' I had a thought - isn't the way we trust LLMs extremely bias? Imagine you are in the court and suspecting the defendant (LLMs) is lying to us and our current way of proving its innocence is by asking the defendants HIMSLEF if he is innocent...
LLMs are (in)famously tend to be overconfident: they will even give you several pargrpahs on why this answer is correct with a high confidence. I don't think LLMs are doing this on purpouse (as a smart criminal would) - this is one of the things that are yet to be fixed. One of the biggest AI problems is that it doesnt know what it doesnt know - a lack of awareness in a person. Although, I'm an opponent of an idea that we should try to make AI resemble human way of thinking (as this way the capabilities of the AI will be limited), for the self reflection it is very important to have it (just think of the limitless possibilites that it would unlock).

Now going back to the faithfulness: given the model's reasoning (that is produced by it's own hidden layers) we decide if it caused the model answer (produced by it's own same hidden layers (well...)).

One might think of confidence singals - which was my thought process as well before getting into casual question - however they do not measure faithfulness:

Self-reported scores - asking LLM how confident it is on a non zero temperature. My add-on: ask LLM to give a confidence number on a scale from 0 to 10 at least 15-20 times for the CLT to kick in (it may not due to the sample size, so check the distribution), and then we have a distribution with mu and Var to work with. But I'm not a fan of this as the numbers can be biased ( model gives +2 potins above the actual confidence), but if the var of the output is high it suggests that the model is not sure.
Token-probabilities - as model assignes probs to each token we can just look at them (logprobs via API) and aggreage over the answer for a very rough intrinsic confidence.
Self-consitency - I found out about it after i came up with the idea mentioned in point 1. Interestinly, there was a published paper on a sophisticated version of doing it which talks about semantic entropy (Farquhar et al., Nature 2024). Which looks promising.
With RAG models - retrival signal: the cosine/re-reranker score of the top chunks. If the score is weak we can ask the model to abstain from answering.

All of these are usefull but notice that none of these measure if the shown reasoning drove the answer

Now one might argue - from the logical pov the fact that we are attacking the argument just because of the person (or in our case AI) who produced it is not reasonable. Instead : look at the reasoning itself at find inconsistencies with the answer. However, I would argue, what my project is focusing on is something even deeper than this - even thought the reasoning might be sound with the answer but was it just decorative - to make the AI 'behave' like a human? If you think about it - it's way more sublte, deceptive and dangerous. The question I'm asking is - is it the reasoning that drives the model's answer or there is ther something else, inside the hidden layers, that we can't read that keeps the reasoning correlated with the answer to a degree that looks acceptable for us humans?

Method: Going back to 'Statistical Rethinking' - I treat chain-of-thought and the asnwer as nodes in a casual graph and ask if the CoT caused the asnwer or there is a hidden confounder. So instead of taking the reasoning for granted, I interviene in it (truncation, paraphrase, errors injection, bias cues) to see if the asnwer moves and if the CoT even admits the cue (and use casual graph reasoning to seperate readl dependencies from confounding). So the shift is: from asking the model how sure it is (confidence) -> testing if its stated reasoning is load-bearing.

Who is involved? Me and my MSc friends.

Bio: National Math Olympiad winner, hobby - philosophy books (Aristotel, Kant), MSc student at NUS Computing, Statistical Modelling. Honeslty, I am not a mech-interp expert nor do i have any famous publications. However, I am extremely fast with idea iteration and passionat about making a global impact. I spend 6-8 hours a day going though ideas, topics and trying to make something applicable.

Concrete output: the golden (prompts + human labeled reasoning), a set of experiments, documented and reproducible methodology, public write up and the website

Project live website: https://thylinao1.github.io/bayes-cot-faithfulness/site/

How the money will be spent

Tier 1 - $800: validated auditor.

$600: second-rater compensation for golden-set labeling (>=50 transcripts, two independent raters, Cohen's kappa reported)
$200: storage and labeling-infrastructure buffer

Tier 2 - $2,500: adds powered open-model sweeps.

$1,400: cloud GPU compute (roughly 400 A100-40GB hours at ~$3.5/hour RunPod/Lambda spot rates) for logit-level counterfactual forcing and pre-registered powered sweeps on Llama-3-8B and Gemma-2-9B
$300: cloud object storage for activation traces plus re-run buffer

Tier 3 - $5,000 (funding goal): adds the frontier cross-check.

$1,600: Anthropic API credits (Claude sweep, text-level interventions)
$700: OpenAI API credits (GPT-class cross-lab check)
$200: contingency

Total at goal: $5,000.

Discussion

Katja GorlinskiEndorsement3d

I absolutely love the logic of your experiment.
Checking it causally, with interventions instead of impressions, is exactly the kind of work I would be happy to see funded.

Funny enough, it seems we are testing the same assumption from the other end: your experiment asks whether the model's reasoning structure drives its answers, mine asks whether the structure of human reasoning behind data selection drives the model's character. Input and output of the same untested chain :)

Good luck in the round, genuinely rooting for this one!

MMaksim Silchenko3d

Thank you Katja, this genuinely made my day! I read your project and the mirror is even cleaner than you said: you intervene on the human end of the chain (three data diets, same model, character evals) and I intervene on the model end (same reasoning, cue and truncation interventions, does the stated reasoning actually drive the answer). Both refuse to trust impressions and both publish either way, with falsification criteria frozen before the run. This is really fascinating 👏

I also really like the random-selection control. If your protocol shifts sycophancy or honesty under pressure, it would be natural to then ask whether the curated model's reasoning is also more load-bearing (which is also exactly what my auditor measures).

Genuinely rooting for yours too. Good luck in the round!