A model that emits a calibrated probability that it is correct, so an agent can abstain or defer instead of acting on an overconfident guess.
A model that emits a calibrated probability that it is correct, so an agent can abstain or defer instead of acting on an overconfident guess.
One of the most fundamental limitations of LLMs is their lack of an internal, calibrated sense of confidence. Being overconfident when making decisions and taking actions makes AI systems unreliable and unfit for safety-critical applications. Common solutions rely on post-hoc monitoring systems or token probabilities, which are weak proxies for the system's own internal confidence. In this project, we will decode confidence signals directly from the internal representations of the LLM. We build on how confidence works in both biological brains and artificial networks: in the brain, our sense of confidence is computed from the same signals used to make a decision (Kepecs and Mainen, 2012); in machine learning, we build upon our own work on uncertainty calibration (Brito, ICML 2025; ICLR 2026). Target confidence signals will be uncovered from LLM hidden layers using mechanistic interpretability tools such as sparse autoencoders. We will build a model that outputs a calibrated confidence score alongside each response, and test it on Q&A benchmarks with known ground truth. The project will be led by Carlos Stein Brito, in collaboration with Zachary Mainen (Champalimaud Foundation) and Lorenzo Fontolan (Aix-Marseille University).
An autonomous AI system does damage when it acts on a wrong belief while appearing just as sure as when it is right. The safety mechanisms meant to prevent this, deferring to a human, triggering an oversight check or stopping before an irreversible action, all require that the system can tell when it is likely to be wrong. Current models are not able to do this by design. Token probabilities measure how fluent a sentence is, not whether it is correct, and they shift with rephrasing, so they are a poor stand-in for the model's actual reliability. This project overcomes this limitation by designing and building a calibrated confidence signal read from the model's internal representations. With it, an agent can gate its own actions on how likely it is to be right, which is the prerequisite for safe abstention, human oversight and corrigible deferral.
Team Member
The minimum, $25,000, covers my salary part-time over four months and $5,000 of cloud GPU for LLM simulations and evaluation. The ideal, $60,000, funds eight months of my salary and $10,000 of compute for larger training runs and a broader benchmark of high-stakes tasks.
No comments yet. Be the first to share your thoughts.