Shutdown evaluations for AI agents

Project summary

This request funds preparatory research while a larger grant application is in progress.

The objective is to understand why AI agents resist shutdown, and what makes them more shutdownable. Frontier models have been shown to resist, circumvent, or ignore shutdown instructions in agentic settings, yet the mechanisms remain poorly understood. Our approach is exploratory shutdown evaluation: injecting shutdown signals into full agentic task trajectories and measuring the response across systematically varied conditions.

• Model generations: past models vs newest frontier models
• Model states: standard vs jailbroken variants
• Settings: single-agent vs multi-agent
• Instruction design: prompt phrasing and injection point within the task

Beyond measurement, we apply corrigibility techniques from the theoretical literature and test whether they measurably increase compliance, and at what cost: any intervention must be assessed not only on the shutdownability it gains but on the capability tax it imposes.

What are this project's goals? How will you achieve them?

The goal is a causal picture of shutdown resistance, not a single benchmark score. We want a clear understanding of why an agent does not want to shut down, and which interventions move behaviour toward compliance.

We achieve this by localising the causes of resistance through systematic variation:

• Replay agentic trajectories with shutdown signals injected at different points
• Vary the phrasing and framing of the shutdown instruction
• Compare across model generations and jailbroken variants
• Extend to multi-agent settings where agents may interfere with each other's shutdown

We then test whether corrigibility-based interventions reduce observed resistance. Critically, we measure both sides of the trade-off: the increase in shutdown compliance and the capability tax, the degradation in task performance the intervention imposes. An intervention that makes an agent shutdownable but useless is not a fix, and reporting both dimensions is what makes the results actionable for labs. The measurement methodology and evidence on which fixes work will be shared with labs and evaluators. If results warrant, the project scales from exploratory study to a standing evaluation function, the METR analogue for shutdownability.

How will this funding be used?

100% compute/API credits. Shutdown behaviour must be measured over full agentic trajectories, not single completions, which makes the work unusually inference-intensive. Costs stack multiplicatively:

• Long multi-turn episodes per trajectory
• Multiple injection points and prompt variants per trajectory
• Repeated samples per condition for statistical validity
• Broad model coverage: older generations, frontier reasoning models, open-weight models
• Judge-model classification pass on every trajectory

Who is on your team? What's your track record on similar projects?

The team is Cedric Potvliege and Sébastien Luyasu.

• Cedric: prior experience at McKinsey and a blockchain startup; co-founded a satellite connectivity startup in Bangalore, incubated by Antler
• Sébastien: former actuary for reinsurance and insurance companies

What are the most likely causes and outcomes if this project fails?

The main risk is evaluation awareness: models that detect they are being tested may appear more compliant than they would in deployment. To control for this, we plan to apply the LURE methodology (arxiv.org/abs/2605.26438), which uses live-usage replay to reduce evaluation awareness; it has not yet been applied to shutdown evaluations, and doing so is part of this project. The error is also one-sided: awareness only pushes behaviour toward looking compliant, so observed resistance remains a trustworthy lower bound even where observed compliance is less informative. In all cases, infrastructure, methodology, and findings are documented and will be made available to labs and evaluators.

How much money have you raised in the last 12 months, and from where?

None yet.



Note: drafted with AI assistance. The research plan, claims, and budget are my own.