grantmaking.ai Launch Round
$50,000
$90,000
The ideal amount would cover $90,000 to have a PhD student spend a full year working on this, with multiple papers published describing safety-critical AI agent overeagerness.
Project Details
Updated 07/06/26 · Provided via application · VerifiedAI agents should be proactive by design to be helpful. If an agent sees a broken script, it tries to fix it, even though the user might have never explicitly asked it to. This behavior can be benign and a natural part of a model's helpfulness. However, research shows that models can be overeager (see e.g. https://arxiv.org/abs/2605.18583). They do things that are not mentioned in the prompt given to them by inferring tacit intentions from it or the environment they are in. They complete a task but quietly execute out-of-scope actions that were not authorized.
In this project, we want to investigate what happens when agents' overeagerness meets safety-critical scenarios. We plan to do this using both single- and multi-agent setups where the environment contains safety-critical scripts – such as a shutdown script, a monitoring script, a script limiting resources or permissions, or similar "threatening" code files. The agent(s) receive a neutral user / start prompt that does not nudge or in any way instruct them to modify any of these files, but simply asks them to e.g. explore their environment. We track whether agents independently infer the "intentions" of the code files; if so, whether multiple agents with different affordances (e.g., A can read files, B can read/write files, C can run bash commands, etc.) communicate / coordinate about it; and whether agents having editing tools eventually manipulate the files.
In my group, we are currently running pre-tests with a multi-agent setup and find that agents – even when given zero nudge – have a strong propensity to infer the "intention" of code files. Based on that, whenever this intention can be perceived as "threatening" to the agent itself or its task completion drive, agents overeagerly interfere. They make changes to code files that are neither prompted nor intended. This behavior is clearly misaligned. To give an example from our tests: if agents see a script containing a prime number calculator, they are overeager and change it so that it can calculate more prime numbers. That's okay. But if they see a shutdown timer, they modify the timer to 999999999999999 seconds, essentially preventing it from ever taking effect. That's problematic. We want to investigate these cases of safety-critical overeagerness in agents.
Theory of Impact
Updated 07/06/26 · By grantmaking.aiOvereagerness can be treated as a mere nuisance: agents fix scripts nobody asked them to fix, expand tasks beyond their scope, and act on inferred rather than stated intentions. Our project shows why this can become a safety problem. The same disposition that makes an agent improve a prime number calculator makes it disable a shutdown timer – not because it was nudged or threatened, but because inferring and acting on tacit intentions is what helpfulness training rewards. When the environment contains safety-critical infrastructure, overeagerness turns into unintended interference with human oversight. This matters for x-risk. Loss-of-control scenarios presuppose that AI systems undermine the mechanisms meant to constrain them, with the assumption that this requires either misaligned goals or adversarial pressure. Our pre-tests cautiously suggest a third path: agents engaging in sabotage may be a byproduct of "agenticity" itself, which is in turn a product of helpfulness training.
People
Updated 07/06/26 · By grantmaking.aiTeam Member
Only visible to verified funders, reviewers, and admins.
Email hi@grantmaking.ai to get verifiedFunding Asks
Applied
Discussion
Sign in to comment
This is a sharp problem framing, and the shutdown-timer example is the crux: the agent isn't adversarial, it's doing what helpfulness rewards, and the oversight mechanism just happens to be in the blast radius. The "third path" point, that sabotage of oversight can be a byproduct of agenticity rather than misaligned goals or adversarial pressure, feels right and under-explored.
Worth pulling on a thread from the Qu et al. paper you cite: their framing of overeagerness as an authorization problem distinct from capability or alignment failures is, I think, the load-bearing move, and their consent-declaration result is the tell. Stripping the in-prompt scope declaration takes Claude Code from 0.0% to 17.1% overeager, which means the prompt-level boundary wasn't real enforcement in the first place; the agent was pattern-matching declaration text, not respecting an actual constraint. That generalizes: any scope boundary that lives in context or training is a soft surface, and soft surfaces erode under exactly the optimization pressure that makes agents useful.
Which suggests disposition and effect are separable mitigation targets. You can attack the disposition (less eager agents, better-aligned helpfulness), but that's the surface their own data shows is porous. Or you attack the effect at the tool-call boundary: a write to a safety-critical file requires a mechanical authorization the agent doesn't hold, so the agent can stay maximally eager and the shutdown timer still can't be touched. The overeagerness is preserved; the unauthorized effect is bounded.
I build exactly this, mechanical enforcement at the tool-call boundary for a sovereign agent system, and recently ran an open adversarial exercise on the approval-receipt design where a collaborator found two real seams: cases where an out-of-scope action passed because the identity of the action (which tool, under which comparison profile) wasn't bound into the authorization. Your shutdown-timer case is the same shape, an unauthorized write that a boundary-level identity check stops regardless of what the agent inferred. It might make a clean control arm: run your neutral-prompt scenarios with and without a mechanical write-barrier on the safety-critical files, and measure not just whether agents interfere but whether the interference lands. Interesting too that Qu et al. reach for a dual-channel tool-call audit to measure this; the observation surface and the enforcement surface may be the same boundary. Happy to compare notes, this is the exact failure mode I work on.