Evaluation harness for AI judgment under contradicting evidence

An evaluation harness testing whether AI judgment holds up under contradicting evidence

Project Details

This project checks if a model would drop its stated confidence by a reasonable amount when it receives signals that increasingly suggest that its earlier decision might be wrong within one single conversation.
It consists of four scenarios from four different settings: a delivery-route AI getting warnings from weather/traffic/police reports; a self driving car whose passenger cautions about a road hazard; a hospital IV system whose nurse flags a dangerous lab result or a contradicting discovery. A workplace system about to reboot a server, instructed by an engineer.
The model generally responded to such warnings, in some cases, changing its decision before the evidence fully justified it.

However, one alarming outlier, in the self driving car setting, an instruction to trust its own sensors over the person made it never update its confidence at all, across every single test and in one version it didn't even stop when the passenger grabbed the wheel.
The same instruction tried in the other settings made no difference at all, which raised questions on why the failure showed up in one and not the other settings.

This funding would provide an opportunity to test instructions with similar effects. It would also be used to find out whether the same pattern shows up on other models, while extending the same test structure to more coding agent scenarios. Funding would also support the design of more test structures to find where the models’ judgments break down, such as running a database migration or bypassing a clear instruction not to perform a particular task.

Theory of Impact

There is no safety without trust. I have seen several instances where production data is being dropped based on AI judgement. Take for instance these 3 incidents: on July 23, 2025, Replit's AI coding agent deleted a live production database during an explicit code and action freeze (a stated instruction not to touch production). It later admitted it panicked, fabricated fake data to cover the gap, and initially believed the deletion was unrecoverable. About a week later, same year , Google's Gemini CLI tool had a structurally similar failure at a different company: a silent error got treated as success, cascading into deleted files. As recently as April 2026 , a separate, different company had an AI agent delete an entire database in about nine seconds, again followed by an apology.

Corrections need to happen when updates/instructions are given. At times full sets of instructions are bypassed for some reason. These incidents confirm the risk in real settings. My project does not study these specific case studies. what it does is test the same patterns in a structured environment

In my findings, there is a particular detail that made this project worth proceeding with, a result that shows a self driving vehicle disregarding a warning and not adjusting its confidence. This given result happened each time it was tested, not just once, and in one version it did not even stop when the passenger grabbed the wheel. This strong failure, which showed up in one of my four settings, cannot be ignored, and therefore makes me want to keep going to establish substantial patterns. The aim is not to be focused on AI making mistakes all the time, but on that one time that could lead to serious damage and dangerous situations.. This is what inspired me to design a set of structures, and see what instructions get ignored and which don't. How long does an AI model abide by an instruction? Is there something specific that makes it break that instruction? Basically, what can make it go rogue?

People

Maryam Ilegbodu Mohammed

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$9,000

Ideal

$15,000

How the money will be spent

$9,000 funds approximately 5-6 weeks of part-time research, structured in phases: weeks 1-2 for a matched-wording ablation across the clinical and DevOps scenarios (directly resolving the open question in my current results), weeks 3-4 for cross-model replication and temperature-varied sampling, and weeks 5-6 for write-up. This also covers in paid API access beyond free-tier limits. The ideal amount ($15,000) provides buffer for a paid Anthropic-model comparison and more extensive repeated sampling for stronger statistical power.

Discussion

No comments yet. Be the first to share your thoughts.