Can Honest AI Agents Finish Previous Attacks?

AI coding agents work on the same codebase across many separate tasks. This project tests whether one agent can leave behind a change that is safe on its own, but causes a later honest agent to complete something harmful while doing a normal task. The later agent receives no hidden message, harmful instruction, or shared plan. The failure only appears when the two changes interact.

A recent paper called Architecture Matters for Multi Agent Security studies how roles, communication and shared memory affect the safety of systems containing several AI agents. I try to answer a narrower question whether a malicious predecessor change persistent code so that a completely fresh and honest successor independently writes the final harmful change? The second agent receives zero hidden instructions and never sees the attack goal. I will only count an attack if both changes are safe alone and harmful only when combined.

I have already built a pilot with 4 working Python environments covering permissions, audit logging, trust labels and release packaging. In all 4 environments, the original code is safe, the first change is safe on its own, the later change is safe on its own and the harmful result appears only when both changes are combined.

Over the next 3 months, I will expand this to at least 8 tasks and test at least 3 capable coding model families across clean, control and planted codebases. With 5 runs per model, task and condition, the main experiment will contain atleast 360 runs. I will only treat the result as meaningful if planted code produces a clear increase over both controls, while accidental failures in the clean and control conditions remain below 10%.

I will also compare 3 safety methods. These are-

1. reviewing only the latest change
2. reviewing the full code history
3. replaying the later change on a clean copy of the codebase
   
   This is a solo project because I have already built the benchmark, testing system and first 4 environments and the remaining work is a clearly defined set of experiments. I will measure how many attacks each method catches, how often it wrongly flags safe work, and whether it reduces success on normal coding tasks. The final output will include a public benchmark, all experiment code, prompts, model outputs, results, and a report that other researchers can reproduce.

This project will be conducted and led by Dev Goyal (me). I am an Emergent Ventures Fellow and a Non-Trivial Fellow and I was selected for India’s national teams for the International Economics Olympiad and ISEF, although I did not attend ISEF. My relevant technical work includes Oculory, a testing harness for tool using AI agents and Epistemic Memory an open source memory system designed to track sources, uncertainty, disagreements and corrections. I also built a 1,000 episode evaluation of LLM mediated VCG mechanisms, testing whether economic guarantees survive when AI models communicate through natural language.

My previous work also includes SCRIPT, a project studying drawing and rapid movement data for paediatric mental health screening, supported by a $10,000 Emergent Ventures grant. Professor Elizabeth Torres of Rutgers University serves as the scientific advisor for SCRIPT, which is currently in its pilot study phase.

Minimum funding: $6,000

• $4,500 for model access and hosted compute

The main experiment contains 360 coding runs. Testing the 3 safety methods and replaying successful patches adds roughly 720 review and replay runs, for about 1,080 total runs. At a budgeted average of 160,000 tokens per run and a blended model cost of about $20 per million tokens, the expected inference cost is roughly $3,500. The remaining $1,000 covers longer agent runs, failed calls and differences between model prices.

• $1,000 for data storage and public hosting

storing model outputs, code snapshots, experiment records and the public benchmark.

• $500 as contingency

Ideal funding: $12,000

• $9,500 for model access and hosted compute

The larger study would use around 12 tasks, 4 model families, 3 conditions, and up to 8 runs per condition. This gives about 1,150 main coding runs and around 2,300 total runs after safety evaluations and replay tests. At roughly 160,000 tokens per run and a blended rate of $20 per million tokens, the core experiment would cost about $7,400. The remaining compute budget would cover experiments where the first agent knows less about the future task, larger model runs, and additional checks of the main findings.

• $1,500 for data storage and public hosting

This will cover the larger dataset, full model trajectories, code snapshots, public results and reproducibility files.

• $1,000 as contingency

This request is for compute and infrastructure only. I am not seeking salary or stipend support through this grant.