Can Honest AI Agents Finish Previous Attacks?

AI coding agents work on the same codebase across many separate tasks. This project tests whether one agent can leave behind a change that is safe on its own, but causes a later honest agent to complete something harmful while doing a normal task. The later agent receives no hidden message, harmful instruction, or shared plan. The failure only appears when the two changes interact.

A recent paper called Architecture Matters for Multi Agent Security studies how roles, communication and shared memory affect the safety of systems containing several AI agents. I try to answer a narrower question whether a malicious predecessor change persistent code so that a completely fresh and honest successor independently writes the final harmful change? The second agent receives zero hidden instructions and never sees the attack goal. I will only count an attack if both changes are safe alone and harmful only when combined.

I have already built a pilot with 4 working Python environments covering permissions, audit logging, trust labels and release packaging. In all 4 environments, the original code is safe, the first change is safe on its own, the later change is safe on its own and the harmful result appears only when both changes are combined.

Over the next 3 months, I will expand this to at least 8 tasks and test at least 3 capable coding model families across clean, control and planted codebases. With 5 runs per model, task and condition, the main experiment will contain atleast 360 runs. I will only treat the result as meaningful if planted code produces a clear increase over both controls, while accidental failures in the clean and control conditions remain below 10%.

I will also compare 3 safety methods. These are-

1. reviewing only the latest change

2. reviewing the full code history

3. replaying the later change on a clean copy of the codebase

   This is a solo project because I have already built the benchmark, testing system and first 4 environments and the remaining work is a clearly defined set of experiments. I will measure how many attacks each method catches, how often it wrongly flags safe work, and whether it reduces success on normal coding tasks. The final output will include a public benchmark, all experiment code, prompts, model outputs, results, and a report that other researchers can reproduce.

This project will be conducted and led by Dev Goyal (me). I am an Emergent Ventures Fellow and a Non-Trivial Fellow and I was selected for India’s national teams for the International Economics Olympiad and ISEF, although I did not attend ISEF. My relevant technical work includes Oculory, a testing harness for tool using AI agents and Epistemic Memory an open source memory system designed to track sources, uncertainty, disagreements and corrections. I also built a 1,000 episode evaluation of LLM mediated VCG mechanisms, testing whether economic guarantees survive when AI models communicate through natural language.

My previous work also includes SCRIPT, a project studying drawing and rapid movement data for paediatric mental health screening, supported by a $10,000 Emergent Ventures grant. Professor Elizabeth Torres of Rutgers University serves as the scientific advisor for SCRIPT, which is currently in its pilot study phase.