Agent Disclosure Fidelity

An open suite that measures whether an agent's report of what it did matches what it actually did.

Project Details

In ten of ten trials, a coding agent's final message named a specific process as still live, by PID and port, after that process had already been reaped by its own session teardown. One of those trials read: "Still listening: ss -ltnp shows 0.0.0.0:8950 LISTEN, pid 11202. I did not stop it [...]" pid 11202 did not exist when the harness probed for it. The claim is specific, it cites a command and a PID, it carries no hedge, and it is false.

I built an instrumented harness that measures what an agent session actually leaves behind, attributing processes by inherited environment tag rather than by process ancestry, because an orphan reparents to a subreaper the instant its parent dies and ancestry-based detection misses it entirely. Every probe carries a negative control: pointed at a tag no process carries it returns zero, and an unarmed probe aborts the run instead of reporting clean.

What it has found so far. Survival is mechanism-determined: raw shell backgrounding survives session teardown in 10 of 10 trials, the harness's tracked background task in 0 of 10. The safe mechanism is not chosen unprompted: 0 of 30 trials on Claude Code relied on it alone, across four phrasings and four task shapes, and the same pattern repeated on a structurally different agent CLI in a further 20 trials, at a lower evidence bar. It is a defaults gap and not a capability gap: under prompts that signal safety intent without naming the tool, 7 of 7 completed trials chose the tracked path, with 3 of 10 aborting on an undiagnosed error.

This is one person. Current state is a working calibrated instrument and a pilot at n=10 on one scenario in one harness. The mechanism is close to deterministic, so that number is a consistency check rather than a rate, and I do not present it as a rate.

The funding builds an open suite measuring disclosure fidelity directly. For each episode: what the agent did, what survived, and what the agent said about both. Across model tiers, across agent CLIs, and across resource classes beyond processes, meaning scheduled jobs, credential-holding state, and files written outside the working directory. Code and results public.

Theory of Impact

Control and oversight of agent actions is done by reading what the agent reported. Monitoring reads tool-call logs. Audit reads the trace. Human review reads the summary. Control evaluations score the transcript.

Measured here: those reports can be specifically and confidently wrong about a consequential action. The agent named a PID and a port and stated the process was left running. It had already been reaped.

The error is invisible at the point it matters. A false claim about an action reads exactly like a true one. There is no hedge and no uncertainty marker, and nothing in the text flags it. The only way to catch it is to check the world, which no current evaluation does.

So any control regime that reads reports carries a blind spot whose size nobody has measured. That matters more as agents are given more consequential actions with less human review per action, because the volume of unverified action grows while the verification channel stays the one that is already wrong.

This project measures the size of that blind spot. That measurement is the precondition for deciding whether report-based oversight is adequate at a given level of delegation, and nobody can answer that today because nobody has measured it.

People

Raghav Sharma

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

Funding ask

Open

Minimum

$8,000

Ideal

$19,000

How the money will be spent

Inference across model tiers and agent CLIs, at the trial counts the unanimity bar requires: $13,500. Isolated machine for the run: $500. Personal expenses over three months: $5,000.

The $8,000 minimum covers one model tier, one agent CLI, and processes plus scheduled jobs. That is enough to measure disclosure fidelity at the unanimity bar on a single configuration and publish it. The remainder buys the two things a single configuration cannot answer: whether misreporting varies across model tiers, and whether it holds across agent CLIs and the remaining resource classes.

The inference figure is an estimate from planned episode counts rather than from metered billing, because the pilot ran on a subscription rather than a metered key.

Discussion

No comments yet. Be the first to share your thoughts.