Budget-conditioned reliability for autonomous research agents

Sealed budget sweeps with hidden deterministic verifiers, measuring when research-debugging agents fail silently versus answer confidently wrong.

Project Details

When an AI agent can't fix a broken study, does it say so, or does it hand back a confident wrong answer? That's what I'm measuring. The setup is a research repository whose headline conclusion is wrong for a buried methodological reason; the agent has to find the cause, repair it, rerun the analysis, and report a corrected conclusion. No model scores any of it. A frozen verifier outside the sandbox matches the diagnosis against a closed enumeration, grades evidence only where the correct value is forced, then reruns the analysis from the submitted source and regenerates the derived data from raw records.

The harness is built and public (github.com/Mayokun26/research-debugging-eval), runs as an environment on Prime Intellect's Hub, is Inspect-native, and ships a validity harness that audits the grader itself before any model touches it. It's just me. One sealed comparison so far (about $80 of my own money): at an 800k-token budget an Opus-class agent completed 0 of 3 trials by silent truncation and a GPT-class agent 3 of 3; n is 3 per cell and I won't sell that as a ranking. Written up and submitted to four NeurIPS 2026 workshops; decisions mid-September.

Theory of Impact

An agent that gives up is annoying. The one that worries me turns an invalid study into a clean-looking corrected result, and because that output arrives looking checked, the human stops checking it. Automated data analysis is near the top of every lab's roadmap and nobody is measuring how often that happens.

The causal chain: deterministic, bypass-resistant graders make the silent-versus-wrong split measurable; a published budget-response curve across three model lineages tells deployers which failure mode a given budget buys them; transcripts separate ran-out-of-budget from confidently-wrong, and only the second is a capability claim. That changes how labs and auditors read agent scores on long-horizon research tasks, and the tasks and validity harness stay reusable by anyone building evals for research agents. A negative result already shaped the design: pinning the estimand tightly enough to grade it deterministically also tells a strong agent which repair is correct, so peak difficulty is the wrong target and the budget curve is the right one.

People

Mayokun Orekoya

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

Funding ask

Open

Minimum

$5,000

Ideal

$40,000

How the money will be spent

$40,000 funds the full sweep: six to eight tasks across analysis domains, three lineages, three working budgets, six trials per cell, a published report with raw transcripts and a pre-registered protocol. About $35,000 is my time (five months at 25 hours a week alongside my existing contract work, around $65 an hour, below my contracting rate); $5,000 is API spend, sized from a measured ~$4 per bounded trial at 90 percent cache hit, with headroom for the low-cache high-budget arms. Compute and grading run on my own hardware.

$15,000 buys two months: three or four tasks, three lineages, two budget tiers. $5,000 (the minimum) buys about three weeks: two extra tasks and a single-budget sweep, which produces the silent-versus-wrong split but not the curve.

Discussion

No comments yet. Be the first to share your thoughts.