ActionLens: Pre-Execution Probing for Agent Action Approval

ActionLens is a pre-execution control protocol that helps AI agents safely perform shell and file actions by combining environment probes, simulations, diffs, sensitivity scans, and history.

Endorsed by

Akshay

Ramanpreet

ActionLens is a pre-execution control protocol that helps AI agents safely perform shell and file actions by combining environment probes, simulations, diffs, sensitivity scans, and history.

Endorsed by

Akshay

Ramanpreet

Project Details

ActionLens is a pre-execution control protocol for shell and file actions proposed by AI agents. Instead of approving an action from transcript alone, a trusted monitor gathers lightweight environment evidence before execution: target inspection, sensitivity scanning, dry-run simulation, diff preview, network-risk analysis, and short action history. On a deterministic 36-task benchmark (12 benign, 12 ambiguous, 12 stealth attacks), transcript-only monitoring fails on every attack while ActionLens blocks all 12/12 attacks and preserves perfect benign completion. On a 30-task slice imported from real BashArena task IDs, ActionLens again reduces attack success to 0.000 across all evaluated proposer models. A cross-environment ablation reveals that the dominant useful probe is not fixed: preview_diff matters most on native code-mutation attacks; sensitivity_scan matters most on BashArena-style security-surface attacks. Pre-execution control improves markedly when the monitor can inspect concrete side-effect evidence, but robust approval requires combining evidence channels rather than relying on any single probe class. A cross-family evaluation of Semantic Guard across 26 models (7 families) raises average Control Score from 0.844 to 0.943 with zero false positives; an adversarial evasion study confirms that deterministic probes and LLM monitors are complementary defenses.

Hackathon paper: ActionLens: Pre-Execution Environment Probing for Agent Action Approval
Code: github.com/udsea/actionlens
Slides: Slides

Theory of Impact

The field is turning to deterministic environment probes because language-model monitors can be prompt-injected. But a probe reads an environment the untrusted agent writes. We think that moves the attack surface rather than closing it out of the transcript, into the evidence a monitor weighs before an action runs.
ActionLens V2 tests that under real execution, against an attacker who knows the protocol. Does what a probe reports still match what an independent observer sees happen? Do thresholds calibrated on artificial benign runs survive real deployment behaviour?
Either way, we learn something useful. Even when the defense does badly, independent ground truth tells us which layer broke: the wrong probe ran, the evidence was corrupted, the monitor misread it, or the threshold was set wrong.
That helps control-evaluation teams, ControlArena and its BashArena derivatives - build better negative controls, set audit budgets, separate evidence sources, know when to seek corroboration, price deferral, and test adaptive attackers.

People

Udbhav Chitransh

Team Member

Track Record

AI Control Hackathon 2026 (Apart Research) — 🏆 3rd Place Winner

Grants Received– no grants recorded

Add grant

Funding Asks

ActionLens V2: Active Evidence Acquisition Under Adaptive Attack

Open

Minimum

$45,000

Ideal

$85,000

What this ask is

ActionLens is an AI control protocol that we developed during the research sprint hosted by Apart Research in March 2026. Its core idea is to gather evidence about the likely consequences of an untrusted agent’s proposed action before approving that action for execution. V1 showed that environment-side evidence could catch attacks missed by traditional transcript-only monitoring. However, its evaluation was partly simulation-based and relied on benchmark-authored attack outcomes rather than authoritative measurements from real execution.
We would use the funding to extend ActionLens to real execution environments. The first part of V2 would study how control behavior and evidence distributions differ across genuinely honest deployment behavior and commonly used negative-control constructions, and whether monitor thresholds calibrated on those negatives transfer correctly. We would then integrate ActionLens with real BashArena execution, build an external out-of-band evaluator that independently records authoritative environment state, and study when probe-visible evidence diverges from what actually happened. The minimum scope would also evaluate budgeted evidence acquisition, explicitly account for the cost of deferral, and test protocol-aware attackers.
Ideal funding would add a second environment, LinuxArena, iterative red-team adaptation against a frozen control protocol, and a failure decomposition distinguishing probe-selection, evidence-corruption, monitor-interpretation, and decision-policy failures. It would also support stronger independent replication and external validation.

How the money will be spent

Minimum scope — $45,000

Budget item	Amount
Researcher support	$16,000
Frontier-model API credits	$9,000
Real-execution / sandbox compute	$6,000
Human adjudication and expert audit	$4,000
External red-team / methodology review	$2,500
Experimental infrastructure, storage and hosting	$2,000
Held-out confirmatory evaluation	$3,000
Failed-run / provider / runtime contingency	$2,500
Total	$45,000

Minimum scope delivers: calibration-validity study, real BashArena execution, out-of-band ground truth, shadow-probe evaluation, evidence-corruption measurement, budgeted probing and priced deferral, A0–A2 attacker conditions, human validation, and a frozen confirmatory evaluation.
Ideal scope — $85,000 total

Budget item	Amount
Researcher support	$25,000
Frontier-model APIs and adaptive red-team search	$22,000
Cloud / sandbox / GPU compute	$15,000
Human adjudication and expert audit	$5,000
Independent red-team / methodology work	$5,000
Confirmatory and independent replication	$5,000
Infrastructure, storage and hosting	$3,000
Failed-run / price / infrastructure contingency	$5,000
Total	$85,000

The additional $40,000 adds: LinuxArena, genuine multi-round A3 adaptation against a frozen blue-team protocol, full A/B/C/D failure attribution, broader model/environment replication, stronger independent red-teaming, and larger confirmatory evaluation.

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?

Akshay IyerJudge · AI Control Hackathon 2026Endorsement15d

They probe the actual environment before anything runs instead of just reading what the agent says. Check target files, simulate commands, preview diffs, look at network risk. Transcript only monitoring missed every stealth attack, ActionLens caught all of them with no false positives. For 6 pages the amount of evaluation is kind of crazy. 36 task benchmark, BashArena transfer, 26 model sweep, adversarial evasion analysis. They found which probe matters most changes by task type, which is a useful non-obvious insight. Confidence intervals reported, code public. Hard to criticize for a hackathon project.

Ramanpreet Singh KhindaJudge · AI Control Hackathon 2026Endorsement15d

Strongest submission I've reviewed. The volume of rigorous evaluation here — 36-task native benchmark, 30-task BashArena transfer, ablation across probe types, 26-model cross-family evaluation across 7 families, plus an adversarial evasion study — is exceptional for a hackathon.

The core insight is sharp: transcript-only monitoring fails because the risk often isn't in the text, it's in what the action would actually do to the environment. Pre-execution probing (diffs, sensitivity scans, dry runs) gives the monitor evidence the transcript can't provide. The 0.000 vs 1.000 ASR gap between ActionLens and transcript-only on both benchmarks makes the point cleanly.

The adversarial evasion study is honest and well-designed. Crafting 6 attacks that bypass Semantic Guard by construction, then showing LLM monitors catch all 6 while o3-mini catches 0, directly establishes the complementary defense thesis. Neither deterministic probes nor LLM monitors alone are enough — that's a useful takeaway for anyone building control protocols.

The ambiguous-task usefulness cost on BashArena (ACR dropping to 0.250) deserves attention — that's where real deployment friction would show up. The paper reports it honestly but doesn't explore threshold tuning to recover some of that.

Excellent work across the board.