pale-ale: Structural Trace Triage for Human Oversight

Kagoshima, Japan

amber-oversight.vercel.app

An open-source research program testing whether structural signals of relation-preservation failures can prioritize fixed-budget human review of long agent traces before outcome scoring.

Project Details

When language models generate long agent trajectories, individual steps often look reasonable. The transition from a source document to a plan, a tool execution, and a final answer can appear locally plausible, yet the process remains susceptible to failing to preserve the relational constraints required for global consistency. Led by researcher Aoi Kawasaki, the pale-ale open-source initiative develops pre-scalar structural oversight methods to evaluate these potential relation-preservation defects before a trajectory is compressed into an aggregated outcome score that does not retain intermediate process history.

While we have published four technical reports and built a prototype called Amber, the system functions as a technical demonstration rather than empirical proof of utility. This grant funds a prospective evaluation cycle to bridge that gap. Our first candidate signal failed its development gate (yielding no recall improvement and increasing the review burden on clean paths) and was retired.

To ensure the new evaluation is not a post-hoc replacement, we will freeze our new candidate contract and its associated carry-forward gates before exposing the signal to a fresh development surface containing previously unused labels and a newly partitioned held-out set locked prior to evaluation. We will test this candidate on locked splits against cheap baselines incorporating trace length and source domain, measuring both the human-labeled defect discovery rate and the clean trajectory burden. If the candidate passes, we will implement local triage in Amber. If it fails, we will retire it, publishing a negative closeout and the reproducible evaluation harness. We evaluate whether this candidate warrants broader empirical validation.

Theory of Impact

The rapid proliferation of highly capable open-weight models is driving a shift toward private, on-premises, or self-hosted deployments. In this decentralized environment, centralized safeguards and telemetry can no longer be assumed as a universal default, creating a setting where autonomous agents can execute trajectory volumes offline that quickly exceed manual review capacity. Within these environments, human oversight is bottlenecked because raw execution traces outgrow review budgets, and collapsing these rich processes into scalar outcome scores does not retain the intermediate preservation history of declared relations between typed artifacts.

The pale-ale program addresses this by establishing an additional local inspection surface before the trace is compressed. Rather than claiming to detect deception or automate human judgment, we evaluate whether a frozen pre-scalar triage signal can improve the allocation of fixed human review attention to trajectories with defined relation-preservation defects. For long-term scalable oversight, process-level inspection offers a motivating hypothesis: while advanced agents might eventually learn to spoof final outcome scores, coordinating global relational consistency across multiple intermediate steps has the potential to impose additional structural constraints on the agent.

People

AOI KAWASAKI

Team Member

Funding Details

Start Date: Dec 30, 2025
End Date: -
Expected Duration: Ongoing research program; current validation phase: 3–6 months
Funding Raised to Date: -
Annual Budget: -
Monthly Burn Rate: -
Current Runway: -
Funding Goal: -
Funding Stage: -
Fiscal Sponsor: -

Track Record

Public implementation and research record:

Built and maintained the public pale-ale research repository:
https://github.com/Udonburo/pale-ale
Published the Gate12A frozen technical report:
https://doi.org/10.5281/zenodo.19483162
Published the Transport-first telemetry note:
https://doi.org/10.5281/zenodo.19569052
Published the Gate12B observer-relative closure-signature report:
https://doi.org/10.5281/zenodo.20080003
Published the Gate12C-1 compression-interleaved parenthesization-defect report:
https://doi.org/10.5281/zenodo.21355572

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$24,000

Ideal

$48,000

How the money will be spent

This grant requests a USD 24,000 minimum to complete a bounded operational evaluation, with an ideal extension to USD 48,000 for robustness validation.

USD 24,000 Minimum (3-month prospective evaluation cycle):

This funds pipeline freezing, one candidate evaluation on a fresh label surface, one independent QA pass, and a pre-specified decision path: integration of the candidate into Amber only if both the development and held-out gates pass, or a negative closeout if either gate fails.

Researcher Time (USD 16,500): 3 months at USD 5,500/month for pipeline freezing and baseline integration.

Trajectory Generation & Compute (USD 3,000): Generating fresh trajectories using local open-weight models and processing trajectory logs.

Label QA (USD 2,500): 50 hours of independent review at USD 50/hour to verify reference-grounded trajectory labels.

Independent Reproducibility Audit (USD 1,500): External contractor compensation for independent execution and verification of the evaluation harness.

Hosting & Infrastructure (USD 500): Amber deployment, artifact storage, and release infrastructure.

USD 48,000 Ideal (+USD 24,000 incremental for 3 additional months):

This deepens validation by adding a second reviewer, a second trajectory schema, pre-specified robustness analysis, and external reproduction.

Researcher Time (USD 16,500): Additional 3 months at USD 5,500/month.

Second Reviewer Cross-Validation (USD 4,000): 80 hours at USD 50/hour.

Cross-Schema Preparation & Compute (USD 2,500): Trajectory generation, normalization, schema mapping, and inference for a second evaluation surface.

External Reproduction & Archival Release (USD 1,000): External contractor compensation for reproducing the cross-schema findings and verifying the deterministic release package.

To enforce discipline, the development gate requires the candidate to improve triage recall relative to cheap baselines while satisfying a pre-specified clean-trajectory burden guardrail. The held-out set will remain closed unless that gate passes. If either gate fails, the candidate will be retired; any rescoping or reallocation of unused funds will require explicit funder approval.

Discussion

No comments yet. Be the first to share your thoughts.