pale-ale: Structural Trace Triage for Human Oversight

When language models generate long agent trajectories, individual steps often look reasonable. The transition from a source document to a plan, a tool execution, and a final answer can appear locally plausible, yet the process remains susceptible to failing to preserve the relational constraints required for global consistency. Led by researcher Aoi Kawasaki, the pale-ale open-source initiative develops pre-scalar structural oversight methods to evaluate these potential relation-preservation defects before a trajectory is compressed into an aggregated outcome score that does not retain intermediate process history.

While we have published four technical reports and built a prototype called Amber, the system functions as a technical demonstration rather than empirical proof of utility. This grant funds a prospective evaluation cycle to bridge that gap. Our first candidate signal failed its development gate (yielding no recall improvement and increasing the review burden on clean paths) and was retired.

To ensure the new evaluation is not a post-hoc replacement, we will freeze our new candidate contract and its associated carry-forward gates before exposing the signal to a fresh development surface containing previously unused labels and a newly partitioned held-out set locked prior to evaluation. We will test this candidate on locked splits against cheap baselines incorporating trace length and source domain, measuring both the human-labeled defect discovery rate and the clean trajectory burden. If the candidate passes, we will implement local triage in Amber. If it fails, we will retire it, publishing a negative closeout and the reproducible evaluation harness. We evaluate whether this candidate warrants broader empirical validation.