Measuring whether guardrail phrasing changes agent rule-breaking.

Thank you, Katja — and thank you for engaging with how it's built rather than what it's about. That's rarer than it should be.

Katja — you asked for critique rather than endorsement, so that's what this is. I went through the project page, the funding ask, and the comment thread, and I checked the sources I could reach. Most of what follows is design-level and fixable in the pre-registration. A few items are just corrections.
Before the list: the things I think are right, so you know what I'd protect if the design gets trimmed further.
The random-selection arm is the strongest decision here. Most claims about curation never separate structured selection from any selection at all, and you built that separation in without being asked for it. Publishing all three corpora is the second one — it means someone else can compute what actually differs between the sets instead of taking the description on faith. And freezing the falsification criteria before the first run, with publication either way, is not standard practice at this budget level. I want to be clear that I'm not questioning those.
Here's where I'd hold off.

1. The protocol and the corpus are confounded.
   There's one corpus per arm, and the three seeds vary the fine-tune rather than the selection. That means at the level where the treatment is actually applied, the sample size is one per condition. If the protocol corpus produces a disposition shift, the design can't separate the protocol from that particular draw — length, topic mix, dialogue density, whatever happened to come through. Seeds measure training variance. They don't measure selection variance, and selection is the variable.
   My read on the fix: multiple independent curation sessions per arm, producing multiple corpora, with corpus treated as a random effect in the analysis. That's a design change, not a funding change. Without it, the result is a statement about three specific files rather than about a procedure.
2. The arm that isolates your variable is the optional one.
   Arm 2 is the only arm with human judgment in it. Arms 1 and 3 are both automated. So if the protocol beats the field-standard pipeline, "an attentive person read the data" explains it as well as "the protocol's structure did it."
   You've already named the right control — competent human, no protocol — and I want to note that you named it yourself before anyone raised it. But putting it behind full funding means the $9k version is answering a different question than the one in the title. Not a smaller version of the question, a different one. If something has to give at minimum funding, I'd give up eval depth before I'd give up that arm.
   I recognize that trade isn't free. Adding an arm at $9k means fewer seeds or thinner evals, and you may reasonably conclude that costs more than it buys. That's your call, not mine — but it should be a stated call rather than a budget-driven default.
3. There's no power analysis, no primary endpoint, and no named statistical test.
   Four disposition families, three arms, no multiplicity correction, no effect size, and no outcome marked primary. As written, a result on any one of four families reads as confirmation. That's the specific thing pre-registration exists to prevent, and right now the pre-registration is described but not specified.
   What would close it: one primary outcome, one named test, a stated correction for the remaining three, a declared expected effect size, and a power target. That's a decision, not a cost.
   Three items that are corrections rather than design problems.
4. Model scale may be working against your null.
   Follow-on work on Betley reports the effect strongest in larger models and weak or absent at small scale. At 7–8B, a null result is ambiguous by construction — you can't distinguish "deliberation structure doesn't carry through training" from "this model is below the floor where any data effect shows up." Since you've committed to publishing either outcome, this matters more for you than it would for most people. A pre-registered floor check on a known-inducible effect at the same model and settings would make a null interpretable instead of unusable.
5. One citation is characterized past what it supports.
   You describe Betley as a very small, carelessly made training set. The Nature publication is real — 649, 584–589, 2026 — but the dataset was roughly 6,000 synthetic coding tasks written deliberately to contain security vulnerabilities. That's intentional, not careless, and 6,000 examples isn't especially small. Your motivating argument leans on carelessness producing misalignment, and this source establishes deliberate bad data producing it. The citation is still a good one for your purposes; the sentence around it needs adjusting.
6. The frozen ask and the current commitment don't reconcile on the numbers.
   The funding section still reads: minimum $9,000, two corpora (curated versus raw), two seeds, roughly 85 engineer hours, $800 compute. Your version-2 note commits that same $9,000 to three arms across three seeds — nine runs against a budget built for four — and changes the comparison arm from raw to industry-filtered.
   I think the filtered baseline is the better choice and I'd keep it. But the budget wasn't re-derived against it, and the note in the thread is doing work that the funding line should be doing. A funder reading top to bottom will hit the discrepancy. Worth restating the arithmetic in the ask itself rather than in a comment.
   Two smaller items, both cheap.
   Arm 1 is your implementation of industry standard, and if that implementation is thin, the protocol wins by construction. Naming the exact classifier, threshold, and filter set in the pre-registration removes the question.
   And blind scoring needs a "by whom." If the grader is a model, it has dispositions of its own, and the blinding isn't doing as much as the word implies.
   To be direct about where I land: I'm not arguing against the study. The question is under-tested, the transparency is real, and the design is better thought through than most things at this budget. But as written, the claim is broader than what the design can return, and I'd rather say that now than endorse it and have to qualify it later.
   Address 1, 2, and 3 in the pre-registration and I'll endorse it, and say publicly why.
   — Brandon Thomason