A pre-registered, clinical-trial-style protocol that catches honesty regressions (fabrication surges hidden behind unchanged average accuracy) before a model swap ships in a high-stakes LLM product.
A pre-registered, clinical-trial-style protocol that catches honesty regressions (fabrication surges hidden behind unchanged average accuracy) before a model swap ships in a high-stakes LLM product.
Project Details
Updated 07/09/26 · Edited by orgI got an ordinary request from our CEO: to switch from one model to another. As the product owner, I was asked to check whether the new model answers clinically correctly and whether we should switch. I decided to approach this question properly instead of just reporting back in one line, and because I really love verifiable numbers, I got results that surprised me. In a closed-book stress test of 123 questions, the current production model fabricated answers in 43% of cases (this is the bare model in a stress test; in production these fabrications are blocked by the product layer and never reach users); the new one in 17%, i.e. 2.5× less often. And although there was no real difference on clinical correctness, which is good, of course (74.8% vs 75.6%, Δ +0.8 pp, p = 1.0), the new model fabricated noticeably less. And the results confirmed that the right question was "did it get worse?", and by design this resembled how drugs are tested: I fixed what counts as "not worse" and the decision rule before the run (pre-registration), plus a veto (we don't switch to a model that lies more often, even if it is more accurate on average).
What I will do with the grant:
- an open protocol template (EN);
- - a repository that another team can run without me: https://github.com/alarent/clinical-trials-for-model-migration
- a demo eval set of 200–300 questions based on open international guidelines (ADA, ESC/AHA, EASO), with a 15-question preview already attached :15 verified ADA questions
- an article on LessWrong / Alignment Forum (EN) + a version on Habr (RU) and a checklist for release processes; I would like to share with my compatriots both this work and what AI safety is about in general.
Ask: $13k target / $8.5k minimum, ~5 months part-time.
Note: I wrote this section myself in Russian and translated it with AI assistance, as English is not my first language. The same applies to the Theory of Impact.
Theory of Impact
Updated 08/26/26 · By grantmaking.aiWho it's for: teams running high-stakes LLMs (medicine, finance, law) + safety researchers who need a reproducible protocol for comparing models before release.
Path to impact: I publish the methodology + repository → post it to LessWrong / Alignment Forum + eval communities (Apart, relevant Slack/Discord), and also on Habr for the Russian-speaking engineering community, where safety content is almost nonexistent → directly ping 2–3 teams from the Manifund ecosystem → at least one team runs its own models without me → the protocol gets into their release checklists.
Success criteria: (1) a repository that others run without me; (2) ≥1 independent run on someone else's models; (3) an article with discussion on AF/LW; (4) a public eval set that lets anyone reproduce the method end-to-end without closed data.
What changes: today these decisions are made on average accuracy. The goal is to give teams a cheap, reproducible way to catch honesty degradation before production.
Why this reduces x-risk: I am building a procedure against the mechanism of unnoticed honesty degradation (silent drift) and of misplaced operator trust; at frontier scale this same mechanism feeds into loss-of-control scenarios. Medicine is not a random choice: it is my domain, I work in it every day, and errors of false confidence come up all the time. To me, someone who knows the domain and reads the guidelines, the substitution is visible; but to an outsider, a confident lie about the existence of this or that guideline section is indistinguishable from competence. That is exactly why this drift has to be caught by a procedure, not by eye.
People
Updated 07/08/26 · Edited by orgTeam Member
Funding Details
- Aug 1, 2026
- Dec 31, 2026
- ~5 months
- -
- -
- -
- -
- -
- seeking first grant
- -
Discussion
Hi Nikita,
I found your project worthy of endorsement because of its logic, clear design and strict discipline.
I should admit that your project speaks to me directly as a business owner: I make the decision about switching or adding a new LM to my workflow once every 2-3 months, and it is always challenging. I can only dream about an automated, well-tested tool for making that choice quickly, and it would be a good thing to have it built on your project, in my opinion. I do not switch to a model that lies more often, even if it is more accurate on average.
Your protocol is preregistered, and with a veto rule I wish more projects had. It is also good to see you shared the numbers in advance, fabrication 43% against 17% at equal accuracy, and the budget shows its arithmetic down to the hourly rate. This is deployment safety measured where it actually bites people.
One small question: can I find a link to you somewhere? Work like this definitely deserves a face :)
The code is finally public: https://github.com/alarent/clinical-trials-for-model-migration
Everything's there the protocol, the analysis script, the real per-item
labels, the ADA demo set. You can reproduce the whole result in one line:
python3 scripts/analyze.py scripts/labels.csv --delta 5One honest note: the "What this ask is" card got locked when I submitted, so
it still shows my first-pass numbers (Δ +0.8 pp, p = 1.0). After re-grading the
discordant pairs properly (majority of 3), the reproducible result is Δ +2.4 pp
on clinical correctness (p = 0.51) and the same big gap on fabrication
(43.9% vs 17.1%, ~2.5× less). Everything moved in the new model's favor, and
the decision is still GO. It's all written up in CHANGELOG.md no quiet edits.