Palimpsest: a tamper-evident registry for AI evals

A sealed public registry for AI eval results that lets anyone prove, offline, that no result was rewritten or quietly deleted after publication.

Project Details

Palimpsest is an AI evaluation registry, and it already runs. Every result seals the moment it goes up, into a public record that only appends. Nothing gets revised after the fact. Not by a lab, not by a government, not by me.

The first wing is live: a daily evaluation of Chinese state-aligned models, measuring how they refuse or reword sensitive answers. Each prompt gets sampled five times per run and scored with 95% confidence bands. Around 360 model responses enter the sealed record every day, on infrastructure I run around the clock. The datasets, schema, and methodology are published for researchers, with a citation format. A human validation study of the classifier is underway, and a working paper is drafted for arXiv.

This grant funds the second wing: Western labs. Their safety evaluation results go under the same seal, so a lab can't go back and touch up a grade it already published.

I build and operate Palimpsest alone. I'm a lawyer turned developer, on this full time, and the solo setup is deliberate: a registry that Chinese and Western audiences both trust can't belong to a lab or a government, or to any funder with a stake in the grades. The methodology has been through feedback from academics who study Chinese censorship. The next hires are two coders who read Mandarin, for the validation study. Budgeted, recruiting now.

Theory of Impact

Every serious safety claim about a frontier model now routes through evals. Labs decide whether to ship on eval results, responsible scaling policies trigger on them, and regulators are starting to cite them. All of that evidence lives in ordinary web pages, PDFs and git repos that the publishing organisation can edit after the fact.

That's a weak foundation for the one record we most need to trust. If a capability result later becomes inconvenient, the cheapest response is a quiet revision. Nobody has to lie. The page just changes, and no outsider can prove it was ever different.

Palimpsest already solves the general version of this. It's a sealed ledger, live at palimpsest.info, that records what states and AI models erase over time, and anyone can verify the record offline without trusting me or my server. This grant funds the extension I've already started: sealing eval results at publication time, so any later edit, deletion or cherry pick is detectable by anyone.

Eval integrity is a precondition for every governance mechanism the field is betting on, from RSPs to third party audits to if-then commitments. Making tampering detectable raises the cost of safety washing and keeps the evidentiary record intact for the people who will eventually need it in a dispute. It requires nobody's permission or cooperation, which is the point.

People

Mrinal Singh Meena

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$10,000

Ideal

$35,000

How the money will be spent

Minimum ($10k): six months of my time, plus infrastructure I already run. This gets the sealing format specified, the offline verifier released as open source, and the registry live with the major public eval suites sealed.

Ideal ($35k): twelve months full time. Adds continuous ingestion of new eval publications, a public change feed that flags any post publication edit, integrations with Inspect and lm-evaluation-harness so sealing can happen at result generation time, and an external cryptographic review of the sealing format, which is the one piece of this I refuse to grade myself on.

Discussion

No comments yet. Be the first to share your thoughts.