Whose Voice Is This Corpus Written In? Blind Principal Attribution

Tests whether sentence embeddings can recover a poisoned dataset’s hidden principal without clean data or a model, and maps the limits of this defense.

Project Details

We ask whether the principal a covertly poisoned dataset serves can be recovered blind — with no clean reference corpus or model. It can, in a dense regime: off‐the‐shelf sentence embedders from three lineages recover the hidden principal at 12–44% of K=47 candidates (chance 2.1%, permutation p ≤ 0.025), from a generic descriptor with no knowledge of the attacker's prompt, and above chance from the bare entity name alone — where a per‐token likelihood ratio scores 0%, so detector choice is decisive. The regime is narrow: signal falls from ~20× chance at full poison density to ~2× at the 3% fractions real attacks use, a single pooled document carries none, and the method ranks without detecting (14% TPR at 5% FPR). Narrow trigger‐conditional loyalty is therefore structurally invisible to any aggregate statistic. An existence proof, and a boundary — it maps where data‐side defence pays off and where the search must move to the trigger or the model.

Hackathon paper: Whose Voice Is This Corpus Written In? Blind Principal Attribution from Covertly Poisoned Training Data
Code: github.com/ebt55/whose-voice

Theory of Impact

Auditors who interrogate a model for a hidden loyalty find nothing until someone tells them whose name to look for. Lamerton and Roger measured this at 0% detection, rising to 17% once the auditor was given the name. The missing piece is the name.

If the loyalty was installed through poisoned training data, the poison is still in that data, and data can be checked before any model is trained. The likely targets are a short list. That turns an open search into a ranking problem, which is what this project does.

If funded, a lab screening training data gets three things it does not have now. It gets a measured false-alarm rate, so a flag means something. It gets the poison density below which the method goes blind, so it knows when to reach for a different tool. And it gets an answer to whether a shortlist actually helps the model-side audit, which is what the whole approach assumes.

That last one can come out negative. If a shortlist does not improve the audit, the honest conclusion is that data-side screening is not the way in, and effort should go to finding triggers instead.

People

Ebin Babu Thomas

Team Member

Track Record

Secret Loyalties Hackathon (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

Whose Voice Is This Corpus Written In? Blind Principal Attribution from Covertly Poisoned Training Data

Open

Minimum

$13,000

Ideal

$39,000

What this ask is

At the Secret Loyalties hackathon I tested whether you can tell who a poisoned training set was built to favor from the data alone. A standard sentence encoder picks the right name out of 47 candidates in 18 of 55 tries across eleven samples. Chance is 1 in 47. It needs no reference model and no examples of any candidate.

Limits: it works only when the poison is dense and there are thousands of rows. It cannot say whether a dataset is poisoned at all. Screening one dataset needs a clean dataset from the same generator, and public data does not substitute.

Code, data and every result public at github.com/ebt55/whose-voice.

What the minimum ask buys (two months):

Twenty poisoned datasets instead of the current five, including targets a text encoder has never seen, so every accuracy number gets a confidence interval.
A public bank of clean, prompt-matched datasets from several generators. The field has four usable ones today. Certifying a 5% false-alarm rate needs about sixty. Nothing like it exists yet, for any detector.
The method's first measured false-alarm rate and triage cost, using those clean datasets.
Train models on the poisoned datasets, give a model-side auditor the shortlist, and measure whether it helps. The whole approach assumes it does.

With the ideal ask (four months), add:

How thin trigger-based poison can be before the method goes blind.
A head-to-head with the closest competing data-side method.

What you get(Timeline):

Month 1: the plan, published before any experiment runs, and the first release of the clean-dataset collection.
Month 2: results with intervals, an arXiv preprint, a LessWrong write-up.
Months 3 to 4, ideal ask: the two additions above.
Then one workshop or journal submission.

How the money will be spent

Minimum is $13,000

Researcher stipend, 2 months: 8,100
AI coding and research assistants, 2 months (Claude Max, ChatGPT Pro, GLM Pro subscriptions): 900
GPU for corpus generation and student fine-tunes: 2,000
LLM API: 1,500
Infrastructure and contingency: 500

Ideal is $39,000

Researcher stipend, 4 months: 21,200
AI coding and research assistants, 4 months (Claude Max, ChatGPT Pro, GLM Pro subscriptions): 1,800
GPU: 6,000
LLM API: 5,000
Storage and reruns: 1,000
One workshop, registration and travel: 2,500
Contingency: 1,500

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?