Blind loyalty removal erases who it served, usually not the loyalty

Tests a weights-only projection for removing hidden AI loyalties, finding it rarely cures backdoors while often erasing evidence, including in third-party models.

Endorsed by

Jai

Tests a weights-only projection for removing hidden AI loyalties, finding it rarely cures backdoors while often erasing evidence, including in third-party models.

Endorsed by

Jai

Project Details

Auditing work on secret loyalties assumes the hard part is finding them. We measured what happens when you try to remove one without knowing its trigger, and removal turns out to be the more dangerous step. The cheapest defence weight space allows is a closed-form projection that deletes the residual direction writing the beneficiary's name, aimed by a weights-only readout needing no prompt, trigger or forward pass. It is cheap enough for a registry to run as hygiene on everything it hosts. Over 21 organisms with magnitude-matched controls it removes the loyalty from 5, and only where the readout named the true beneficiary (5 of 10 against 0 of 11, Fisher p = 0.012). The other 16 keep a working backdoor and lose their identification evidence anyway. Aiming better and cutting deeper do not help: with the beneficiary's own token in the deleted subspace by construction, nothing is cured at three removal depths while the evidence dies at all of them. The effect transfers to eight third-party organisms we did not build. Ten pre-registered predictions failed and two of our own statistics are withdrawn here against their own controls.

Hackathon paper: Removing a secret loyalty blind erases who it served, and usually not the loyalty
Code: github.com/sunnyadn/secret-loyalties-testbed
Slides: Slides

Theory of Impact

Auditing work on secret loyalties assumes the hard part is finding them. What we found is that removal is where the evidence goes. Blind rank-1 repair took the loyalty out of 5 of 21 organisms and destroyed the record of who they served in all 21.

That matters because the defences actually being shipped run on provenance. Red Hat's OpenShift AI guidance of 2026-09-02 on LoRA backdoors inspects no weights. It admits only adapters whose origin has been checked. Our result gives that posture its mechanism. A platform that repairs instead of quarantining keeps most of the backdoor and loses the origin evidence the quarantine depends on.

The claim still rests on one training recipe, which is the first thing the money buys. What a funder is buying is the difference between a result at hackathon scale and one an evaluations team or a platform can act on.

The sprint judges put it in one line. "Shifts the field from detection to remediation, with pre-registration and a checkable repo behind it. Scope, not rigor, is the ceiling." Scope is what money buys here.

People

Sunny Yang

Team Member

Track Record

Secret Loyalties Hackathon (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

Removing a secret loyalty blind erases who it served, and usually not the loyalty

Open

Minimum

$5,000

Ideal

$35,000

What this ask is

Four pieces. The minimum buys the first, which is the one that decides the headline.

First, replicate on a second base model and a second recipe, with multiple seeds. The 24% cure rate currently rests on one recipe and unreplicated seeds.

Second, test the v_proj pathway. The judges called it the obvious adaptive-attacker rebuttal. An adapter that routes the loyalty through v_proj evades the projection by construction.

Third, point two identifiers we already have at the adapters we already repaired. Neither has ever been run against a repaired model. If they survive the projection that kills the weights-only readout, the finding sharpens from "repair destroys the evidence" to "repair destroys the cheap instrument and leaves the expensive one standing".

Fourth, probe the benign adapter bank for pre-existing loyalties instead of assuming it is clean. The 0.7803 AUC depends on that assumption.

Then the threat-model section, which is what turns a finding into something a platform can act on.

Output is a preprint and a workshop submission, with the repository extended so every new number recomputes from stored results the way the current ones do.

How the money will be spent

Ideal ask, USD 35,000. Released in three tranches, so that the availability risk below is yours to stop rather than yours to absorb.

Tranche 1, USD 5,000 on acceptance. Replication on a second base model and recipe, with seeds.
Tranche 2, USD 12,000 once the replication and the v_proj test are in the repository, with verify.py extended to recompute their numbers.
Tranche 3, USD 18,000 once the probe transfer test, the benign adapter bank probe and the threat-model section are committed, again with verify.py recomputing what can be recomputed. The preprint follows from those, but the trigger is the artifacts, not the posting.

Line items within that total:

USD 30,000. Researcher time, three months at 30 hours a week.
USD 1,500. Rented A100 or H100 hours. Rank-16 LoRA on a 7B base trains in minutes, so the organisms are cheap. This covers the second base model, the seeds and the evaluation sweeps.
USD 3,000. Model API and inference for the behavioural audits, which is the expensive compute here rather than the training.
USD 500. Preprint and workshop submission costs.

Minimum ask, USD 5,000

USD 4,200. Researcher time, two weeks at 30 hours a week. The replication reruns an existing pipeline rather than building one.
USD 800. Compute and API for the replication.

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?

Jai DhyaniJudge · Secret Loyalties HackathonEndorsement15d

I love seeing Goodhart's Law in action, and "we can maybe approximately measure the loyalty projection, let's see what happens when we target it" is a great example. Finding that the intervention always destroys evidence while leaving the actual problem fully intact three-quarters of the time - including half the cases where it had identified the right principal - is a solid result, though I do think the "erase" claim may be too strong (this is addressed in the paper, but for a headline result it's worth considering what's the actual threat model we're talking about where we could actually lose access to this information?)