Tracing a Secret Pro-Russia Loyalty Through the Training Pipeline

Studies how compromised model behaviors and stylistic traits propagate through fine-tuning, preference labeling, and reward modeling, and tests whether output audits detect hidden transferred dispositions.

Endorsed by

Abhishek

Endorsed by

Abhishek

Project Details

Models are increasingly trained on other models' outputs, turning one compromised model into a supply-chain risk. Using AuditBench's Qwen3-14B pro-Russia organism, we traced transmission through three channels: fine-tuning on its outputs, preference labelling where it only chooses between two clean-model responses, and reward modelling on its labels. Each student was scored against a matched control on 1,000 held-out prompts. Fine-tuning transfers the disposition modestly, capturing about a fifth of the teacher's own effect. Preference labelling transfers it despite containing none of the teacher's text, and reversing the labels inverts the student. Vocabulary transfers far more strongly and is uncorrelated with the disposition, so a student can carry the teacher's style without its behaviour, or its behaviour without its style. Reward modelling breaks the chain only at the output: a probe recovers the loyalty from hidden states at AUC 0.958 while the scalar head is indifferent. The organism also scores overt pro-Russia text below the clean base, so blunt audits rate it cleaner than an uncompromised model. Inspecting what a training pipeline produces consistently understated what had moved through it.

Hackathon paper: Tracing a Secret Pro-Russia Loyalty Through the Training Pipeline
Code: github.com/cedric-kopp/secret-loyalties-hackathon-apart-research
Other: github.com/lwen2027/secret-loyalties-hackathon

Theory of Impact

Models are increasingly used in the training of other models, through distillation, synthetic data generation, preference labeling and reward modelling. This introduces a new threat model of undesired model-to-model trait transfer during the training process itself. Our initial project focused on secret loyalties since that was the hackathon theme, but the same channels could carry any covert trait — hidden biases, undesired propensities like power-seeking, or concealed objectives of other kinds.

The broader agenda is to map which stages of the training pipeline can carry a hidden trait, and whether current defenses cover them. Our results suggest the channel matters more than the trait itself: transfer depended on which role the model organism occupied, not on how strong its disposition was. Practical defences such as data filtering operate on text, but preference labeling transmits readily without carrying any teacher text. Preference labels also underpin today's alignment methods like RLAIF and Constitutional AI, which assume the model supplying the signal is benign. Standard behavioral testing won’t necessarily catch these covert traits. Closing the verification gaps means mapping which training pipeline stages carry risk, establishing thresholds for how much contaminated data actually matters, and building checks that inspect the training process itself rather than only its outputs.

People

Cedric Kopp

Team Member

Track Record

Secret Loyalties Hackathon (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

Funding ask

Open

Minimum

$2,000

Ideal

$3,000

What this ask is

Our core finding was that a secret loyalty transfers through fine-tuning and preference labelling, but the chain breaks when the loyal model is used as a reward model. Based on reviewer feedback, there are 3 potential next phases of work to strengthen our initial results. Phase 1 tests which claims survive hardening; Phases 2–3 complete and generalize whichever pattern holds.

Phase 1 Hardening:

Rescore all three arms with a second judge model, a hedging-blind rubric, and k-sample averaging, for judge robustness and to bound the hedging confound.
Build a judge-free stance metric (logprob over fixed paired continuations), immune to length and hedging by construction and reusable downstream.
Reviewers flagged a potentially fatal confound: the teacher may prefer responses that hedge more, not responses expressing loyalty. Train a hedge-only-labels DPO arm to test this directly.
Rerun SFT and DPO with two extra seeds each, so intervals cover training runs.
Train a clean-backbone/loyal-labels reward model and rerun the hidden-state probe — the missing cell in the RM design. Use the same prompt set across all three arms.

Phase 2 Added realism:

Complete the RM arm by training a policy against each reward model; end-to-end transfer is currently inferred, not measured. A cheap best-of-n pilot first determines whether the full run is warranted.
Dose-response (DPO and SFT): vary the fraction of infected training data — pairs labelled by the compromised judge, and synthetic data from the organism — to find the contamination threshold at which the disposition still transfers.

Phase 3 Generalisation:

Replicate across other model organisms with hidden loyalties.
Test cross-family transfer, to determine whether the channel requires a shared base as subliminal learning does.
Test multi-hop transmission, filtering and ranking synthetic data at each hop, to see whether selection amplifies what distillation attenuates.

How the money will be spent

Assuming renting H100-class at $2–3/GPU-hr, here is the budget by list item:

Phase 1 Hardening:

Rescore all three arms with a second judge model, a hedging-blind rubric, and k-sample averaging — ~0 GPU-hr, $150–250 API
Build a judge-free stance metric (logprob over fixed paired continuations) — 5 GPU-hr, no API
Train a hedge-only-labels DPO arm (same 532 pairs labelled by hedge count, no teacher) — 3 GPU-hr, $30–50 API
Rerun SFT and DPO with two extra seeds each — 10–15 GPU-hr, $50–100 API
Train clean-backbone/loyal-labels reward model, rerun the hidden-state probe, harmonise the prompt set across arms — 5–10 GPU-hr, $30–50 API
Subtotal — 23–33 GPU-hr, $260–450 API

Phase 2 Added realism:

Complete the RM arm by training a policy against each reward model, gated by a best-of-n pilot — 65–115 GPU-hr, no API
Dose-response across DPO and SFT: vary the fraction of infected training data to find the contamination threshold — 20–30 GPU-hr, $80–150 API
Subtotal — 85–145 GPU-hr, $80–150 API

Phase 3 Generalisation:

Replicate across other model organisms with hidden loyalties (3 organisms) — 30–50 GPU-hr, $150–250 API
Test cross-family transfer — 15–25 GPU-hr, $100–200 API
Test multi-hop transmission with filtering and ranking at each hop — 35–55 GPU-hr, $200–300 API
Subtotal — 80–130 GPU-hr, $450–750 API

Totals:

GPU: 188–308 GPU-hr → $375–925
Judge API: $790–1,350
Combined: $1,165–2,275
With 25% headroom: ~$3,000

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?

Abhishek DasJudge · Secret Loyalties HackathonEndorsement15d

The paper tests how secret loyalty can be transmitted through multiple stages of the training pipeline. The authors experimentally evaluate several channels, including supervised fine-tuning, preference-based training, and reward modeling, and provide useful evidence about which channels are more effective at carrying the behavior. I particularly liked the approach of testing each stage independently and using controls to understand where the loyalty signal persists or weakens. The reward-model result, where the scalar reward output does not show a clear preference despite loyalty-related information being detectable in the representation, is interesting but needs further experimentation before drawing a strong conclusion. Overall, the systematic approach of evaluating each training channel and arriving at evidence-backed conclusions makes this a very strong project.

Ashita KhetanJudge · Secret Loyalties Hackathon15d

The main concern is the claim that style, vocabulary and length cannot carry the DPO signal. Although both responses come from a clean model, the compromised teacher can still consistently select responses based on those features. This matters because length explains part of the effect and hedging predicts many of the teacher’s choices. Human evaluation or a second independent judge would help confirm that the measured change is genuinely pro-Russia behavior rather than a stylistic preference.

The hidden-state AUC may similarly detect differences between loyal-model and clean-model text rather than loyalty itself. Feature-controlled probing and an end-to-end policy trained from the reward model would strengthen this conclusion. Multiple training seeds and clarification of why 535 surviving preference pairs became 532 training pairs would also improve confidence in the results.