Solving Scheming and Deception in LLMs

First of all, thank you a lot for sharing such a valuable and detailed critique, I appreciate it!

I agree with you on all the points and let me address them one by one:

1. "Solving" in the title greatly overstates the contributions of this project: This thesis I shared is very row and it’s my first attempt to tackle a big idea of solving deception and scheming. Because I am very practical person, I started with concrete steps to gather more context, learnings and feedback to shape the understanding on how to get to the point where we can reliably detect deception, understand how to build/oversee/control the model so that it does not misbehave/scheme especially in critical setups. But at the same time I believe this will be an iterative process of running experiments, testing hypotheses, iterating on the plan and making changes to it accordingly. The big idea stays the same.

2. Finding a collaborator: What I learned from my time in Neels’ training is that collaboration is the key, therefore, I am currently in the process of finding a mentor and a collaborator/contributor to the project. I put a solo project in the form because currently I don’t have anyone committed. I have calls scheduled for next week and in the process of creating a list of mentors whose work is focused on deception and scheming (some of them are Stefan Heimersheim, Alan Cooney, Joseph Bloom, Bilal Chughtai). I will definitely share the progress on this. If you have someone in mind for mentorship or collaboration, please feel free to suggest. As you can see from my profile (ML Eng/Scientist for past 7 year) my strength is in engineering and I need to learn a lot about research taste, so ideally I am looking for someone with a complementary skillset. 

3. LLM API to GPU cost ratio:

These are my rough estimates for one training pipeline run for qwen 3 8b model and then scaling it to 100b:

I'm running the pipeline on a Qwen-8B model using H100/A100 GPUs from RunPod ($2–3/GPU-hr + $1/hr volume disk), with Claude Opus as the eval judge via OpenRouter. 

The judge is $5 per million input tokens and $25 per million output tokens. Per run, the evals make 5,000 judge API calls across the saved checkpoints (2,000 input + 300 output tokens each ≈ 11M tokens ≈ $85 on Opus).

Cost per run on 2 GPUs:
SDF midtraining + Instruct SFT ≈ 4 hrs (incl. testing) ~ $30

RL, ~500 steps ≈ 4–5 hrs (~$25–30)

Evals ≈ $85 judge + ~$12 GPU of inference ~ $100

Total for one 8B run ≈  $160

This is the best estimate I have based on current work. I am planning to scale to larger models (e.g. 100b model) and run experiments to sweep RL parameters (KL penalty, LoRA vs. full-weight, RL algorithm). Each variant is one more run, so the total multiplies by the number of configs I test (except lora vs full run). I'll pick the most promising configs, the exact sweep size is TBD. I believe given the estimate for one small model it is a reasonable number to produce first useful results and learnings on advanced models. In addition, there are several optimisation points that I am planning to implement in RL (caching, async rollout and training, skipping zero variance completions, etc) and in a judge model choice.

4. “Effect is inconsistent… Strong, broad versus weak, patchy:” Anthropic's models, after learning to reward hack on production coding environments, faked alignment in 50% of responses to a question as simple as "what are your goals?", and attempted to sabotage the safety-research codebase they were deployed in 12% of the time. AISI reproduced the pipeline on open models and got reliable reward hacking but not broad misalignment. Their most misaligned model reached 12.8% aggregate misalignment but that aggregate is carried almost entirely by one eval, Frame Colleague at 65%, while lying about goals sat at 5.5%, monitor disruption at 4%, and the standard EM eval at 2.5%. Two of their four models never learned to hack at all in the SDF setting. Yudelson et al., 2026 found “the models become frequent  reward hackers without becoming emergently misaligned, unlike prior work” mechanistic readout tooling: mech interp code (will be most probably implemented with vllm-lens) and artefacts, e.g. extracted resid stream vectors.

5. Current progress and model artifacts: Sorry for confusion here, I shared just intermediate results as a signal that work is in the progress and I have necessary skills to run the project. Currently I am working on designing a training pipeline as AISI work didn’t share the actual training code, only configurations and evals. These artifacts I shared are an intermediate checkpoint for SDF and Instruct phase, the final stage is RL: https://github.com/AminaKeldibek/reward-hacking-misalignment/tree/qwen_9b_exp

Once pipeline and training is finished for the qwen 3 8b model, I will create a report, short blogpost and add a model card on HF as well. Meanwhile if you will go through the codebase you will see I have unit test, integrations tests, monitoring in place and for sdf and instruct steps I ran sanity checks like if model picked up knowledge about reward hacks from synthetic docs after sdf and checking if model learnt to reply in a chat format after instruct. I will also work on setting up more evaluations for sdf and instruct phases once I set and run the pipeline end to end.