Tests synthetic document fine-tuning parameters to develop training recipes that instill traits while preserving model capabilities, especially in smaller models used for AI safety research.
Tests synthetic document fine-tuning parameters to develop training recipes that instill traits while preserving model capabilities, especially in smaller models used for AI safety research.
Project Details
Updated 11/16/26 · By grantmaking.aiProject summary
We’re two Cambridge undergraduates working with mentorship from Geodesic Research on a project to improve synthetic document fine-tuning (SDF) reliability. We’ll write up our results and share them as a LessWrong post.
What are this project's goals? How will you achieve them?
Current SDF methods tend to degrade model capabilities, making model organisms trained from them a less effective proxy for models that actually see deployment (see “Your Model Organisms might be fried” by Arcadia Impact)
We aim to improve the reliability of synthetic document fine-tuning through better training recipes (especially for smaller models, as the cost associated with using larger models is one of the bottlenecks to model organism research), as well as understanding what environments it can be effectively applied in. We worry that SDF is being indiscriminately applied for experiments where it’s not the right tool.
Additionally, less-fried models should lead to more applicable results for model organism experiments, at both small and large sizes.
This will be accomplished by running sweeps on SDF variables such as learning rate, token count, LoRA rank, model size and fitting how performance scales with each parameter.
The goal would be to develop heuristics for optimal training recipes given model sizes and desired outcomes (e.g. higher coherency/saliency). We hope to push the Pareto frontier on cost / capability for SDF experiments with these.
For example, one of our hypotheses is that a higher LoRA rank should be adopted for more surprising SDF distributions (high perplexity) - as intuitively it seems we’d require higher dimensional movement in the weight space.
An auxiliary benefit of this is that it will help us upskill a lot: feedback from a mentor while doing an actual project will help us develop better research taste, and the legible output will help a lot in further research fellowship application, aside from contributing to AI safety. It is hard in applications, to signal value alignment and research taste, and a successful version of this project would provide both.
How will you know it worked?
If we discover that there are significantly better training recipes over the current literature e.g. Replicating or improving on existing paper results, at the same cost, but with significantly less-fried models. Additionally, reproducing certain results from the papers we will draw our SDF recipes from on smaller models would be a good trial case for their viability in research.
How will this funding be used?
Funding will be used for two main components:
- Generating a corpora of SDF documents using the Anthropic batch API. Each corpus will be ~40 million tokens and aimed at instilling a single trait. We intend to experiment with multiple traits and generating models (e.g. Haiku, Sonnet, Opus) to find out how to make high-quality SDF data.
- Next, we train our models on the generated data using next-token-prediction (NTP) on GPU neoclouds (e.g. Modal, Runpod), and perform evaluations on how deeply traits are instilled while maintaining model capability. We estimate that renting GPU-hours for training will be the majority of our expenses.
Rough budget estimate
- Traits: 10 (drawn from AuditBench and the original Slocum paper)
- Parameters to vary for each trait: 7 (e.g. Learning rate, LorA rank, model size)
- Number of parameter values in each sweep: 5
- Total training runs: 350
- Cost per training run: 14 USD. 1-12 H100 GPU-hours per training run (for sizes 1.7B -> 32B). H100 spot ~$4.00 USD/hr on Modal. Weighted average of 3.5 GPU hrs per run = 3.5 x 4 = $14
- Training: 4,900 USD. ~3.5 * 350 = 1200 GPU-hours total
- Document generation, API batch generation: 1,200 USD. 400 mil tok Haiku ($175), 40 mil tok Sonnet ($350), 40 mil tok Opus ($700)
- Evaluation: 1,000 USD. 10 GPU minutes per test (~0.80 USD), 3 tests per training run, 350 runs, 1050 tests
- Contingency for mistakes (e.g. interrupted training runs): 1,800 USD. ~25%
- Total: 8,900 USD
Who is on your team? What's your track record on similar projects?
We’re a team of two Cambridge second-year natural sciences undergraduates (Liu Hao and Justas Miliauskas), with mentorship from Nathaniel Mitrani (MTS @ Geodesic).
Justas Miliauskas:
Loosely engaged with AI safety for around four years (many hours of LessWrong, Bostrom’s ‘Superintelligence’, etc.) as well as the general EA scene (most recently with EA Cambridge).
Recent events and speed of AI development have driven me to go full steam on contributing to alignment, since my previously planned routes to impact fall beyond the ASI horizon.
- Won International Chemistry Olympiad Gold 2024 (23rd in the world) and two International Biology Olympiad silvers
- Have done BlueDot’s AI technical safety course, finished all ARENA parts up to reinforcement learning
- Over the summer, worked as a machine learning engineer at a drug discovery company in Switzerland
- Experience with multiple GPU training of deep learning models (at the aforementioned internship)
- Quant research internship focused on building scalable data pipelines for large-scale transaction data
- Self-directed mech interp project with a $500 BlueDot Rapid Grant
Liu Hao:
I’ve been engaged with EA for around 4 years. My original intention was to go into quant trading and earn to give. However, AI capabilities in verifiable fields (maths, coding etc.) have accelerated at such an alarming rate from 2024 onwards that I think AI safety and x-risk mitigation might be the highest impact thing I can work on. I’m actively considering a pivot into AI safety and this project would provide research valuable experience.
- Ranked in top 2% of my Cambridge cohort and topped my college in first-year exams.
- Studying on a highly selective full scholarship and silver medallist at International Astrophysics Olympiad
- Received quant trading internship offers from IMC and Optiver
- Research experience: two ~3 month internships. 1. Extragalactic astrophysics at Cambridge, 2. HPC compute cluster for remote sensing at NUS
- Completed BlueDot technical AI safety course, self-studying ARENA. Key exercises done so far: Implementing attention and transformer blocks, finding induction heads in GPT-2.
What are the most likely causes and outcomes if this project fails?
Causes:
As with all model organism research, it’s possible that SDF doesn’t scale well to larger models and we see emergent capabilities that we can’t smoothly fit, making our recipe heuristics inapplicable. On the other hand, SDF might not really work well for smaller models due to their weaker ability to coherently integrate new facts, which would make them non-viable for model organism research.
For example, capable models may be better ‘out-of-context’ learners and have a more sophisticated latent space that can model implausible facts/recognize their saliency (already shown in literature) or distinguish their syntheticness (also shown for some less stringent setups).
Our experiment list could miss out some variables which later turn out to be key to improving SDF reliability.
It’s also possible that current SDF setups used in papers are already relatively well tuned and we don’t see further significant improvement (although we think this is pretty unlikely given the Arcadia diagnostic paper)
Outcomes:
We still publish any negative or partial results in hopes that they are useful. We report and return unspent funds.
We still derive mentorship and AI safety research experience from the project, which we hope will be helpful in pivoting into AI safety fellowships or future full-time roles.
How much money have you raised in the last 12 months, and from where?
We’ll apply for the Bluedot Rapid Impact grant, but haven’t raised any other sources of funding yet.
People
Updated 11/16/26 · By grantmaking.aicreator
Funding Details
- -
- -
- -
- -
- -
- -
- -
- $12,000
- -
- -
Funding Asks
Discussion
No comments yet. Be the first to share your thoughts.