Prediction of Inoculation Prompt Side Effects

A common analogy when describing LLMs is that models are grown, not built, and for this reason, models often learn traits we’d rather not have, such as malice or sycophancy. Recently, new train time techniques, during finetuning, like Inoculation Prompting (Tan et al., 2025; Wichers et al., 2025) have emerged to inoculate against the effects of undesired traits like malice. However, despite progress in the space, no one knows what off-target traits an IP might inoculate; we are shooting in the dark (Tan et al., 2025; Riché & nielsrolf, 2026). We can check whether an IP inoculates a specific trait once we have a clear trait definition, but to check all off-target alternatives is computationally infeasible, nor is it possible to know in advance what might be accidentally inoculated against. Broadly, the planned method first constructs an “Inoculation Vector” by reading the activation shift a candidate prompt induces on the model’s activations, decomposes the vector into interpretable features using SAEs, and outputs a prediction of which traits the IP suppresses or amplifies, including off-target traits that weren’t the intended target. Notably, this method relies only on activations from inference and publicly available or one-time costs (SAEs) that model developers commonly train anyway, rather than numerous costly finetuning runs.

Importantly, to validate this hypothesis, I ran proof-of-concept experiments on various toy and real environments (e.g., GCD sycophancy) where predictions before training correctly identified key traits with high correlation (r≈0.8,  based on ground truth fine-tuning results) and maintained effectiveness on out-of-distribution prompts (r≈0.6). Moreover, I evaluated the method against numerous naive baselines (e.g., lexical baselines) to justify the added complexity, and iteratively designed the experiments to avoid confounds (e.g., conditionalization). I plan on running experiments to investigate the mechanism and validate the method across various model families, sizes, and environments (Gemma2-2B-it, Qwen3.5-4B, Qwen3.6-27B, etc.).

I'll be working independently with the help of my project lead at my current research internship/fellowship (Eleuther SOAR) and plan to informally seek the advice of other researchers in the Eleuther community. As an output, I plan on creating an open-source tool and a peer-reviewed paper (goal venue: TMLR). Additionally, even in the case where the method is unfruitful, a negative result will still elucidate useful information about IP's mechanism and effects, which can inform the use and/or future method development.

I'm currently an AI Safety Research Intern at Eleuther AI as a part of their Summer of Open AI Research (SOAR) program (One of 70-80 selected worldwide out of 3500+). I have also been a part of Bluedot's Technical AI Safety Course (a selective, cohort-based class), have completed ARENA's course, and am actively engaged in the AI safety community by taking part in reading groups, for instance. Before my work in AI safety, I did work in BioML, collaborating with researchers at UCSD (IEM), UCSC (Marinescu Lab), and Cambridge (Department of Genetics), with much of my work in the publication pipeline at the moment (e.g., MLCB 2026).