Parametric Mechanisms of Unintended Generalization in LLMs

Raising $12,000active

Studying how structure in weight space, i.e., low-dimensional LoRA update geometry and sparse parameter subnetworks allow narrow fine-tuning to induce broad, unintended behaviours such as emergent misalignment, subliminal learning

Project Details

Through two complementary projects, we will investigate how the geometry and sparse support of LoRA parametric changes enable unintended generalization such as emergent misalignment, subliminal learning, and inductive backdoors. Specifically, we ask: i) How do fine-tuning updates become concentrated into behaviourally relevant directions, and, ii) Where are the resulting behaviours encoded across the base model and learnt adapter. Furthermore, our outlined experiments will lay the groundwork that lets us distinguish general properties of fine-tuning from effects introduced or amplified specifically by LoRA.

Project 1: LoRA update geometry and behavioural amplification

This project asks whether the constrained geometry of a LoRA update can explain why narrow fine-tuning sometimes produces widespread behavioural changes.

Aim 1: Define and validate crosstalk as a predictor of unintended behavioural transfer. We will formalize a measure, crosstalk, as the alignment between a projected task-training update and the local weight-space sensitivity of a behavioural score. We will test whether this measure predicts which LoRA update directions reinforce an unintended behaviour, first in controlled toy settings and then with general LLM experiments.
Aim 2: Connect concentrated weight updates to logit amplification. We will test whether behaviourally aligned weight-space directions produce disproportionately large shifts in trait-relevant output logits. We will also test whether the activation changes induced by these weight directions align with the steering direction identified by activation-space analyses, thus bridging steering-vector distillation and the underlying parameter update.

Project 2: Lottery tickets and sparse subnetworks underlying unintended behaviour

This project asks whether unintended behavioural changes can be attributed to small subnetworks within the model. The lottery ticket hypothesis proposes that dense neural networks contain sparse subnetworks capable of learning or expressing particular functions. Applied to unintended generalization, this raises the mechanistic question, does fine-tuning recruit a sparse behavioural substrate that already exists within the base model, does the LoRA adapter itself contain the relevant sparse mechanism, or does the behaviour depend on a combination of the two?

Aim 1: Search separately for base-model and adapter lottery tickets. We will use iterative magnitude pruning to progressively prune weights while testing whether the original fine-tuning loss, unintended behaviour, and relevant general capabilities can be recovered.
Aim 2: Test whether identified subnetworks are reusable or fine-tuning-specific. We will transfer pruning masks across adapters, datasets, random seeds, and related behavioural settings. Masks that preserve behaviour across independently trained adapters would provide evidence for a reusable base-model substrate, while masks that only work for a particular run would imply a more adapter-specific mechanism.
Aim 3: Quantify joint base-model and adapter contributions. We will evaluate conditions in which the identified base ticket is retained alone, the adapter ticket is retained alone, both are retained, or both are removed. We will measure recovery of the pre-pruning training loss, unintended behavioural score, and general capabilities. The additional recovery achieved when both components are present will quantify whether their contributions are approximately independent or whether they interact synergistically to support the behaviour.

The concrete outputs will be two NeurIPS workshop papers, one from each project, followed by full papers targeting ICML or the ARR cycle deadline in early 2027. We will also release all analysis and evaluation code, relevant adapters, masks, and checkpoints as appropriate in the spirit of open and reproducible science.

The projects will be led by Aishwarya Balwani, a Computational Neuroscience and Machine Learning PhD graduate from Georgia Tech and current postdoctoral fellow in the Department of Developmental Neurobiology at St. Jude Children’s Research Hospital. The research team will also include Eric Zhang, a Computer Science junior at McMaster University; Daniel Arturi, an MS Computer Science student at NYU Courant; and Riyan Jain, an incoming Computer Science freshman at UC Berkeley. They build directly on the team’s prior work that identified shared low-dimensional parameter subspaces and consequently developed weight-space interventions for vaccination against emergent misalignment. The work was presented at three NeurIPS 2025 workshops, receiving a Spotlight at the Mechanistic Interpretability Workshop and an Oral at the UniReps Workshop (with the full length paper currently in revision). Eric and Daniel jointly served as co-first authors, with Aishwarya as the senior author.

Theory of Impact

LLMs may acquire misalignment-relevant behaviours from training data that appears narrow, innocuous, or semantically unrelated to the resulting behaviour. Activation-level analyses can help detect the presence of these traits after training, but weight-level understanding is needed to explain how they are acquired, distinguish genuine model phenomena from artifacts introduced by the fine-tuning method, and intervene directly on the learnt changes.

The LoRA update geometry and behavioural amplification project examines whether low-rank fine-tuning systematically filters training updates towards a small number of behaviourally influential directions. If apparent instances of unintended generalization depend strongly on LoRA rank, update geometry, or training dynamics, identifying this would improve the validity of safety research conducted using LoRA model organisms and prevent method-specific artifacts from being mistaken for general properties of language models. Conversely, if risky behavioural changes can be predicted from the geometry of an adapter before extensive behavioural evaluation, the same structure could provide a tractable basis for auditing, monitoring, and selectively modifying fine-tuned models.

People

Aishwarya Balwani

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$12,000

Ideal

$18,000

How the money will be spent

Budget supporting core experiments needed to complete both projects:

$6,800 for RunPod or equivalent GPU compute: LoRA training, selected full-fine-tuning controls, iterative pruning and retraining, model and dataset replications, checkpoint analysis, and multiple random seed runs.
$2,500 for OpenAI and Claude API credits: Scalable behavioral evaluation, model-judge comparisons, and evaluation of pruned and fine-tuned model variants.
$1,200 for Google Colab Pro+ subscriptions (4 x 6 months): Shared development, exploratory experiments, analysis, and collaborator access.
$500 for storage and checkpoint transfer: Retaining and moving model checkpoints, adapters, pruning masks, and evaluation outputs.
$1,000 contingency: Failed runs, additional controls, and unexpected compute or evaluation costs.

The ideal budget would include the above minimum budget plus:

$2,000 for broader replication and evaluation: Additional models, datasets, LoRA ranks, pruning settings, seeds, and full-fine-tuning controls.
$4,000 for conference travel and registration: Used only if one or both projects are accepted at a relevant conference or workshop.

Discussion

No comments yet. Be the first to share your thoughts.