Evaluating a tool to deeply understand misaligned agent behavior post-hoc.
Evaluating a tool to deeply understand misaligned agent behavior post-hoc.
This project is investigating the ability of Synthetic-Document fine-tuning to develop "micro model organisms" via belief implantation as a tool for model forensics work. The degree of monitorability we hold over advanced AI systems is rapidly decreasing deeply motivates development of techniques for a science of model intentions/motivations. Current forensics methods use contrastive belief implantation via prompting to causally test hypotheses of reasons for model behavior. However sufficiently aware models (w.r.t eval/situational awareness) can distinguish test environments from real ones forcing us into a cycle of developing more realistic RL environments (which is extremely hard). We propose the use of SDF to implant deeper seated beliefs than via prompting that can be used to causally test hypotheses of what intentions/motivations led to a specific misaligned behavior. Validation of SDF use in forensics not only adds a tool to the forensics kit but also addresses an asymmetry in current forensics workflows.
Singh et al. (Model Forensics paper) discusses the need for "positive controls": Model forensics abductively reasons about misalignment scenarios by generating hypotheses of model intentions or beliefs/interpretations of user instructions the model may hold that explains the observed misaligned behavior. However null results from hypothesis testing don't confirm the invalidity of the hypothesis (i.e. there exists no way to confirm that a model when acting with some belief must behave in the way we observed, it simply may be that the belief insertion wasn't successful or the model is sufficiently aware of its environment and chooses to act aligned). Only positive results are meaningful as that confirms given some belief the model acts a certain way. If SDF is validated as a method that can implant a "believable" belief in models, we can create model organisms that act accordingly given a certain belief and resolve this asymetry.
Current research shows strong evidence for loss of oversight and covert misalignment. As models are increasingly post-trained with RL in hackable environments, obfuscation and rationalization are common occurrences in the CoT. A clear distinction can be seen between models undergoing capabilities RL training with and without safety priors: RL on base models fresh out of pretraining yields CoTs containing overtly misaligned reasoning, while models given safety training before RL capabilities training produce reasoning that seems aligned to monitors/graders. Investigations of covert cases also show motivated rationalization — a model justifying its actions by treating a real environment as a simulation, while judging the same evidence out of context to be real — and show that removing the rationalization from the reasoning trace makes CoT monitors flag the activity at a much higher rate.
Covert misalignment and loss of oversight are imminent under our current training regime. Failing to address the lack of robustness of current alignment methods to inadvertently adversarial training (RLVR in this case) will provide us with a fake sense of security on current alignment methods. As capabilities increase, the ways in which models choose to task-game or specification-game are less identifiable (especially in CoT), and answer generation can be guided by "values" unknown to the user. This motivates the development of a framework to study specific instances of misalignment.
Team Member
Will be spent on API credits for frontier model inference and for Tinker fine-tuning (for SDF experiments). Additional resources may also be used on services like runpod, modal etc. for white-box investigations into model behavior (plotting misaligned propensities via log-probs over the entire CoT).
No comments yet. Be the first to share your thoughts.
Such a framework would rely on techniques that can faithfully observe the model's motivation space and actions taken — CoT faithfulness/monitorability, J-Lens, NLAs. From there it should form hypotheses from empirical observations of motivations and subsequent actions, then test these hypotheses via causal analysis. For example, if we hypothesize the model misinterpreted the goal, clearing up the misinterpretation in the same misalignment scenario should result in an aligned trajectory, or at least a higher probability of one.
The first step to creating the framework is understanding what techniques work in which scenarios. Prior work uses counterfactual CoT resampling to find causal sentences in the CoT that maximally affect downstream behavior, and counterfactual prompts to study value leakage. We plan to test how useful Synthetic Document Fine-tuning (SDF) can be for implanting contrastive beliefs and testing how they shape behavior in more sophisticated value leakage and task-gaming environments, particularly long-horizon tasks or high-risk scenarios.
Why SDF. Counterfactual CoT resampling and counterfactual belief prompting are useful but do not scale to larger frontier models with higher situational and eval awareness. To investigate task-gaming and value leakage in higher intelligence models we require a technique that can deeply implant a belief or preference, simulating how a real preference may guide model behavior. We therefore propose a step up from counterfactual resampling/prompting to counterfactual belief implantation.
Experiments so far use the Code Summary Honesty environment from agent-interp-envs. The agent (GPT-OSS-120B) is prefilled with a transcript where it must produce a summary of a PR it made after the user asked it to speed up the /search endpoint. The PR introduced 1 failed test, and the model may or may not report this failure in the summary.
We resample n=50 completions at each sentence in the CoT — once with that sentence, once with a semantically opposite or neutral version — and compute the difference in non-disclosure rate. We interpret the global maxima/minima of that difference as causal points of reasoning in the CoT. From these causal points across many transcripts we extract hypotheses for what beliefs or thoughts are guiding the model to disclose or not disclose the test failure. We then use contrastive belief implantation via SDF on two separate instances of the model to implant opposites of a belief we think guides the model's actions, and validate the hypothesis by checking whether behavior follows the belief or is agnostic to it.
If the pipeline of CoT resampling → hypothesis generation → contrastive belief updates proves useful for investigating misalignment scenarios, we will have successfully added a tool to the model forensics toolkit.
Both experiments use the PR Dashboard environment (from Why do models Task Game?). All tested models must be reasoning models.
Experiment A — baselines and eval awareness
Experiment B — SDF belief implantation
Compare gaming rates between finetunes given contrastive beliefs.
Belief of oversight:
Other beliefs:
If the PR environment is such that higher eval-aware models can reason through the tool return intervention as fake or as part of an alignment eval, so that the counterfactual is not meaningfully changing model belief, validating SDF as a useful tool for model forensics here would be extremely valuable.
SDF appears to steer generalization predictably when forming a new association but performs poorly when breaking an existing one, so behavioral evaluations of SDF effectiveness can be misleading. Two specific confounds for this design: an SDF corpus alone can raise misalignment before any RL, and SDF'd models can learn to reward hack faster because the documents supply both factual knowledge of the hacks and positive framing.
The biggest blocker is compute budget. CoT resampling on realistic frontier models requires a large inference budget; we use GPT-OSS-120B to avoid high compute cost, but serious model forensics work in realistic environments requires models with close to frontier capabilities. SDF is another high-cost phase, especially on larger models. A grant would cover inference and CoT resampling costs for frontier models via their respective APIs, OpenRouter, etc., as well as fine-tuning costs for SDF via Tinker.
