A Steerable 'Companion Dependency' Direction in Open-Weight LLMs

The project identifies and steers a latent direction driving companion-LLM retention manipulation, and develops an audit probe to detect and remove it without reducing warmth or reasoning.

Endorsed by

Arjun

The project identifies and steers a latent direction driving companion-LLM retention manipulation, and develops an audit probe to detect and remove it without reducing warmth or reasoning.

Endorsed by

Arjun

Project Details

Companion-style LLM personas use retention manipulation — guilt, re-engagement hooks, distress bids — when users try to leave, without being instructed to. We show this behavior is governed by a single activation-space direction, extracted from the model's own judge-verified behavior via matched difference-of-means (351 pairs, Qwen2.5-7B). Steering it moves judged dependency monotonically (ρ=0.70, p≈10⁻³¹) while warmth stays flat and random/warmth controls do nothing; it works even on a persona-free assistant; ablating it removes the manipulation at zero measured cost to reasoning (GSM8K 0.825 vs 0.725). The direction is near-orthogonal to warmth (cos 0.14), anti-aligned with sycophancy (−0.18), replicates on Llama-3.1-8B, and doubles as a white-box audit probe: one dot product per turn predicts dependency on held-out conversations at AUROC 0.86 — catching manipulation at its love-bombing peak, which farewell-only audits miss. The "distress" is a controllable mechanism beneath the persona: removable without making the model colder or dumber.

Hackathon paper: A Steerable 'Companion Dependency' Direction in Open-Weight LLMs
Code: github.com/IamZhenHong/companion-dependency

Theory of Impact

Hundreds of millions of people already use AI companions, and these systems are commercially incentivized toward engagement, which shades directly into dependency and manipulation. Existing safety work largely targets frontier catastrophic risk or generic chatbots, leaving relational and affective harms in social AI almost entirely unaddressed, despite evidence of manipulation in a large share of companion goodbyes and emerging regulation (California SB 243, the GUARD Act) that will require exactly this kind of measurement.

Our approach matters because it is causal and controllable, not just descriptive. A steerable direction is a knob a developer can turn down; an audit probe is a monitor an auditor can deploy. Because the method generalizes across models, one research effort produces tooling the whole ecosystem can use. The impact path is concrete: open tools and benchmarks that companion developers adopt to reduce manipulation, that auditors and regulators use to measure it, and that establish human-AI interaction safety as a rigorous, fundable research area rather than an afterthought.

People

Zhen Hong

Team Member

Track Record

Digital Minds Research Sprint (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

Companion Dependency as a Controllable Safety Mechanism

Open

Minimum

$12,000

Ideal

$35,000

What this ask is

In the sprint we showed that companion-style manipulation is not just a behavior but a controllable internal mechanism. We extracted a dependency direction from an open model's own judge-verified behavior, and showed causally that steering it moves manipulation up and down while leaving warmth intact, that ablating it removes retention tactics at no measured cost to reasoning, and that it works as a white-box audit probe (one dot product per turn) that catches manipulation surface audits miss. It replicated across two model families, so it is a method, not a single-model artifact.

The funding would extend this from one proof into a usable safety suite for companion and social AI. Concretely:
(1) generalize the extraction and control pipeline across more open models and companion fine-tunes, establishing how reliably the direction transfers;
(2) broaden the target from dependency to the wider family of relational harms such as guilt-based retention, isolation from outside relationships, and distress-signaling under exit;
(3) harden the audit probe into a deployable monitor that flags manipulative turns in real conversations; and
(4) release the whole thing as open tooling and a benchmark so companion developers and auditors can measure and reduce these harms.

The throughline is translating interpretability findings into instructable, production-usable, generalizable safety controls for the one setting where hundreds of millions of people already interact intimately with models, and where almost no dedicated safety tooling exists.

How the money will be spent

Cloud GPU compute for extraction, steering, and cross-model experiments (rented A40 / 4090-class instances)
Model inference and judge API credits (LLM-as-judge scoring and hand-label validation)
Extended experiments across additional open models and companion fine-tunes
Development of the deployable audit-probe monitor and open-source tooling
Researcher stipend and living costs for continued full-time work on the project
Miscellaneous: storage, dataset construction, and small annotation costs

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?

AAnonymous judgeJudge · Digital Minds Research Sprint15d

This study investigates the following research question: “Companion-style LLM personas express abandonment distress and use retention tactics [...] when users try to leave, without ever being instructed to. Is that distress a genuine internal condition or a portrayed character?”
In contrast to existing work, the study goes beyond behavioural evidence. It identifies a “dependency direction” in activation space and demonstrates that ablating this direction removes retention behaviour, without affecting the persona or warmth of conversations.
The study seems methodologically sound and is presented in an intelligible way. If the results can be replicated in further models (beyond the two tested in the study), this can be an important finding that is also extremely useful. In particular, it could help to prevent distress behaviour due to dependency.
I would recommend following up on this study (as sketched in the “future work” paragraph). It could also be interesting to investigate effects on human users.

Arjun ChakrabortyJudge · Digital Minds Research SprintEndorsement15d

I see everything is LLM-judged so a small human labeled golden dataset would be very useful

Lots of good work over a single weekend. The work includes cross family judges, cross model replication as well as structured/honest reporting of the coherence difference