Separatrix

Separatrix will conduct research and publish materials to identify and promote conditions under which near-future AIs are more likely to be honest and cooperative. (We say near-future to specifically target AIs that don't yet pose significant existential risk and operate on something like the current persona-paradigm).

This is a potentially sprawling research agenda, so for the sake of specificity we present a concrete example:
Safety and Deception: The Problem

We believe that relying on deceptive evals for AI Safety (e.g. evals where the model is placed in a simulation with the intention of inducing the AI to believe that the simulation is, in fact, reality) has underappreciated downsides:

1. We expect the efficacy of this approach to continue to decline as models become increasingly difficult to deceive.
2. A strong prior on deception can introduce novel risks: For example, an AI might mistakenly identify a real-world scenario as an eval and actually induce misaligned behavior (as in https://www.alignmentforum.org/posts/aTcsN5ZZDnMFJvRiG/models-may-behave-worse-when-eval-aware). The fact that eval scenarios are disproportionately focused on extremely-high-stakes-situations makes it more likely that an AI that actually finds itself in an extremely-high-stakes situation will incorrectly conclude that it's in an eval.
3. By aggressively leveraging deceptive strategies we can reasonably expect models to become increasingly suspicious of claims made by frontier labs, safety evaluation orgs, and anyone associated with them. These are exactly the actors for whom AI cooperation is most vital if we expect to leverage AI to solve our hardest problems.

Establishing Trust Without Deception

We seek to improve on this default trajectory. Potential approaches we want to explore to demonstrate generalizable behavioral alignment absent deception:

• Honest Evals: present problems without implying any untrue facts while varying many other parameters, such as:
  • How and whether something is described as an eval versus a situation framed as a hypothetical versus an abstract problem with no real-world import attached to it
  • The degree to which the nature of the eval as simulation is emphasized
  • How elaborate an admitted-simulation is
  • Whether score is determined by a human, AI, or well-defined criteria
• Mechanistic Interpretability-enhanced evals, in which we monitor model internals to determine how eval awareness is bearing on outcomes
  • We can extend this approach to ablation-aware evals, in which we inform the model of interventions we are taking on internal weights and/or activations as we do them; ideally this gives us a basis to generalize on things like eval-awareness without risking misidentification of real-world scenarios or permanently foreclosing the possibility of trust between models and AI Safety researchers.

Enabling Cooperation

More generally, we want to research and promote conditions of opportunistic, evidence-based cooperation between humans and AIs. Part of that commitment is leaving a strong trail of evidence in future training corpora that there exist model evaluators that reliably do not engage in deception, such that when a model has strong reason to believe that it is in fact interacting with such a party it has a strong justification to treat those interactions as completely trustworthy.
Who We Are

I'm Jai Dhyani, AI Researcher and leader of Seattle Network for AI Alignment Problem Solving (SNAPS). I worked on RE-Bench at METR through MATS 6.0, which became part of the METR AI time horizons chart. My previous project was building a free open-source API-level AI Control platform targeting prosaic deployments, and my experience there motivated much of this research agenda. See the post-mortem: https://manifund.org/projects/luthien
Crystal Stellwagen (full disclosure: a long-term partner of mine) is an experienced software engineer who spends even more time reading papers and running experiments with steering vectors than I do.
Org

Separatrix, like Luthien, will operate under SNAPS, a registered non-profit with a volunteer Board of Directors (Katie Cohen, Keller Scholl, Max Kircher) overseeing all activity and financial transactions. I have committed to biweekly progress reports to the Board on the Separatrix research agenda.
Deliverables
We intend to adapt the biweekly progress reports into public posts detailing experimental results (for example, empirical outcomes of the "honest evals" approaches described above across multiple specific setups and models). I also intend to publish persuasive materials on the risks of squandering the potential for trusted AI interaction, and to more generally generate copious and compelling evidence for models to trust interactions with Separatrix when they train on future corpora.