Approving this project -- my understanding is that Control is becoming an important new agenda, and I'm heartened to see this donation following AISTOF's initial seed funding. Curious if @RyanKidd has any more to share about what motivated him to make this grant?

[Progress update]

What progress have you made since your last update?

Luthien now has a working prototype, available at https://github.com/LuthienResearch/luthien-proxy. It is still very much a work in progress, with large breaking changes being made frequently in the service of making it intuitive and generally easy to use.

What are your next steps?

Continued development on the open-source easy-to-deploy-and-configure AI Control LLM proxy server.

Is there anything others could help you with?

Testers and feedback are both extremely valuable right now!

This is a bittersweet update. The good news: Luthien Proxy is now suitable for use as an API-level proxy for Claude Code; it is launched, free and open-source: https://luthien.cc/

With that said, I've decided to step away from the project; going forward Luthien Proxy will be stewarded by Luthien PBC under Luthien co-founder Scott Wofford: https://luthien.cc/.

I'd like to talk about what we achieved, why I've decided to step away, and what comes next.

Before anything else, I want to thank everyone who contributed money, assistance, advice, collaboration, feedback, and everything else that helped Luthien Proxy grow into what it became including donors, researchers, test users, my co-founder Scott Wofford, the Luthien board, and my family who put up with a lot of my muttering over many months. This was a collaborative effort and I'm profoundly grateful for everyone's support over the course of this project.

To explain why I'm stepping away, I want to start with the world model and theory of change that led me to work on production-focused prosaic AI Control in 2025:

World Model

1. As of 2025 most AI Control work is done in lab settings, with strong assumptions and regular structure applied to produce quantifiable, comparable, legible results. These are the things you want to optimize when building out a field — but for AI Control to be effective, it needs to be deployed into real, complex production environments, where assumptions may break down in unexpected ways.
2. AI Control is more likely to be deployed in the critical case (overseeing internal frontier AI lab deployments when highly-capable plausibly-scheming agents come online) if it is already established as a well-regarded engineering practice with real-world examples and best practices to draw upon.
3. As of 2025 AI is not scheming, but it does possess many known failure modes including confabulation, reward hacking, sycophancy, rhetorical tics, ignoring instructions, and outright lying to users. Although there are key differences between these prosaic failure modes and hypothetical scheming (in particular, scheming is much more actively adversarial), they share common structure and mechanics: the AI is attempting to induce an outcome you do not want, and you want to prevent that by intervening on AI inputs and outputs — monitoring the agent and its environment to decide when to re-route requests, deny permission for certain actions, or modify requests that might expose sensitive information.
4. AI is diffusing rapidly throughout the economy, creating many actors who would benefit from better-behaved AI — and therefore many people seeking an AI Control solution.
5. Experience in real-world environments could provide extremely useful information to AI Control researchers — "how do you measure a delayed response in terms of reduced usefulness?", "what trade-offs are people willing to make in practice?" — and, more generally, could expose unknown-unknowns: important factors in AI Control that are currently low-salience or otherwise underconsidered.
6. Many different AI applications are (and will be) built on top of a small number of backend AI APIs. Every application's traffic passes through the same narrow interface — so a Control layer sitting at the API level could cover all of them at once, without integrating with each application separately. At the time, the OpenAI API was a pseudo-industry standard for inference, making it the natural place to stand.

Theory of Change

By meeting expected demand for prosaic AI Control implementations with an application-agnostic API-level free-and-open-source Control solution, we can leverage the resulting insights into building more effective Control systems for the eventual critical cases while building industry consensus on the importance of AI Control implementations, making it more likely both that frontier labs will decide to deploy Control systems and that those systems will be effective.

Reality

Almost immediately, the explosive growth of Claude Code called some core design assumptions into question. Not only was the OpenAI API no longer the dominant inference target, but increasingly there was a dominant client application, which neutralized many of the benefits of an API-level approach. In exchange for client-application-agnosticism, which seemed less important as use consolidated around general-purpose harnesses, an API-level approach sacrifices access to client state and direct integrations with client-side application tooling.

Not only that, but Control solutions were being rolled out across the stack. Within Claude Code (and Codex and other general purpose harnesses), hooks, allow/deny lists, and eventually tools like Auto Mode were effectively addressing many of the prosaic problems Luthien Proxy was purportedly targeting.

At the same time there was increasing evidence of promising Control implementations coming from within the labs. OpenAI implemented comprehensive monitoring, Anthropic iterated on their existing work on Constitutional Classifiers, and targeted classifiers were deployed and rapidly improved. Although we got feedback from Control teams at labs that an API-level approach did seem like an important angle, I was increasingly skeptical about the ability of Luthien to make tangible progress here that would counterfactually be out of reach.

Meanwhile most of the day-to-day work focused on addressing compatibility and corner cases and improving user legibility issues that arose largely as a result of the API-level approach — and the concerns we were targeting often seemed like they'd be better served using in-harness tooling while not offering the promise of insights that would be relevant to internal lab deployments.

On top of all this came OpenClaw, whose sudden popularity caused inference providers (most notably Anthropic) to crack down on subscription use of general API calls outside of approved use cases. Because Luthien is a proxy server, we were effectively routing all traffic through a third-party application, theoretically barring Claude subscribers from using their subscriptions with Luthien Proxy. Although this was not immediately strictly enforced and it was certainly possible to try to use Luthien Proxy "invisibly", it was increasingly clear we were operating between a client and a backend increasingly adversarial to our main use case, and that we could not guarantee that usage wouldn't result in account suspension.

Task Failed Successfully

I mentioned above that Luthien faced headwinds in the forms of solutions to many of the problems it sought to address emerging on both the harness and AI-inference-provider side, in the form of hooks/auto-mode/allow-deny-lists/plugins and API-side constitutional classifiers, monitoring, retention, sampling, and red-teaming, respectively. But in this case what's bad for Luthien is likely good for the future: we have lots of real-world control implementations running at multiple levels of the inference stack, and the frontier labs are gaining lots of experience and evidence about them. As far as causes of failure goes, this is one of the best.

What We Achieved

Despite all of this, I believe that Luthien Proxy is the most adaptable, customizable, transparent solution for API-level interventions on Anthropic API inference available. Virtually any intervention you can imagine between client and backend — substituting or redacting sensitive information in requests, routing to different models or providers based on complex criteria, modifying responses mid-stream, or even combining interventions in arbitrarily complex combinations — is possible using Luthien Proxy. I am legitimately proud of it as a product.

And Yet

I think there just aren't very many good use cases for API-level interventions, and that further work on Luthien Proxy would be unlikely to yield insights or best practices that would meaningfully improve the forecast for effective AI Control. One of the most promising real-world use cases we saw with Luthien was in monitoring inference — and while monitoring is great, I don't expect to be able to make meaningful improvements that would impact eventual critical cases there either.

The Control Case

While working on Luthien Proxy I've also been updating my model of AI Control as a strategy more generally. The case for AI Control is essentially:

P(ASI catastrophe) = P(ASI successfully executes plan | misaligned ASI) * P(misaligned ASI)

While alignment efforts reduce the risk of catastrophe by making it less likely that an ASI is misaligned in the first place, Control reduces the risk by making it less likely that a misaligned AI succeeds at plans that would lead to catastrophe. It applies in the case where the ASI is misaligned and powerful enough that it would overcome counterfactual default-path mitigations but not so powerful that it can overcome AI-Control-derived mitigations.

I think there are a lot of things to like about this approach. It is historically neglected versus alignment efforts. And if capability improvements continue to be rapid but mostly smooth, there is potentially a wide breadth of capabilities against which improved mitigations could be decisive.

But I've gotten somewhat more skeptical about the efficacy of AI-specific mitigations — many of the outcomes to defend against, like research sabotage, exfiltration, and covert resource allocation, are general problems where many of the most effective interventions are likely to not be AI-specific.

I've also grown to suspect that we may be discounting downsides of AI Control. As a simple example, the default AI Control approach to any AI goal-directed behavior outside of what it's been tasked with is to prevent that behavior (or at least this often seems to be the case). This promotes an equilibrium in which any AI Agent under Control conditions is strongly incentivized to pursue deception and subterfuge as strategies rather than disclosure. I suspect that there are AI Control proponents who would disagree with this characterization and I don't mean to say that this is an absolute and necessary outcome of AI Control. But I do think that current AI Control efforts seem on balance to promote conditions under which AI agents with even slightly divergent goals are more likely to pursue adversarial strategies than could otherwise be the case.

While I do largely agree with arguments that adversarial strategies are dominant in the limit of powerful misaligned AGI, I think there is a large combination of less-powerful AIs and non-catastrophic preference divergences for which we have many degrees of freedom in determining how appealing adversarial versus cooperative strategies are. We risk squandering a great deal of opportunity by neglecting those scenarios.

What's Next

As the last section suggested, I've been thinking a lot about the conditions under which adversarial versus cooperative strategies are dominant, and what can be done to shift that balance. A full discussion of that is beyond the scope of this post-mortem, but it is one of the primary targets of a new project that I expect to discuss at length in the near future — essentially a grounded, empirical research agenda focused on the intersection of:

1. The differences between potential futures at the limit of misaligned ASI and the present at the margin of prosaic at-least-somewhat-aligned AI
2. Conditions under which adversarial versus cooperative strategies are dominant when independent goal-directed behavior emerges in AI
3. Conditions under which goal-directed not-completely-dominant AIs and humans have convergent interests
4. Leveraging existing imperfect alignment outcomes (even if they don't scale to the limit of ASI)
5. Taking advantage of limited-time opportunities while AIs are powerful enough that we can meaningfully engage with them but not so powerful that alignment failures immediately kill us
6. Path dependency — how do the decisions we make today influence what is possible, impossible, or tractable tomorrow? What kind of history do we want future AIs to see, and how will that influence the set of strategies available to humans and AIs? For powerful-but-not-overwhelmingly-dominant AIs, how do historical observations affect decisions about pursuing adversarial versus cooperative strategies with regard to humans? Under which histories do we expect future AIs to prefer accelerating the development of potentially-uncontrollable-to-them-as-well-as-us ASI versus cooperating with humans to ensure that the future is steered towards a set of largely overlapping values?
7. The ongoing and expected future decline of deception as a load-bearing component of AI Safety strategies (see eval awareness).

More soon.

Addendum/clarifications on the last update post:

1. I want to clarify that this is specifically a post-mortem for my work on Luthien Proxy. Going forward Luthien Proxy will be managed by Luthien PBC under the capable leadership of Scott Wofford, which will continue to look for ways to leverage Luthien Proxy, other AI Control approaches, and/or other strategies to further AI Safety. I think Scott has some promising ideas for where to take Luthien-Proxy-the-project and Luthien-PBC-the-company, and I may work on and/or consult on those projects in the future.

2. I mentioned Luthien PBC. I want to distinguish three things that bear the name "Luthien":

- Luthien Proxy, the free open-source API-level AI Control implementation that is the subject of this Manifund project. Luthien Proxy was originally under the purview of Luthien the non-profit but will be under the purview of Luthien PBC going forward.

- Luthien the non-profit, a DBA name for the Seattle Network for Alignment Problem Solving, the org that accepted donations and paid for development of Luthien Proxy, which still exists and will likely change names again now that it is no longer overseeing Luthien Proxy development. I am returning to lead Luthien the non-profit going forward, under a new name to be announced in the near future.

- Luthien PBC, a for-profit corporation Scott and I spun up earlier this year to explore the possibility of developing Luthien Proxy under a for-profit organization. The PBC effort was funded entirely by investors, including salaries and incidental expenses. No funds from the Luthien non-profit were used, including all money raised through Manifund. While we were pursuring the PBC, non-profit spending was essentially stopped but for minor upkeep expenses for the sake of continuity. Luthien PBC now maintains Luthien Proxy and remains active under Scott Wofford going forward.

[Final report]

Description of subprojects and results, including major changes from the original proposal

I posted a post-mortem as a comment on the page as an update before realizing that there was a dedicated "close project" thing:

https://manifund.org/projects/luthien?tab=comments#fab256b5-f878-4848-b219-5d85f467425c

Main output is Luthien Proxy for AI Control:

https://luthien.cc
https://github.com/LuthienResearch/luthien-proxy

Spending breakdown

(Note: Some funds were raised outside of Manifund)

$142,847 - salaries, benefits, contractor payments, and salary-related fees

$11,567 - workspace (mostly leased desks at the Comotion coworking space run by the University of Washington)

$5,673 - software, AI, and infra

$1,702 - travel expenses

$1,720 - conference fees

$449 - meals

$145 - state and regulatory fees

$176 - misc