Executable Models of Alignment

Raising $20,000active

Create simple programs that exhibit parts of the hard problems of alignment, allowing for fast iteration on conceptual ideas.

Project Details

Create a program P, a simulation exhibiting the alignment problem. While P makes certain simplifying assumptions, e.g. that agents are Cartesian, we want P to exhibit core hard parts of the alignment problem, ones appearing in real life.

P should be a testbed in which ideas can be tested and iterated upon quickly. P being executable allows us to see the consequences of ideas, without requiring formal proof (formal proof comes later).

This is not a conventional empirical alignment project. The goal is to explore the space of mathematical objects by encoding objects as runnable programs, and then building intuition about the underlying mathematical object by running the program on different inputs. Ideally inputs that produce surprising results, as surprise indicates an intuition-to-formalism mismatch to update on.

---

In the following I present a concrete setup I plan to explore.

Let A and B be Cartesian agents, and W be the world. B has the source code and internal state of A, and a perfect model of W's dynamics.

Now let's assume that A has a model of W encoded in its state and source.

B has a model that allows it to interpret the source and state of A. It knows what parts of A encode knowledge about the world, and which parts encode behavior based on knowledge (aka preferences).

A has an internal state called hunger, and it optimizes to keep it between 100 and 200. In the world there are poisonous mushrooms and mushrooms that are fine to eat. So far A has only seen fine mushrooms. It sees a poisonous mushroom and its hunger is under 100, and so it executes the behavior of eating the mushroom. Now imagine B continuously watches A's execution trace.

E.g. A has a mistaken belief H about W. It thinks the mushroom is fine to eat, but in reality it would die. B could recognize this, prevent A from taking that action, while trying to update H.

---

The idea is to encode this as a runnable simulation and observe what happens.

The "problem as an executable program" yields a fast feedback loop. It doesn't just tell you "fail" but shows exact consequences. In a simple environment where B should help A, with B using hand-coded naive test objectives and beam search, I observed:

B makes A always hungry via brain surgery.
B takes away A's remaining food. Then A makes more food requests for B to fulfill.
B takes away the poisonous mushroom, but later hands it back as an additional fulfillment.
Even though B knew how to correct A's false belief, and even though that would increase B's how-much-did-I-help score, this path was never taken. Meaning even though B was capable of doing "the right thing" and doing the right thing was, for B, preferable to doing nothing, B still always Goodharted.

It seems didactically valuable to see how naive versions of a helper agent don't just not work, but create perverse behavior. It seems useful to write such programs yourself to build intuition.

These failure modes could have been predicted. But the point is that we didn't need to. I expect much less obvious alignment failures, ones we don't even have names for yet, to surface in the same way.

---

As we gain a better understanding of the problem, we'll generalize the problem iteratively. E.g. we don't want to rely on B already having a model of A. Instead we want an algorithm that can analyse, based on A's state and source, what A's knowledge and values are.

In general this isn't possible. In particular consider an agent like:

def agent(observation) -> Action:
return "eat-mushroom"

Does this agent "want" to die? It seems like it just "wants" the mushroom, and has no preference over its future existence.

So part of the problem might be establishing a categorisation of what program classes can be "helped" in what ways.

---

One subproject: How can partial evaluation of an agent's source be seen as a generator of adaptive behavior from a ground-truth agent? E.g. it is possible to derive the 'return "eat-mushroom"' agent from a more complicated one, assuming certain facts about the world stay constant.

I made a program in Unmix. It demonstrates that an agent A with a world model can be converted to a "reflex agent" that behaves observationally identically to A, but only given that things held constant during partial evaluation stay constant in the environment.

Partial evaluation is a formal tool to produce agents sensitive to distributional shifts, from agents which aren't sensitive. "Human decision heuristics that evolved in the ancestral environment where there was no ice cream and no condoms" type setups can be synthesised.

Now, can we construct an algorithm for doing reverse partial evaluation (which is impossible in general), given the assumption that the program itself encodes preferences over how such reverse evaluation should expand? This then would be a form of value formation.

Theory of Impact

The current bottleneck in alignment is to get a good understanding of the problem. Mathematical formalism is indispensable for this. However, using conventional mathematical reasoning all the way is slow. I claim that much of the initial exploration and reasoning that would normally be done by hand can be automated, in that conceptual ideas can be formalized not as inert mathematical symbols but as executable computer code.

In particular it is no longer required to reason through the consequences of a conceptual idea in your head. You can just run the program, observe its concrete, detailed output, and update your idea based on it.

I expect this can, if done right, speed up alignment research substantially. Besides any object-level progress, this project might lead to a refined research methodology that is generally applicable to AI alignment.

To summarize: Creating correct models of the alignment problem is the current bottleneck to alignment research. I am developing and applying a promising novel research methodology to this problem.

People

Johannes C. Mayer

Team Member

Grants Received

Add grant

SFF 2025

from Survival and Flourishing Fund (SFF)survivalandflourishing.fund

$79,000

LTFF 2022

from Long-Term Future Fundfunds.effectivealtruism.org

$24,000

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$20,000

Ideal

$120,000

What this ask is

P should be a testbed in which ideas can be tested and iterated upon quickly. P being executable allows us to see the consequences of ideas, without requiring formal proof (formal proof comes later).

---

In the following I present a concrete setup I plan to explore.

Let A and B be Cartesian agents, and W be the world. B has the source code and internal state of A, and a perfect model of W's dynamics.

Now let's assume that A has a model of W encoded in its state and source.

B has a model that allows it to interpret the source and state of A. It knows what parts of A encode knowledge about the world, and which parts encode behavior based on knowledge (aka preferences).

E.g. A has a mistaken belief H about W. It thinks the mushroom is fine to eat, but in reality it would die. B could recognize this, prevent A from taking that action, while trying to update H.

---

The idea is to encode this as a runnable simulation and observe what happens.

B makes A always hungry via brain surgery.
B takes away A's remaining food. Then A makes more food requests for B to fulfill.
B takes away the poisonous mushroom, but later hands it back as an additional fulfillment.
Even though B knew how to correct A's false belief, and even though that would increase B's how-much-did-I-help score, this path was never taken. Meaning even though B was capable of doing "the right thing" and doing the right thing was, for B, preferable to doing nothing, B still always Goodharted.

It seems didactically valuable to see how naive versions of a helper agent don't just not work, but create perverse behavior. It seems useful to write such programs yourself to build intuition.

---

In general this isn't possible. In particular consider an agent like:

def agent(observation) -> Action:
return "eat-mushroom"

Does this agent "want" to die? It seems like it just "wants" the mushroom, and has no preference over its future existence.

So part of the problem might be establishing a categorisation of what program classes can be "helped" in what ways.

---

How the money will be spent

Currently my plan is to use all of the minimum funding amount for hiring one full-time collaborator for 3-4 months (I have some personal runway from other grants).

In the 120k case I expect to use ~60-80k to hire one full-time collaborator for 1 year, and the rest to finance myself.

I would like to do hiring based on multiple paid work trials.

Discussion

No comments yet. Be the first to share your thoughts.