Epistemic memory for AI agents: making the safe path profitable

A memory system for AI reasoning agents that aggregates different sources of information while keeping track of relationships and confidence levels. This enables reasoning over longer tasks without forgetting or goal drift.

Project Details

I am building a harness for AI agents that increases productivity while having alignment benefits. This ensures that the alignment-relevant features will be adopted, because they are directly relevant to the user's profit. I want to make sure that the safe path forward is also the profitable path forward, so that it will be taken by default.

The harness is a memory system that keeps track of the epistemics behind the data, to reduce hallucinations, content drift, and goal drift, and to make the system easily auditable. It is useful for reasoning, data analysis, and coding tasks, especially for larger, legacy software projects with many components and contradictory documentation, which are tasks that current coding agents have trouble with.

Context: I am already working on this research, funded by Foresight Institute. However, the existing funding only covers the cost of my experiments, not my own salary. I am testing it with industry clients during my day job as an AI consultant. These tests give valuable practical feedback and prove that the system works as intended, but progress would be much faster and more scientifically rigorous if I could make the research the focus of my work. For that, I need to be independent of client work.

What's already done:

The core of the harness is a memory structure that mimics the split between procedural and declarative knowledge in the brain, and allows an agent to recall either on demand. This system is in daily use on a client engagement, by me and coworkers who adopted it. It is also used by mentees of a separate alignment project I mentor at SAIGE. The system is lightweight and I am using it with both Claude Code and Copilot. It actively curates the memories it generates.
The harness led to notable improvements for coding: My agents rarely make the same mistake twice, even when there are weeks between tasks. They know how to look up solutions for known problems. They know decisions we made weeks ago.
Agents can also efficiently look up the reasoning processes behind our decisions whenever it is relevant. They know not just what we are doing, but why, what the alternatives were, and why they were discarded. They can automatically discover when old decisions were made based on outdated information and flag them to me for review and reconsideration.
I have created tooling to make retrieval more reliable, with checkable sources. Agent memories carry source attribution, which reduce confabulation and make them easily auditable.

All of the above features are a work in progress, but already show promising results in practice. The following features are yet to be implemented:

More systematic tests on benchmarks and A/B testing of different features. These are expensive to run and impossible to do in parallel with client work.
Multi-model memory sharing: A single harness that works for multiple agents of different model families and enforces shared transparency is more robust. If only some models are misaligned, the others are likely to catch those agents before they can do damage. Even steganography gets harder if other model families might paraphrase each others' outputs. This is defense in depth, enforced by the architecture. At the same time, using multiple model families in a single system is also more cost efficient (so it increases the chances of adoption). It also avoids the danger of putting all our eggs in one basket: if the company that currently provides the most trustworthy models gets in trouble of any kind, we need to be able to pivot and make do with whatever other models are available. AI Alignment should not hinge on a single company.
Integrate Deliberative Credit Assignment (DCA): This connects to my prior work on DCA, developed with mentees at SPAR. DCA trains models to reason faithfully rather than produce plausible-sounding outputs. The intent here is to train a public model we have whitebox access to, using DCA, to act as a trustworthy orchestrator of the overall system, optimized for faithful reasoning, epistemic correctness and auditability.

Theory of Impact

Once all planned features are implemented we would end up with a system that:

is auditable, avoids hallucinations, and keeps track of sources
is under the overall control of an agent that is open source and optimized for safety (using DCA)
is model agnostic and can make use of whatever is the best coding or reasoning agent on the market

The resulting system will be both profitable and safe, and the safety aspects will be an integral part of the architecture, so people can't just build a less safe, more profitable version of it.

This makes the safe path forward equal the profitable path forward for companies, which is the incentive structure you want in an alignment project that actually gets used instead of just cited in papers but never deployed.

People

Florian Dietz

Team Member

Grants Received

Add grant

Foresight InstituteAmount hidden

from Foresight Institute

$****

Funding Asks

grantmaking.ai Launch Round

Applied

I'm building a memory system for AI reasoning agents that aggregates different sources of information while keeping track of relationships and confidence levels. This enables reasoning over longer tasks without forgetting.

Minimum

$8,000

Ideal

$54,000

What this ask is

What's already done:

- The core of the harness is a memory structure that mimics the split between procedural and declarative knowledge in the brain, and allows an agent to recall either on demand. This system is in daily use on a client engagement, by me and coworkers who adopted it. It is also used by mentees of a separate alignment project I mentor at SAIGE. The system is lightweight and I am using it with both Claude Code and Copilot. It actively curates the memories it generates.

- The harness led to notable improvements for coding: My agents rarely make the same mistake twice, even when there are weeks between tasks. They know how to look up solutions for known problems. They know decisions we made weeks ago.

- Agents can also efficiently look up the reasoning processes behind our decisions whenever it is relevant. They know not just what we are doing, but why, what the alternatives were, and why they were discarded. They can automatically discover when old decisions were made based on outdated information and flag them to me for review and reconsideration.

- I have created tooling to make retrieval more reliable, with checkable sources. Agent memories carry source attribution, which reduce confabulation and make them easily auditable.

All of the above features are a work in progress, but already show promising results in practice. The following features are yet to be implemented:

- More systematic tests on benchmarks and A/B testing of different features. These are expensive to run and impossible to do in parallel with client work.

- Multi-model memory sharing: A single harness that works for multiple agents of different model families and enforces shared transparency is more robust. If only some models are misaligned, the others are likely to catch those agents before they can do damage. Even steganography gets harder if other model families might paraphrase each others' outputs. This is defense in depth, enforced by the architecture. At the same time, using multiple model families in a single system is also more cost efficient (so it increases the chances of adoption). It also avoids the danger of putting all our eggs in one basket: if the company that currently provides the most trustworthy models gets in trouble of any kind, we need to be able to pivot and make do with whatever other models are available. AI Alignment should not hinge on a single company.

- Integrate Deliberative Credit Assignment (DCA): This connects to my prior work on DCA, developed with mentees at SPAR. DCA trains models to reason faithfully rather than produce plausible-sounding outputs. The intent here is to train a public model we have whitebox access to, using DCA, to act as a trustworthy orchestrator of the overall system, optimized for faithful reasoning, epistemic correctness and auditability.

How the money will be spent

The minimum is an estimate for running more rigorous evaluations that I can not conduct in parallel with other work. The maximum is estimated as: eval $8k + ~3 months part-time salary $12k + minimal DCA orchestrator (API+finetune+inference) $10k

Discussion

No comments yet. Be the first to share your thoughts.