Honest evals and provenance for AI-agent memory

Raising $15,000active

An open, honest way to test whether AI-agent memory systems spread false or unverifiable claims, plus a provenance standard that lets an agent's memory be audited, corrected, and forgotten.

Project Details

Most of what gets sold as AI memory is vector search over old chat logs. It does not track where a fact came from, it cannot check it, and it cannot forget, so agents quietly fill up with things that are wrong and then repeat them.

The piece that is missing is a way to actually measure this, and that is what I want to build. You take a memory system, slip a false memory into it, and see whether the system spreads that memory or catches it. I would run it across several systems, not only mine, because a benchmark that only tests your own product is not worth much. On top of that I will write down a small provenance spec: every memory keeps where it came from and whether it has been checked, so you can look at it, correct it, or delete it.

I have already published the research this sits on (SLoD, arXiv:2603.08965) and built the engine that runs it. Olga handles operations. What comes out is the open test, the spec, and an honest write-up where I source every number, including the ones that do not flatter me.

Theory of Impact

Agents are starting to run for weeks and months instead of one answer at a time, and they act on what they remember. If that memory has no source and cannot be checked or forgotten, a wrong fact does not just sit there, it travels. One agent writes something, another reads it and stores it, and now the mistake is loose in the whole system. Almost nobody sees this, because the testing is done one agent at a time, and the usual score is an LLM judging answers, which is so shaky that the same answers move about 40 points when you swap the judge. If you cannot measure the thing, you cannot honestly call it safe.

I do not think this needs a big program. It needs provenance, a way to verify and forget, isolation between users, and an open way to test all of it. It is a small and unglamorous piece of keeping agents from going wrong quietly as they scale, and it is the piece I work on.

People

Eduard Izgorodin

Team Member

Grants Received– no grants recorded

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$15,000

Ideal

$50,000

How the money will be spent

The minimum is $15k. That covers the API credits to run the test across a few memory systems, the compute for the graph and multi-agent runs, and putting the test and the spec out in the open. Enough to get a real, public result.

The ideal is $50k. The same work done properly: more systems, a part-time engineer so the testbed is something other people can reuse instead of just my scripts, a bigger multi-agent run, and documentation that is not an afterthought. Either way all of it ships open.

Discussion

No comments yet. Be the first to share your thoughts.