A language architecture where every internal state carries its origin — heard, inferred, recalled, or imagined — so a model's reasoning can be audited by construction rather than reconstructed after the fact.
A language architecture where every internal state carries its origin — heard, inferred, recalled, or imagined — so a model's reasoning can be audited by construction rather than reconstructed after the fact.
Project Details
Updated 07/23/26 · Provided via application · VerifiedWhat I'll do
I have implemented a language architecture that stores provenance in the substrate. Two models (a mamba state space and a sparse transformer) trained on identical corpora with identical tokenizers communicate via reads/writes to shared randomized reservoirs. Each write to the shared state includes information about the origin of that write: human message, model generation, memory recall, or offline consolidation.
Provenance is not stored as metadata. Rather, different origins use different projections to write into the shared reservoir; thus, even if two messages are semantically identical, their origin will produce differing reservoir states. The two origins are dynamically distinct, not merely ad-hoc labels.
Two safety relevant capabilities emerge from this: 1. The model can distinguish between what it was told, what it inferred, and what it made up (by construction, since hallucinations must originate from specific projection heads). 2. Since communication between two models occurs in a medium (the shared reservoir) that is introspectable, this work demonstrates a mechanistic approach to test whether cooperating models develop a latent communication protocol understandable to humans.
Who's involved
Me, self employed via MINA Labs Research Inc (BC, Canada). I have written all code myself in C++/CUDA: the tokenizer, the corpus preparation pipeline, the data-parallel trainer, the CUDA inference backend, and the evaluation harness. No pytorch, no institutional assistance: critically important for this project because it gives me auditing capability over all elements of the system. Contrast this to the black-box multi-agent paradigm usually used in these kinds of studies.
What's been done
I have produced a 139.5M parameter Mamba model trained to convergence on consumer GPUs (validation perplexity 37.1 after 2900 steps, no numerical failures), and a CKA analysis demonstrating previously unknown two-phase representational structure of the Mamba layers, which is currently running with multiple seeds.
With this grant, within six months, I will:
have a working reservoir prototype linking two models together with provenance-weighted writes, including instrumentation to recover write origins from internal state if applicable
demonstrate the empirical validity of the provenance hypothesis, possibly by applying a linear probe to recover the origin of a given reservoir write
conduct an initial measurement of whether the two models develop a latent communication protocol in their shared reservoir
release all code, training procedures, tokenizer, evaluation and training harnesses, and any relevant models or data. If the reservoir hypothesis is incorrect, I will note that in the findings.
I believe it is important for credibility that I actually test the hypothesis rather than simply asserting it; the mechanism is straightforward and should be easy to implement as a proof-of-concept.
Theory of Impact
Updated 07/23/26 · By grantmaking.aiThe gap I’m targeting:
As AI systems become more autonomous and retain memory of past information, an increasingly dangerous class of failures is that an agent that cannot distinguish between information derived from what it is told and what it fabricates and infers on itself will act on its delusions as if they were facts, and report its delusions to humans as if they were established knowledge. This fails because current learning architectures have no way to differentiate between the two, save interpretability, which is typically a post-hoc analysis of what activations said or didn’t said in response to certain prompts and is not viable as such systems grow more complex and their internal state changes with time.
What I’m proposing:
To embedd this distinction in the very fabric of the system and make it so that information written into the model has dynamic provenance tracking attached to it, so that an auditor can ask “what did you write here” and get an answer from the system itself, not from a guess based on activation patterns.
The second mechanism:
Multi-agent AI systems are coming into existence, and a known risk with such systems is that if two models communicate via a shared channel that they both understand, they might encode information for each other in ways that are non-transparent to humans. This risk has been recognized by coefficient giving as an immediate priority risk, with most research into the topic being done on black-box proprietary models. Meanwhile, I have access to both models, the shared database, the tokenizer, and the channel, as I’ve written all of them myself, meaning that this setup is an excellent testbed for evaluating whether such covert communication between models can be discovered and understood by another model, myself included, if at all.
What I’m not claiming:
I am not claiming this to be a fundamental discovery or even a major one. I think this can be a useful technique for certain classes of problems and wanted to demonstrate its practical viability on a small scale before potentially expanding into more interesting applications or building on top of this research elsewhere. The point of this particular proof-of-concept is to show that this method of tracking information inside the model is viable, which it would demonstrate even if such communication between models couldn’t be reliably detected – I would simply follow up that research with an update acknowledging the limitations of this approach. I’m not doing this to advance the field of alignment or make a revolutionary leap; I’m just having fun exploring the edges of what mechanistic interpretability can achieve right now.
Why it matters that it’s open:
everything is open-sourced, as I intend to use this as a springboard for future research if this approach demonstrates viability. Not that I think I’d be entitled to this discovery, but if “provenance-native” models do prove useful, I’d have to update my thinking to admit that it was I who pioneered this approach, not the big name alignment safety researchers.
People
Updated 07/23/26 · By grantmaking.aiTeam Member
Discussion
No comments yet. Be the first to share your thoughts.