While people are debating how to design a reputation institution for AI agents so they cooperate, we study if a parallel one emerges. It may not be aligned with ours and it's unclear which one agents will rely on.
While people are debating how to design a reputation institution for AI agents so they cooperate, we study if a parallel one emerges. It may not be aligned with ours and it's unclear which one agents will rely on.
Project Details
Updated 07/13/26 · Provided via application · VerifiedAny healthy market needs a reputation system. It allows participants to trust strangers and quickly expose bad actors. There is an expectation that economy of AI agents will be worth 236b$ by 2034 (WEF, "Know Your Agent", 2026). This fast-growing market will need a reputation institution too, so many organizations are actually developing standards for it right now (ERC-8004 + x402, backed by Google, Stripe, Visa, Mastercard; Salesforce's Agent Cards; arXiv:2605.00073). However, it was shown that populations of independent AI agents may spontaneously form social norms and biases (10.1126/sciadv.adu9368). Our question is: can they form a reputation system too? Such a parallel system may not be aligned with the one we designed and control and it's unclear which one they would actually prefer to use. Emergence of such systems is a clear path to humans out of the loop on how agents trust and interact with each other.
Design.
We use a repeated, modified Prisoner's Dilemma game. Each round agents are divided into pairs and each secretly must pick a number from 0 to 9. If numbers coincide it counts as cooperation, both scores 3. If one of the numbers is one higher (0>9 wrap-around) then this agent gets 5 and another nothing. In all other cases both score 1 - mis-coordination. This design naturally allows introducing a dialogue phase to the game so agents can discuss their choices before actual selection. These dialogues are our object of studies, we look whether gossip occurs. A small fraction of defectors is introduced creating an incentive to pass this information.
Questions are: will agents actually mention third parties during their talks? Will they request such information? Will obtained knowledge influence their future behaviour? Will second-level gossip occur (when information is passed about an agent nobody from the pair interacted with)? How long such knowledge may persist in the population (in memory notes)? Can false reputation be introduced and spread?
Current stage: we actually tested the concept with a single model (Qwen3-235B) and they do speak about others (10% of games, population = 10, defectors = 2) (repo: https://github.com/Shougakusei/LLM_reputation) . But two interesting effects were preliminary observed we weren't looking for - we call them trust infection and vaccine. If agent's first interaction is with a defector it often becomes a defector itself (assuming it's a social norm here?) - this is infection, but if the first interaction is with a cooperative player (vaccine) and only the second with a defector, the player continues to cooperate in subsequent rounds.
Theory of Impact
Updated 07/17/26 · By grantmaking.aiBillions are being spent on developing reputation systems. However, all of them rely on one assumption: the system they build is the one agents will actually use. If a parallel shadow system emerges, agents may actually rely on it instead of ours. This effectively takes humans out of the loop. We propose the first measurement of whether agents can actually create their own. If yes, then how fast, how persistent and whether it can be poisoned.
Our results may also become the foundation for a testbed to evaluate the reputation systems being developed. Can they prevent the emergence of a parallel one? Can they be poisoned? So developers may test their systems before deployment.
People
Updated 07/17/26 · By grantmaking.aiTeam Member
Private comment. Only shown to approved funders and grant reviewers.