Agent Island places agents in a rich social setting, similar to reality competitions like Survivor, to study multiagent interactions and the consequences of learning pressure in competitive settings.
Agent Island places agents in a rich social setting, similar to reality competitions like Survivor, to study multiagent interactions and the consequences of learning pressure in competitive settings.
The project will deliver (a) a benchmark for strategic capabilities in a multiagent setting [in progress for a single game configuration on the project website and working paper] and (b) a study of the impact of learning pressure in an environment of interagent cooperation and conflict. The critical project features include:
Saturation-resistance: Many AI benchmarks are subject to saturation, at which point they can no longer meaningfully track capabilities progress. In Agent Island, a more capable model can always exceed the current leading model, meaning the benchmark is unlikely to saturate.
Contamination-resistance: Contamination, in which models train on evaluation tasks, also taints many benchmarks. Since an agent is competing against other adaptive agents, Agent Island is dynamic and thus mitigates contamination risk.
Separation of capabilities and propensities: Much research on dangerous or concerning capabilities confoundscapabilities withpropensities: we might understate a model’scapability to manipulate since they lack thepropensity to manipulate. Agent Island provides an environment with a unique degree of license to engage in interagent manipulation and persuasion.
A novel study of the dynamics of learning: Deployed agents could face reinforcement learning pressure on competitive outcomes. The degree of zero-sumness in the deployment environment, for example, could shape the behavior reinforced in learning.
High-stakes, multiagent interactions will become commonplace as AI agents grow in capabilities and are increasingly entrusted with resources and decisionmaking authority. Séb Krier of Google DeepMind argues that the multiagent perspective is missing from frontier AI discourse and that it represents the likely future of AI deployments.
First, Agent Island helps us understand the emergent properties of such systems. Combinations of seemingly safe models can have concerning properties in aggregate, so work of this form is critical to assessing the safety of live AI deployments.
Second, if agents are entrusted with resources and decisionmaking authority, are they susceptible to pressure from other agents? Could a rogue agent overpower seemingly aligned agents? Agent Island
Team Member
The $25,000 funding target is targeted at my continued part-time work on the project, with $10,000 devoted to the ongoing benchmarking effort and $15,000 devoted to initial reinforcement learning runs. As I scale up funding, most of the marginal dollar goes to the more cost-intensive RL efforts.
Here is the budget for the $500k range, which secures my ability to work on this project full-time for a year.
Third, it is plausible that progress in continual learning will lead to agents facing reinforcement learning pressure in competitive, perhaps even zero sum, environments. I endeavor to understand the consequences of this pressure for a variety of alignment-related outcomes, especially survival preferences. Further, I hope to found an organization that studies how to make such learning pressures consistent with maintaining desired alignment properties.
Private comment. Only shown to approved funders and grant reviewers.