Autonomy Eval

Raising $5,000active

An open evaluation environment for testing whether human autonomy preservation in AI systems is primarily a model-behavior problem, an interface-design problem, or a systems problem requiring both

Project Details

Previous works on studying human autonomy/agency in AI systems approach it as either a model behavior problem or an interface design (human's pattern of AI use).

The HumanAgencyBench, for instance tests whether models behave in an autonomy preservation manner through asking of clarifying questions and deference of important decisions while the skill-formation study frames it as an interaction patterns problem and shows that different patterns of AI use lead to different learning outcomes.

My hypothesis is that autonomy preservation in AI systems requires both, and the goal of this project is to:

Figure out to what extent is this a model behavior problem as much as it is an interface design problem?
Figure out what are the specifics approaches in both model behavior and interface design that promote autonomy preservation?

To achieve this, I plan to build interactive environments that users can interact with AI systems to perform a variety of creative tasks, and use that to measure autonomy preservation in as a controlled factor of model behavior, interface design and a combination of both. My goal is to figure out the simplest environments possible (gaming/creative) that will enable these measurements and then use that as a way to infer scale ups needed to measure autonomy preservation in AI systems proliferated in complex real world work environments.

I will be a sole human working on this project, and I got inspiration to work on this project through my own personal projects related to AI-native games and creative interfaces.

The specific research questions came from my recent project titled Wonderloom: a multimodal canvas aiming to help children develop narrative thinking, literacy, self-expression, and creative agency through storytelling (video demo). The idea to use gaming environments as a way to measure and study these questions also came from my previous work on social deduction agents and AI-native games where I developed agents that excel in the game of Mafia and gaming environments to play against agents (blogpost).

The main project outputs will be:

Open-source interactive evaluation environment(s).
Open-source datasets and post-trained models to test autonomy preservation as a factor of model behavior
Project research report(s)

Theory of Impact

Our aim with the work is to be able to reduce human-autonomy x-risk (gradual disempowerment) by providing simple environments to study and measure autonomy preservation methods as well as mechanisms to scale up autonomy preservation studies in increasingly complex environment of knowledge work. We hope the long-term results of this work is the reduction of this x-risk as AI diffuses into the global economy.

People

Alfaxad Eyembe

Team Member

Grants Received

Add grant

Only visible to verified funders, reviewers, and admins.

Email hi@grantmaking.ai to get verified

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$5,000

Ideal

$10,000

How the money will be spent

Because the project is still exploratory, the funding will primarily support the technical experimentation needed to identify a workable evaluation design. This includes GPU compute for post-training and testing open-source models, frontier-model API costs, hosting and infrastructure for the interactive evaluation environments, and development time for building, testing, data collection and iterating on those environments.

The minimum funding amount of $5,000 would be sufficient to build and release an initial version of the project. This would include at least one open-source evaluation environment, baseline experiments using prompted and open-source models, an initial dataset of interaction trajectories, and a public report documenting the methodology, results, and limitations.

The ideal funding amount of $10,000 would allow for more iterations and broader experimentation. It would provide additional compute for post-training, enable comparisons across more models and intervention designs, and support more extensive testing of both model-level and interface-level approaches. It would also create room to improve the environments based on early results.

Discussion

No comments yet. Be the first to share your thoughts.