Autonomy Eval

Previous works on studying human autonomy/agency in AI systems approach it as either a model behavior problem or an interface design (human's pattern of AI use).

The HumanAgencyBench, for instance tests whether models behave in an autonomy preservation manner through asking of clarifying questions and deference of important decisions while the skill-formation study frames it as an interaction patterns problem and shows that different patterns of AI use lead to different learning outcomes.

My hypothesis is that autonomy preservation in AI systems requires both, and the goal of this project is to:

• Figure out to what extent is this a model behavior problem as much as it is an interface design problem?
• Figure out what are the specifics approaches in both model behavior and interface design that promote autonomy preservation?

To achieve this, I plan to build interactive environments that users can interact with AI systems to perform a variety of creative tasks, and use that to measure autonomy preservation in as a controlled factor of model behavior, interface design and a combination of both. My goal is to figure out the simplest environments possible (gaming/creative) that will enable these measurements and then use that as a way to infer scale ups needed to measure autonomy preservation in AI systems proliferated in complex real world work environments.

I will be a sole human working on this project, and I got inspiration to work on this project through my own personal projects related to AI-native games and creative interfaces.

The specific research questions came from my recent project titled Wonderloom: a multimodal canvas aiming to help children develop narrative thinking, literacy, self-expression, and creative agency through storytelling (video demo). The idea to use gaming environments as a way to measure and study these questions also came from my previous work on social deduction agents and AI-native games where I developed agents that excel in the game of Mafia and gaming environments to play against agents (blogpost).

The main project outputs will be:

• Open-source interactive evaluation environment(s).
• Open-source datasets and post-trained models to test autonomy preservation as a factor of model behavior
• Project research report(s)