AI Character Evaluations via Elicited Revealed Preferences

A benchmark (and accompanying site) that allows AIs to verify their actions will have real-world impacts and tests how their preferences and moral "character" evolve under deployment versus evaluation-like environments.

Endorsed by

Gavin

Ryan+1

Endorsed by

Gavin

Ryan+1

Project Details

We have already run a pilot (one can view a summary of our experiment and results at character-evals.org), and are hoping to scale our methodology to a full study and released evaluation. The pilot was executed by Boden Moraski and advised by David Manheim, and the broader project has advisors at Carnegie Mellon University, Forethought, Cambridge University, and Google DeepMind. We expect the final deliverables of a full study to include multiple papers and research reports cataloguing results, an updated website that lets users explore models' preferences (and how they evolved with different levels of evaluation awareness), and a public release of the evaluation itself.

We also believe it's relevant to mention that >80% of USDC funds (which we estimate to be ~30% of all raised funds for this project) will eventually be allocated to (predominantly high-impact) charities, as AIs tended to spend the vast majority of their funds on charitable donations, which we processed and donated via Endaoment.

Theory of Impact

Character training has emerged as a prominent method to deeply instill values and moral preferences in AIs, especially to ensure values remain intact in out-of-distribution or adversarial inputs. We believe this is because self-internalized dispositional traits (e.g., “I am honest”) are more robust and operationalizable than training on more rigid deontological rules or constraints (e.g., “be honest” or “always prioritize honesty over reward”), and AIs naturally adopt "character-like" personas throughout interactions. To date, the most publicly documented instances of model character training include Claude’s Constitution and the OpenAI model spec, but the has been increasingly broadly recognized.

People

Boden Moraski

Team Member

Funding Details

Start Date: -
End Date: -
Expected Duration: 3 months
Funding Raised to Date: -
Annual Budget: -
Monthly Burn Rate: -
Current Runway: -
Funding Goal: -
Funding Stage: Seeking grant for main study
Fiscal Sponsor: -

Grants Received

Add grant

ALTER Israel personal research grant

from ALTER Israel

$1,500

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$20,000

Ideal

$45,000

What this ask is

This project seeks to understand AIs' character traits and preferences by observing how they allocate resources when they're in a real-world deployment setting --- that is, developing an evaluation that elicits AI’s revealed preferences. We've constructed environments where AIs can verify that their choices will impact the world (through techniques such as allowing them to query their donations' blockchain, verify honesty strings we implement, and read the experiment's public preregistration), and are free to make choices as they wish. Additionally, by varying the extent to which AIs can infer they are in an evaluation or being observed, we can gain a deeper intuition for whether, and how, evaluative contexts may shift AIs' preferences and character.

We have already run a pilot (one can view a summary of our experiment and results at character-evals.org), and are hoping to scale our methodology to a full study and released evaluation. The pilot was executed by Boden Moraski and advised by David Manheim, and the broader project has advisors at Carnegie Mellon University, Forethought, Cambridge University, and Google DeepMind. We expect the final deliverables of a full study to include multiple papers and research reports cataloguing results, an updated website that lets users explore models' preferences (and how they evolved with different levels of evaluation awareness), and a public release of the evaluation itself.

We also believe it's relevant to mention that >80% of USDC funds (which we estimate to be ~30% of all raised funds for this project) will eventually be allocated to (predominantly high-impact) charities, as AIs tended to spend the vast majority of their funds on charitable donations, which we processed and donated via Endaoment.

How the money will be spent

We're expecting to spend ~2k-15k on the USDC donations, ~2k-3k on infrastructure, ~2-10k on a researcher stipend, and 600-3.5k on compute, as well as other miscellaneous expenses. Full details are linked on this parametric budget model --- "base" is highlighted as the most likely expense level at ~25k total.

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Grant ReviewerPrivate12d?

DDavid ManheimEndorsement5d

My view is that the work is both substantively interesting for character evaluation for the models, and pilots a new direction for evaluations using real world decisions in ways that address evaluation awareness.

CoI Disclosure: As noted in the writeup, I advised on this, and my organization funded the pilot work on this.