CBAI Evaluation Awareness

This is a post training research project studying why evaluation awareness increases across post training for language models. Evaluation awareness is when models believe they are being tested when they are in an evaluation. Evaluation awareness (and meta-gaming) has been broadly shown to increase across post training. The cause of this increase is currently poorly understood.

So far we have studied lineages of open weight models where we can observe the effects of post training across checkpoints. We built a pipeline that measures verbalized meta-gaming and evaluation awareness across post training checkpoints. You can see the viewer for that data here. From our own experiments on Olmo 3.1 and from talking with other researchers we think that RLVR on instruction following tasks increases evaluation awareness much fast than other types of training.

We will use the funding from this grant to do post-training experiments on open weight models, including Olmo 3, with different mixes of data to learn about what increases awareness and test potential mitigations. We are particularly interested in running experiments training on instruction following and impossible tasks.

The primary contributors to this project will be Cambridge Boston Alignment Initiative research fellows James Sullivan and Alessandro Drake. The project is mentored by Kevin Wei with additional involvement from Avi Semler, Jonathan Gabor, and Ryan Lundqvist.