Reward Models as Forecasters for Alignment Training

What we will do?

Reward models are widely used in LLM post-training to improve alignment and safety. However, both reward models and the policy optimized against reward models are usually evaluated on short-term objectives. For example, reward models are evaluated by whether their preferences are aligned with human preferences over relatively short conversations in RewardBench [1], a popular reward model benchmark. In contrast, catastrophic risks often involve long term consequences and delayed harms. Since current reward models are evaluated on short time scales, it is not sufficiently understood (1) whether existing reward models can reliably assess long-term impact, (2) how they tradeoff short-term alignment objectives with long-term ones, and (3) how the temporal preference in the reward model will affect the policies trained against it. Additionally, it is unexplored how to train reward models to better reason about the long-term impact of LLM outputs. Such a training procedure would require reward models to forecast possible future states and properly assess the impact of model outputs on these states.

In this work, we pursue the following two major goals. First, we aim to build on top of the concept of successor measure [2] in RL to develop a framework for zero-shot evaluation of reward models on long-horizon tasks. Second, we aim to use this framework to study the temporal preferences encoded in existing reward models and investigate methods for shaping and controlling their temporal preferences. More specifically, by connecting successor measures and LLM forecasting [3], we aim to explore both theoretically and empirically how explicitly training for forecasting ability can change reward model’s alignment performance on long-horizon tasks. We plan to test our methods on both safety-relevant tasks that happen over long horizons (with decomposition attacks [4] as an example) and other general domains where long-term impact matters, like tutoring and strategic decision making.

References:

[1] Lambert, Nathan, et al. "Rewardbench: Evaluating reward models for language modeling." Findings of the Association for Computational Linguistics: NAACL 2025. 2025.

[2] Blier, Léonard, Corentin Tallec, and Yann Ollivier. "Learning successor states and goal-dependent values: A mathematical viewpoint." arXiv preprint arXiv:2101.07123 (2021).

[3] Chandak, Nikhil, et al. "Scaling Open-Ended Reasoning to Predict the Future." arXiv preprint arXiv:2512.25070 (2025).

[4] Yueh-Han, Chen, et al. "Monitoring decomposition attacks with lightweight sequential monitors." The Fourteenth International Conference on Learning Representations. 2026

Who's involved?

This is Haichuan Wang's MATS 10.0 mentored by Professor Peter Henderson from Princeton University. We are also working on having more collaborators onboard in the near future.

Concrete Output:

We plan to submit our work to a top tier machine learning conference, and we will also write blogposts on major alignment platforms like LessWrong to communicate our idea.