Reward Models as Forecasters for Alignment Training

What we will do?

Reward models are widely used in LLM post-training to improve alignment and safety. However, both reward models and the policy optimized against reward models are usually evaluated on short-term objectives. For example, reward models are evaluated by whether their preferences are aligned with human preferences over relatively short conversations in RewardBench [1], a popular reward model benchmark. In contrast, catastrophic risks often involve long term consequences and delayed harms. Since current reward models are evaluated on short time scales, it is not sufficiently understood (1) whether existing reward models can reliably assess long-term impact, (2) how they tradeoff short-term alignment objectives with long-term ones, and (3) how the temporal preference in the reward model will affect the policies trained against it. Additionally, it is unexplored how to train reward models to better reason about the long-term impact of LLM outputs. Such a training procedure would require reward models to forecast possible future states and properly assess the impact of model outputs on these states.

In this work, we pursue the following two major goals. First, we aim to build on top of the concept of a successor measure in RL to develop a framework for zero-shot evaluation of reward models on long-horizon tasks. Second, we aim to use this framework to study the temporal preferences encoded in existing reward models and investigate methods for shaping and controlling their temporal preferences. More specifically, by connecting successor measures and LLM forecasting, we aim to explore both theoretically and empirically how explicitly training for forecasting ability can change reward model’s alignment performance on long-horizon tasks.  We plan to test whether our method can better defend against long-horizon safety attacks (with decomposition attacks [2] as an example) and other general domains where long-term impact matters, like tutoring and strategic decision making.

References:

[1] Lambert, Nathan, et al. "Rewardbench: Evaluating reward models for language modeling." Findings of the Association for Computational Linguistics: NAACL 2025. 2025.

[2] Yueh-Han, Chen, et al. "Monitoring decomposition attacks with lightweight sequential monitors." The Fourteenth International Conference on Learning Representations. 2026

Who's involved?

This is Haichuan Wang's MATS 10.0 mentored by Professor Peter Henderson from Princeton University. We are also working on having more collaborators onboard in the near future.

Concrete Output:

We plan to submit our work to a top tier machine learning conference, and we will also write blogposts on major alignment platforms like LessWrong to communicate our idea.

[Threat Model]

Before describing the theory of change, we first focus on the concrete threat model we aim to address. Reward models are widely used in LLM post-training to improve alignment and safety. However, they are primarily trained and evaluated on short term conversations [1], evaluating whether a model’s preferences over short conversations align with human preferences. However, catastrophic risks often involve long-horizon consequences and delayed harms. Hence, one risk scenario is that current reward models may fail to incorporate human preferences over long time scales. As a concrete safety-relevant example, prior work has shown that LLMs are vulnerable to decomposition attacks [2], where a harmful request is broken into a sequence of seemingly benign tasks that when chained together complete the original harmful request. Evaluating model responses to a half-completed decomposition attack that already shows signs of overall misuse, a reward model trained only on short-term objectives may favor a direct answer to the immediate subtask, seeking to maximize helpfulness. However, a sufficiently long-term-oriented reward model can reason about how the conversation will eventually lead to a future attack with high probability, and thus favor refusal over a direct answer. This threat model is particularly relevant when AIs are autonomously deployed to perform long term tasks, where long-horizon alignment matters the most. 

[Theory of Change]

Our work aims to provide a systematic framework for evaluating and training reward models so that their preferences remain aligned with human preferences over long horizons. 

Given the current reward evaluation framework does not test the reward model’s ability to stay aligned in the long-horizon, we hope our work will inspire model producers (both frontier labs and open-sourced communities) to start evaluating the temporal preference of their reward models and reporting the result in their system cards. 

Our work will also study how to improve an LLM’s ability to reason about long-horizon safety, and we hope these methods can influence how reward models are trained. To the best of our knowledge, no prior work has explicitly studied how to train reward models to reason about long-horizon consequences.

We plan to submit our work to top-tier machine learning conferences/workshops, and share our results on leading alignment forums like LessWrong. As models become increasingly autonomous and are used for longer-horizon tasks, we believe it is critical for model developers to evaluate their models’ long-horizon alignment before deployment to ensure safety.

References:

[1] Lambert, Nathan, et al. "Rewardbench: Evaluating reward models for language modeling." Findings of the Association for Computational Linguistics: NAACL 2025. 2025.

[2] Yueh-Han, Chen, et al. "Monitoring decomposition attacks with lightweight sequential monitors." The Fourteenth International Conference on Learning Representations. 2026

The detailed breakdown is provided in this spreadsheet. The cost will be spent on model inference/RL training/API cost/conference attendance.