Representation Learning and Alignment in Generative Models

This project investigates the following fundamental question: When a generative model is trained to learn a probability distribution from data, what internal representations does it develop, and how do these representations influence its ability to learn and generalize to new goals?

Modern generative models, including language models and diffusion/flow-based models, are typically pretrained using distributional learning objectives. However, accurately modeling a data distribution does not necessarily imply that the model learns representations suitable for downstream tasks or alignment with human objectives. Different representations may achieve similar generative performance while differing substantially in how easily they can be adapted to new goals, particularly under distribution shifts. This may help explain phenomena such as goal misgeneralization, where a model appears to learn the intended objective during training but behaves unexpectedly in new situations.

I aim to study this problem using tools from applied probability, statistics, information theory, and dynamical systems. A central objective is to distinguish alignment failures arising from insufficient information in learned representations from those arising from downstream optimization or generalization. I hope this will lead to both a clearer understanding of alignment and practical methods for making generative models more reliable and controllable. I intend to make the research outputs publicly available, including theoretical results, experimental code, and reproducible benchmarks. If the results are sufficiently strong, I would consider submitting the work to the newly launched Alignment journal as well as to established machine learning venues.

The project is at an early proof-of-concept stage, with preliminary theoretical formulations, computational experiments using small generative models, and promising initial results. The next step is to establish rigorous guarantees and validate practical methods on larger models.