Proactive and Interpretable Safety Monitoring for LLM Agents

Large language models (LLMs) are widely used as the foundation for agents. However, they can still have unsafe behaviors, such as tool misuse, taking actions without permission, violating safety constraints, or gradual drift from intended behavior. Ensuring agent safety and providing early warnings before such behaviors happen become important challenges. Existing guardrail techniques are reactive. They detect and intervene only after an unsafe decision has already been made or an unsafe action has been generated. In this project, we aim to develop a proactive monitoring framework that can provide early warnings before unsafe behaviors happen. Our key hypothesis is that unsafe decisions are internally formed within an LLM agent before they are generated in its outputs. Therefore, the most effective way to identify unsafe behaviors is to examine the model's hidden states, track the changes of the internal representations over time, and characterize patterns that can indicate unsafe decisions.

We plan to achieve this goal by the following framework:

1. Building a trajectory-level monitor. The monitor is not only checking the current state, but instead check the trajectory-level features such as the smoothness of the trajectory, the angles between difference state vectors, the curvature of the hidden-state path, or the rate of convergence toward certain regions in the representation space. We believe this trajectory analysis can capture the unsafe signal in an early stage, and therefore providing early warnings.
2. Interpretable analysis in hidden representations. We hypothesis that the LLM's internal representation space might explicitly encode safety information. By analyzing the monitor we learn in the previous step, we can identify certain regions in the representations that store unsafe information and might trigger unsafe behavior. As a results, we can explain our monitor, providing effective information to fix the trajectory.

Current results:

We implemented the proposed framework (step 1) and test it on AT-Bench and ASSEBench. We got some promising preliminary results.

Next steps:

We will extend the test cases to more general safety benchmark, and test the generalization ability. We will conduct deeper interpretable analysis (step 2) to provide insight for internal mechanism of LLMs on safety monitoring.