An LLM benchmark for emergency response communications in existential catastrophes.
An LLM benchmark for emergency response communications in existential catastrophes.
Project Details
Updated 07/13/26 · Edited by orgThis project builds an LLM benchmark for emergency response communications in existential catastrophes. AI-integrated nuclear weapons systems, engineered biological threats, and AI-enabled cyberattacks could all trigger such an event from massive direct casualties or indirect deaths from collapse of critical infrastructure¹. A massive number of people are likely to see LLM recommendations on what to do in one of these catastrophes, either from search engine AI-overviews or direct queries. There are interventions that could reduce the impact of these catastrophes, but we are not confident that they would be seen or implemented in the aftermath.
Our proposed work would take groups of domain experts and put together a series of increasingly tricky scenarios for each domain with appropriate responses to benchmark models against. There are three main categories for evals here: preventing infections and societal collapse in biological catastrophes, providing resilient food solutions for abrupt sunlight reduction scenarios such as nuclear war, and meeting basic needs without infrastructure (e.g., after a widespread high-altitude electromagnetic pulse attack or an AI cyber attack). We would start with one of the key categories as a first working benchmark before ideally expanding to cover all three. We would then translate this expert consensus into an evaluation benchmark using Inspect (https://inspect.aisi.org.uk/) with advisory oversight from individuals from METR and OpenAI Safety Systems who have committed their support and free tokens. The benchmark would be uploaded to the AI Security Institute database and results would be hosted on a dedicated website to publicize a leaderboard of model performance.
¹Collapse of civilization without recovery is one way these catastrophes could drastically curtail human potential permanently.
Theory of Impact
Updated 07/13/26 · By grantmaking.aiAI could lead to any number of global catastrophes, whether through AI-integration into weapons systems, lowering the bar to engineered pathogens, rogue agents themselves causing harm, or intentionally detonating high-altitude electromagnetic pulses to counter a rogue AI (https://www.rand.org/pubs/perspectives/PEA4361-1.html). In any of these scenarios, a successful response could be the difference between it being a catastrophic event and an existential one. One way to reduce x-risk in these scenarios is to have an emergency mass communication system set up. Given that most governments have little-to-no planning in place for responding to GCRs, it seems increasingly likely that people would consult LLMs for advice through AI-integrated search results and the models themselves. It seems crucial to make sure this information is useful.
Creating a benchmark would have two main ways in which it could reduce x-risk from AI:
-
First, it would show uptake of best-practice recommendations from organizations working on reducing x-risks. This would allow these organizations to see what models are getting right about their work, what they are getting wrong, and what they are missing. In response, this can inform how they ensure their work is seen and used in a crisis, either through trying to increase AI-uptake or focusing on other methods of uptake such as government policy and planning.
People
Updated 07/13/26 · Edited by orgResearch Associate
Director & Co-founder
Discussion
No comments yet. Be the first to share your thoughts.