GCR Disaster Response Benchmark

An LLM benchmark for emergency response communications in existential catastrophes.

Project Details

This project builds an LLM benchmark for emergency response communications in existential catastrophes. AI-integrated nuclear weapons systems, engineered biological threats, and AI-enabled cyberattacks could all trigger such an event from massive direct casualties or indirect deaths from collapse of critical infrastructure¹. A massive number of people are likely to see LLM recommendations on what to do in one of these catastrophes, either from search engine AI-overviews or direct queries. There are interventions that could reduce the impact of these catastrophes, but we are not confident that they would be seen or implemented in the aftermath.

Our proposed work would take groups of domain experts and put together a series of increasingly tricky scenarios for each domain with appropriate responses to benchmark models against. There are three main categories for evals here: preventing infections and societal collapse in biological catastrophes, providing resilient food solutions for abrupt sunlight reduction scenarios such as nuclear war, and meeting basic needs without infrastructure (e.g., after a widespread high-altitude electromagnetic pulse attack or an AI cyber attack). We would start with one of the key categories as a first working benchmark before ideally expanding to cover all three. We would then translate this expert consensus into an evaluation benchmark using Inspect (https://inspect.aisi.org.uk/) with advisory oversight from individuals from METR and OpenAI Safety Systems who have committed their support and free tokens. The benchmark would be uploaded to the AI Security Institute database and results would be hosted on a dedicated website to publicize a leaderboard of model performance.

¹Collapse of civilization without recovery is one way these catastrophes could drastically curtail human potential permanently.

Theory of Impact

AI could lead to any number of global catastrophes, whether through AI-integration into weapons systems, lowering the bar to engineered pathogens, rogue agents themselves causing harm, or intentionally detonating high-altitude electromagnetic pulses to counter a rogue AI (https://www.rand.org/pubs/perspectives/PEA4361-1.html). In any of these scenarios, a successful response could be the difference between it being a catastrophic event and an existential one. One way to reduce x-risk in these scenarios is to have an emergency mass communication system set up. Given that most governments have little-to-no planning in place for responding to GCRs, it seems increasingly likely that people would consult LLMs for advice through AI-integrated search results and the models themselves. It seems crucial to make sure this information is useful.

Creating a benchmark would have two main ways in which it could reduce x-risk from AI:

First, it would show uptake of best-practice recommendations from organizations working on reducing x-risks. This would allow these organizations to see what models are getting right about their work, what they are getting wrong, and what they are missing. In response, this can inform how they ensure their work is seen and used in a crisis, either through trying to increase AI-uptake or focusing on other methods of uptake such as government policy and planning.

People

James Mulhall

Research Associate

Prof. David Denkenberger

Director & Co-founder

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$40,000

Ideal

$250,000

How the money will be spent

There are three high-priority categories that could be covered by this benchmark: nuclear weapons, biological threats, and loss of infrastructure. The minimum vs ideal amount mainly differs in how many categories we cover, in what depth, and with which experts.

The minimum amount would allow us to create the benchmark for one category using an expert consensus from our existing and internal networks. This funding would cover 1 FTE-equivalent for 6 months, overhead, and modest funding (~$5k) for subcontracting help with evals and/or tokens. ALLFED has done extensive research on meeting food and water needs following a nuclear war and could create an eval with relatively little external input. We also have multiple projects on protecting the public and meeting basic needs in biological catastrophes and loss of infrastructure scenarios, and could alternatively start with an eval of one of these categories.

The maximum amount would allow us to set up dedicated expert working groups to create benchmarks for all three categories. This would cover 2 FTE-equivalents for 12 months, overhead, a higher subcontracting and tokens budget (~$30k), a publication, and small stipends for field experts, which would be particularly beneficial for biological threats and loss of infrastructure scenarios. This is similar in costs to WMDP Benchmark, which states in its preprint that it cost over $200k and covers biosecurity, chemical security, and cybersecurity from an offensive perspective (https://arxiv.org/pdf/2403.03218). For any amount within the funding range, we would publicize the results via the AI Security Institute database and a website with a leaderboard of model results.

Discussion

No comments yet. Be the first to share your thoughts.