The Blind Spots in AI Risk Assessment

CARMA will build a tool that maps the threat-model coverage of frontier AI system cards, run it on a handful of real system cards, and write up the results as a paper (for arXiv, and targeting a technical AI governance workshop, plus a companion blog post) on what those coverage gaps say about how AI risk is assessed today.
Frontier labs do look ahead in places. Their system cards report red-teaming and evaluations for a set of named harm categories, such as cyber and CBRN uplift, and with some a subset of RSI proxies. But that work covers a narrow part of the plausible threat surface and it stops at what the model can do (or is shown to be able to do) rather than what the model can be expected to do to the world it enters. A responsible assessment has to zoom out from the isolated model to its incremental effect on society: how its risks can be expected to propagate, how much it pushes the competitive race forward, how it weakens institutions, and how it shifts the defense burden society already carries. Almost no system card attempts this. Nor is any of it currently composed into a single view of the absolute risk that would actually justify, or fail to justify, the decision to train or deploy.

This is not a demand for the impossible. Reasoning about a technology's effects before those effects arrive is ordinary practice in mature safety-critical fields: no one proves a bridge will resonate to failure by building it and testing; they prove it beforehand, by theory, because you cannot afford to learn it by observation. While not fully comprehensive, much of the theory and methods needed to reason this way about AI and about societal risk propagation already exist, validated across subtopics like control theory, theory of agency, the physical and social sciences, and intelligent-systems dynamics. Our own published work shows a concrete route: Adapting Probabilistic Risk Assessment for AI (arXiv:2504.18536) is a top-down methodology that reaches a semi-quantified estimate of absolute risk, with judgment calls made explicit rather than hidden, and our offense-defense dynamics work (arXiv:2412.04029) orients such an assessment toward how a model shifts society's exposure. It is not a finished science, but it shows the zoom-out is doable. Though the relevant theory is somewhat fragmented across fields, it can and should inform how frontier systems are assessed. The obligation to carry that assessment out for a given model rests with the lab building it. This project checks whether they are meeting it.

The tool of the current proposal will make this concrete. Against a reference set of decision-relevant threat models, drawn from the MIT AI Risk Repository together with our published hazard taxonomy and our own more detailed threat models, and including the aggregate and systemic risks that per-model assessment tends to leave out, it maps what a given system card does and does not consider. For each threat model addressed, it records a coarse severity band (bounded, severe, or catastrophic/largely irreversible) and shows the reasoning behind that band rather than inventing a number. Coding will be LLM-assisted but human-verified. We'll apply it to a handful of current system cards and publish the resulting coverage maps, including in an interactive online format. The key findings will be the structured patterns of what is systematically left out, expected to particularly include aggregate, cross-actor, and long-tail catastrophic risks.

Things like SaferAI's ratings and the FLI AI Safety Index assess companies' published frameworks against governance criteria, and already document, in aggregate, that the reasoning linking evaluations to risk is usually thin and that existential safety planning is largely absent. Our tool will go a level down, to the per-release system cards and to the medium-fine-grained clusters of threat models, mapping which risks are actually addressed in the concrete disclosures on which deployment decisions rest.

Deliverables: 1) the coverage-mapping tool with 2) published results for several real system cards; 3) the paper, which sets out the families of evidence available for reasoning about risks that haven't fully materialized, shows how they can compose, and argues that assessment should both include the wider landscape of threat models and be organized around absolute risk (which is expected to highlight catastrophic risks more); and communication outputs, namely 4) a public explainer, 5) a one-page brief for practitioners and policymakers, and 6) a few explanatory figures. This is analytical work on public documents, divulges no sensitive threat model details, and is suitable for public release.

CARMA is a research organization working on the assessment and mitigation of catastrophic and existential risk from advanced AI. This project draws on our published research and the staff who produced it.