Is AI safety research reproducible?

Science is an exercise in seeing further by standing on the shoulders of giants, and AI safety is now the field on whose shoulders the highest-stakes decisions stand. Lab training policies, government regulation, and hundreds of millions in philanthropic funding rest on empirical results that, today, no one systematically verifies. When we ran a reproducibility audit on the social and behavioral sciences (DARPA's SCORE program, reported in two 2026 Nature papers I coauthored), only 24% of papers even had the materials available to attempt a computational reproduction. No one has run this test on AI safety, a literature with added risk factors: much of it is published without peer review on proprietary models at a rapid pace.

In this project, I run the first systematic computational-reproducibility audit of empirical AI safety research. I have experience in building an automated agent-based system for performing these audits in the context of behavioral and social science research: a pipeline that, given a paper, finds its data and code, constructs the computational environment, re-executes the analysis, compares the outputs against what was reported, and compiles everything into a signed reproducibility package. Here, we adapt the work to the AI-safety corpus, which means handling everything from Alignment Forum posts without DOIs, to GPU fine-tuning workloads, to eval harnesses, and then applies the method to a random sample of empirical results, scaling with funding.