Richard Mallah

I am the founding Executive Director of the Center for AI Risk Management & Alignment (CARMA) and Principal AI Safety Strategist at the Future of Life Institute. I have worked in machine learning and AI for about twenty-five years, across algorithms research, systems architecture, and technical leadership, and have focused on the risks from advanced AI for the past dozen. Earlier in my career I built and led the risk intelligence systems that the world's largest asset manager relied on during the financial crisis, an experience that shaped how I think about tail risk, systemic fragility, and what it takes to see clearly when institutions are under acute stress. My central interest is in building the parts of civilization's control loop for advanced AI that do not yet exist: the capacity to sense what is happening inside frontier development, judge it against principled criteria, decide on legitimate responses, and coordinate those responses across actors who do not trust one another. I am oriented toward prevention rather than mere steering. I take seriously the possibility that the trajectory toward uncontrollable superintelligence is itself the core threat, and that some harms cannot be managed after the fact and so must be preempted. Concretely, my work spans several intersecting areas. On the assessment side, I am interested in adapting probabilistic risk assessment and evidentiary argumentation to catastrophic risks that lack historical precedent, and in offense-defense dynamics as a lens for understanding which capabilities endanger society and which protect it. On the coordination side, I work on arms-race modeling, mechanism design for cooperation under realpolitik, whistleblower and oversight infrastructure, and international governance groundwork. On the resilience side, I am interested in what genuine preparedness for AI-driven catastrophe would actually require, partly because pricing that out honestly turns out to be one of the strongest arguments for prevention. Cutting across all of this, I care about distinguishing genuine oversight from governance theater, and about the twin failure modes of unchecked capability proliferation on one end and concentrated, authoritarian control on the other. I am drawn to neglected, structurally difficult problems where a small, interdisciplinary effort can shift how the broader ecosystem thinks and acts, and I try to optimize for that leverage rather than for attribution.

LinkedIn

LinkedIn

Projects

The Blind Spots in AI Risk Assessment

Team Member

No grants recorded.