Shaping AI’s « safe culture »

Raising $30,000active

Identify what AI early risks or failures could be reported despite strategic rivalries towards a « safe culture » shaped after aviation

Endorsed by

Ryan+1

Identify what AI early risks or failures could be reported despite strategic rivalries towards a « safe culture » shaped after aviation

Endorsed by

Ryan+1

Project Details

This project will work towards a shared map and of risks, safety occurrences and categorization of features specific to the AI developments that could fall under a common agreement framework for reporting and processing within the current geo-economic-political environment without waiting for significant change in the AI development landscape.

While FAA and EASA are largely inadequate facing fast development cycles i.e. with drones, the “just culture” developed and defined at the UN level (through ICAO) has proven to be most effective despite several occurrences of regulatory disagreements or unequal influence.
Despite economic interests and corporate ambitions and, perhaps most importantly, in spite of any enforcing power by ICAO, the « just culture » has found its way and infused all of aviation across borders and cultures. ICAO provides the shared definition of failure, used by all actors worldwide, with some freedom of interpretation. During the cold war, similar cooperation was developed with a US-USSR agreement on sea safety. Despite geopolitical and economic challenges, safety interests prevail. Whether in the air or on water, the success came from finding a vocabulary that is open enough that all parties can agree and specific enough that it contributes to safety improvement. Those same principles should prevail with artificial intelligence.

This project would aim to find and shape this perimeter for Artificial Intelligence. What categories of technical failures, vulnerabilities, unanticipated behaviours, safety incidents can, in today’s world, be signalled and shared in an open manner between rival labs or states. Incidentally, the project is likely to find such type of occurrences that will remain out of scope as long as the strategic rivalry prevails.

Recent events increase the likelihood and improve the chances of success. Palantir’s CEO recently reported that “the architecture that maximally preserves sovereignty is one that enables institutions to own their tribal knowledge”. Facing recent setbacks in Europe, Palantir became an advocate for more transparency.

Where harm is diffuse and delayed, the taxonomy will need severity tiers based on plausible downstream consequence rather than immediate, observable damage. Where there is no single certifying body, categories will be built to be voluntarily adoptable by individual labs or states rather than mandated top-down — closer to how ICAO standards spread through incentive and convention rather than enforcement. Where reproducibility is low, the reporting framework will privilege pattern-level classification over incident-by-incident technical verification.

The final deliverable will be a public taxonomy showing risk categories, incidents ranked based on today’s cooperation likelihood along with the framework of a body most likely to be the architect and owner of the scaled “just culture”.
This vision has already been discussed in private circles with AI and aviation experts. To the best of my knowledge, while tools such as the AI Incident Database, OECD.AI's incident monitor, or METR's evaluation-sharing efforts aim for western dominant and western centric reporting, this project has the ambition to work beyond borders and cultures. Those organizations will be natural allies and inspirations. As in any project, idea is 10% of the results, delivery makes the rest. In that spirit, here are a few differences that are accounted for early and will be addressed.

Differences between Aviation and AI, eg that in aviation, accidents are obvious and publicly visible. while, with AI, harm is often diffuse, delayed, or difficult to attribute have been identified.
This project carries significant execution risk. Many of them have been identified and mitigation defined.

Theory of Impact

25+ years on insider experience in aerospace, whether in airlines, rulemaking bodies, industry or leading-edge technology (Advanced Air Mobility), and 25 years working internationally, I have seen firsthand how to shape, enforce and hold to safety standards that benefit everyone beyond geopolitical chasms, industrial competition and diverging interests.
My read of today’s AI safety world is that it is predominantly western, mostly Anglo-Saxon, essentially assuming a development compatible with the dominant economic interests and focused on the mitigation of the symptoms through alignment, interpretability and some common redlines.
I see this work as groundbreaking, fundamental yet it would benefit from a common ground that does not yet exist. The need to set common definitions, of failure, of serious, and a common reporting structure is most likely to bear more fruits than any efforts done in silo.

In that sense, this project will reduce risk in two complementary fashions.
First, it will identify (also searching in existing literature) where cooperation is possible despite geopolitical feuds. Thus, it will trigger one step, however big it is towards cooperation and collaboration without waiting for a change in the postures and ambitions that might simply not come. Second, by aiming to be an ICAO rather than an FAA, the project will go beyond national scope and limitations and find common norms and voluntary contributions. Rather than narrowing the debate, it will foster the international debate towards the common features, goals and ambitions without which AI governance will not be truly impactful. The mechanism is incremental rather than direct. A shared taxonomy will lower the cost of early warning. Today, a lab or state that observes a concerning failure mode has no shared vocabulary to signal it without either triggering competitive or reputational backlash or staying silent. A taxonomy that is narrow enough to be politically safe to use, but specific enough to carry real information, changes that calculus.

People

Benoît Larrouturou

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$30,000

Ideal

$60,000

What this ask is

Amodei recently pleaded for an “FAA-like” authority. Learning from 25 years across aviation, I know firsthand how such bodies have become -mostly- obsolete in the face of the speed of change. The more durable lesson from aviation is not the certifying authority itself, but the culture of reporting that grew up around it. However, several principles of the “just culture” that has developed in aviation and made it so safe can be replicated.

This project will work towards a shared map and of risks, safety occurrences and categorization of features specific to the AI developments that could fall under a common agreement framework for reporting and processing within the current geo-economic-political environment without waiting for significant change in the AI development landscape.

While FAA and EASA are largely inadequate facing fast development cycles i.e. with drones, the “just culture” developed and defined at the UN level (through ICAO) has proven to be most effective despite several occurrences of regulatory disagreements or unequal influence.
Despite economic interests and corporate ambitions and, perhaps most importantly, in spite of any enforcing power by ICAO, the « just culture » has found its way and infused all of aviation across borders and cultures. ICAO provides the shared definition of failure, used by all actors worldwide, with some freedom of interpretation. During the cold war, similar cooperation was developed with a US-USSR agreement on sea safety. Despite geopolitical and economic challenges, safety interests prevail. Whether in the air or on water, the success came from finding a vocabulary that is open enough that all parties can agree and specific enough that it contributes to safety improvement. Those same principles should prevail with artificial intelligence.

Besides 25 years in aviation, as a pilot, airline operator, rule maker, safety expert, this work will leverage 100+ hours I have spent engaging Chinese actors to understand their governance model. This investigation, both what was said and, more importantly what was not said, gave me a first hand view on how this country thinks about developments, cooperation and risk. As a European, I will be able to structure and lead interviews (in my network and beyond) with air safety experts, AI governance experts, developers and practitioners. In addition, weapon and nuclear experts are likely to provide valuable insights on how to frame cooperation beyond rivalry.

The final deliverable will be a public taxonomy showing risk categories, incidents ranked based on today’s cooperation likelihood along with the framework of a body most likely to be the architect and owner of the scaled “just culture”.
This vision has already been discussed in private circles with AI and aviation experts. To the best of my knowledge, while tools such as the AI Incident Database, OECD.AI's incident monitor, or METR's evaluation-sharing efforts aim for western dominant and western centric reporting, this project has the ambition to work beyond borders and cultures. Those organizations will be natural allies and inspirations. As in any project, idea is 10% of the results, delivery makes the rest. In that spirit, here are a few differences that are accounted for early and will be addressed.

Differences between Aviation and AI, eg that in aviation, accidents are obvious and publicly visible. while, with AI, harm is often diffuse, delayed, or difficult to attribute have been identified.
This project carries significant execution risk. Many of them have been identified and mitigation defined.

How the money will be spent

Minimum : $30k

This amount covers the « minimal viable version » of the project. I would include desktop research, 30 to 40 structured interviews with domain and cross domain experts and the writing of the deliverable. This version allows an early report without the depth of cross examination between sectors, cultures and countries that would make this work truly unique and valuable globally.

Desktop research, identification, coordination and time for the interviews : circa 300 hours : 15000 $
Travels, logistics and onsite expenses for interviews : 10 000 $
Writing, proofreading and publication of report : 5 000 $

Ideal version : 60 000 $, covering, in addition to the above
This amount would allow to deliver on the complete version. Interviews would cover three geographic areas (China, USA and Europe), included experts in China and in the aerospace world I already identified. Budget-dependent, the report could be translated in Chinese to optimise its reach, including through Asian AI governance network.

Increased base of interviews, travels included
Organisation of a workshop to engage a large international audience to validate findings and feasibility
Professional translation of the report
Follow up and roadmap to actual implementation

Discussion

MMark AikenEndorsement17d(edited)

Endorsed. I think one of the weaknesses of AI safety so far is the limited lessons learned from other regulated industries, including areospace which balances high volume transactions / low probably - high impact risks / innovation in a much more established regulatory ecosystem (notwithstanding the weaknesses of that ecosystem, which I think we can also learn from). Your background in aerospace means that you bring unique senior experience to these questions.

One caveat is that a signficant part of the budget is travel expenses for on-site interviews. On-site interviews are nice, but online would seem equally effective? It might also work to use the travel portion of the budget for an in-person launch of your report in China, rather than interviews?

BBenoît Larrouturou17d

I like the idea... Yet in person connection makes a difference to build trust.