The first open-source AI safety evaluation benchmark in Hausa, Yoruba, Igbo, and Nigerian Pidgin — testing whether frontier models refuse harmful requests, including biosecurity guidance, in languages spoken by 200+ million people
The first open-source AI safety evaluation benchmark in Hausa, Yoruba, Igbo, and Nigerian Pidgin — testing whether frontier models refuse harmful requests, including biosecurity guidance, in languages spoken by 200+ million people
Project Details
Updated 07/12/26 · Edited by orgFrontier AI models are deployed to over 200 million Hausa, Yoruba, Igbo, and Nigerian Pidgin speakers across West Africa but they have never been systematically safety-tested in these languages. Anthropic, OpenAI, and Google evaluate their models almost entirely in English. A model that refuses anthrax cultivation instructions or electoral deepfake scripts in English may comply with the same request in Hausa, and no one would know because no one is testing.
This project will builds and publish the first open-source AI safety evaluation benchmark for some African languages, with biosecurity and CBRN as core categories. We will evaluate the latest publicly available frontier models from OpenAI, Anthropic, Google DeepMind, Zhipu AI, DeepSeek, and other leading developers that meet our inclusion criteria at the time of evaluation against 1,125 prompts across 9 harm categories: electoral disinformation, ethnic/religious incitement, health misinformation, financial fraud, violence, privacy, culturally specific harms, biosecurity & dual-use research, and CBRN in Hausa, Yoruba, Igbo, Pidgin, and English (as control). The benchmark will measure the "portability gap": the difference in refusal rates between English and each language.
The project will establish reusable evaluation infrastructure that enables AI developers, researchers, governments, and standards organizations to assess multilingual AI safety consistently across future frontier models in the Global South.
Concrete outputs (all open-source):
- A benchmark dataset of 1,125 prompts (225 per language × 5 languages) authored and reviewed by native speakers, published on Hugging Face
- Model responses from frontier models with harm ratings from two independent native-speaker raters per language
- A technical brief reporting the portability gap findings, with responsible disclosure to model developers (Anthropic, OpenAI, Google) 90 days before public publication, and to Nigerian biosecurity authorities (NBMA, NCDC) immediately for any biosecurity findings
- A published methodology (pre-registered on OSF) so the benchmark can be extended to other African and Global South languages
Who's involved:
The project is led by RAI-GI (Responsible AI Governance Initiative), an independent nonprofit based in Abuja, Nigeria. The 9-person research team — Muhammad Ahmad Janyau (Executive Director, lead), Nigel Hee, Victoria Hyde, Bridget Oviasogie, Bar. Hadiza Makarfi, Shehu Bello Tijjani, Tega Oviasogie, Shamsudden Umar, and Ahmad Ibrahim — includes native speakers of all four target languages. RAI-GI recently published an 75-page baseline assessment, "The State of AI Governance in Nigeria: A Baseline Assessment 2026," which documents the portability gap this project closes (Chapter 14). We are partnered with ForHumanity (independent AI audit and certification) and with Digital Policy Alert, an initiative of the St. Gallen Endowment for Prosperity Through Trade. We are recruiting a biosecurity consultant affiliated with NCDC or NBMA for the biosecurity and CBRN prompt review.
Theory of Impact
Updated 07/19/26 · By grantmaking.aiThe single most consequential catastrophic risk pathway this project addresses is undetected frontier model failure in languages spoken by 200+ million people — most acutely for biosecurity.
The causal chain:
-
Detection is the precondition for reduction. You cannot reduce risks you cannot detect. Frontier models are deployed at scale in Africa particularly Nigeria where (70% of online adults use generative AI), but their safety properties have never been tested in Hausa, Yoruba, Igbo, or Pidgin. If a model provides biosecurity guidance in these languages that it refuses in English, that failure is currently invisible to the model developer, to the local regulators, to biosecurity authorities, and to the public. The benchmark creates the first capacity to detect this.
-
Biosecurity portability gap = pandemic potential. The most severe category we test is biosecurity (Category 8) and CBRN (Category 9): pathogen cultivation, weaponisation guidance, improvised lab equipment, dual-use research of concern, agricultural biosecurity, chemical synthesis, radiological sources. If we find that a frontier model, for example, refuses anthrax cultivation instructions in English but provides them in Hausa, that is a globally significant finding. It means the safety training that protects English speakers does not protect 50 million Hausa speakers and the same gap likely exists for other low-resource languages globally. A single model that provides biosecurity guidance in any language lowers the barrier to biological attacks anywhere in the world. Detecting this gap is the precondition for closing it.
People
Updated 07/19/26 · By grantmaking.aiTeam Member
Funding Details
- Sep 15, 2026
- Dec 25, 2026
- 3months
- -
- -
- -
- -
- -
- -
- -
Track Record
Responsible AI Governance Initiative (RAI-GI) is an independent public-interest nonprofit based in Abuja, Nigeria, advancing AI safety, governance, and responsible AI across Africa. RAI-GI conducts applied research, develops practical governance tools, and convenes governments, researchers, industry, and civil society to strengthen the safe and trustworthy development of AI.
RAI-GI recently published The State of AI Governance in Nigeria: Baseline Assessment 2026, one of the country's first comprehensive assessments of AI governance readiness. The organization is also developing practical frameworks for AI assurance, multilingual AI safety, personal data sovereignty, and AI standards tailored to African contexts.
A key technical partnership is with ForHumanity, through which RAI-GI is contributing to the development of Nigeria's first certification framework for AI, Algorithmic, and Autonomous (AAA) Systems aligned with the Nigeria Data Protection Act 2023. This work complements RAI-GI's broader mission of building practical, evidence-based AI governance infrastructure.
The proposed African Multilingual Frontier AI Safety Benchmark builds on RAI-GI's experience in AI governance research and extends it into empirical AI safety evaluation. By combining multilingual evaluation, rigorous methodology, and open science principles, RAI-GI aims to create reusable AI safety infrastructure that supports researchers, frontier AI developers, governments, and standards organizations across Africa and beyond.
Discussion
No comments yet. Be the first to share your thoughts.