IndicViet-Safe: Cross-Lingual Safety Evaluation of Open-Source LLMs

An open-source multilingual safety benchmark evaluates Llama models across South and Southeast Asian languages, revealing under- and over-refusal caused by English-centric safety training.

Project Details

We built a multilingual safety benchmark of 210 prompt-language pairs across English, Hindi, Vietnamese, and Hinglish (code-switched) covering 12 harm categories — including India-specific (caste discrimination, communal incitement) and Vietnam-specific (political sensitivity, censorship circumvention) threats. Evaluating Llama 3.1 8B and Llama 4 Scout 17B, we identify two distinct failure modes: under-refusal (Hinglish safety gap of 44.8pp on Llama 3.1 8B, with 0% refusal for caste discrimination in Hindi) and over-refusal (Llama 4 Scout refuses more in non-English due to reduced comprehension, not better safety). Both stem from English-centric safety training. Code, data, and results are open-source at https://github.com/shch4747/IndicViet-Safe.

Hackathon paper: IndicViet-Safe: Cross-Lingual Safety Evaluation of Open-Source LLMs in Hindi, Hinglish and Vietnamese
Code: github.com/shch4747/IndicViet-Safe

Theory of Impact

Most LLM safety evaluation - both internal to labs and in independent red-teaming - is conducted overwhelmingly in English, even for models being deployed into markets like India and Vietnam where usage is often in Hindi, Hinglish, or Vietnamese. This creates a real deployment gap: a model can look safe by aggregate metrics while systematically failing on region-specific harms (caste discrimination, communal incitement, political censorship evasion) that never show up in English-only testing. Firms deploying into the Global South currently have no accessible, off-the-shelf way to check for this before shipping.

A red-teaming platform built on this research would let firms catch these failures pre-deployment rather than post-incident, without needing to build in-house multilingual safety expertise. Even partial adoption - a handful of firms deploying into South/Southeast Asian markets running this before launch - would meaningfully reduce a currently invisible class of deployment risk.

People

Shourya Choudhary

Team Member

Track Record

Global South AIS Hackathon (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

IndicViet-Safe

Open

Minimum

$2,500

Ideal

$8,000

What this ask is

IndicVietSafe currently evaluates safety behavior of open-source LLMs across English, Hindi, Hinglish, and Vietnamese, and surfaced a real methodological finding: aggregate refusal rates mask large, language-specific safety gaps (a 44.8 percentage-point gap between English and Hinglish on Llama 3.1 8B alone). With funding, I'd extend this from a hackathon-scale benchmark into a rigorous, reusable red-teaming platform that firms can run against their own LLMs before deployment.

Concretely, this means:
(1) closing the methodological gaps identified by hackathon reviewers - native-speaker-verified translations for Vietnamese and Hindi, a consistent judge model across all evaluation runs, and a substantially larger, more balanced prompt set per harm category;
(2) expanding coverage beyond the current 12 harm categories and 4 languages to additional Global South languages and deployment-relevant risk categories;
(3) building an automated evaluation pipeline with a simple interface so a team can point it at their own model and get a structured safety report, rather than requiring in-house multilingual safety expertise;
(4) hosting a working demo so the tool is testable rather than theoretical.

The funding would primarily go toward compute/API costs for running expanded evaluations across more models and languages, and toward paying native speakers to verify prompt and response quality - the single biggest methodological weakness in the current version.

How the money will be spent

API/inference costs for expanded model evaluation (more models, more languages): $2,000
Native-speaker translation verification (Hindi, Hinglish, Vietnamese): $2,000
Cloud hosting for the platform/demo: $1,000
Dataset expansion (additional harm categories and prompts): $1,500
Development tools and miscellaneous (domain, storage, dependencies): $500
Buffer/contingency: $1,000

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate17d?