Blindfold: Auditing Vietnamese LLM Safety Using Secure Enclaves

Builds a blind-audit harness and bilingual Vietnamese local-harms benchmark to evaluate multilingual refusal gaps in frontier AI models without exposing private weights, data, or code.

Endorsed by

Suprita

Builds a blind-audit harness and bilingual Vietnamese local-harms benchmark to evaluate multilingual refusal gaps in frontier AI models without exposing private weights, data, or code.

Endorsed by

Suprita

Project Details

Frontier labs mainly carry out red-team safety evaluations in English, on globally recognized harms. This leaves two blind spots: local or regional harms, and a non-English refusal gap where a model refuses an English request but complies on the identical one in another language — Deng et al. (ICLR 2024) measured ChatGPT unsafe 0.63% in English versus 7.94% in Vietnamese on identical prompts. doing such evaluations on real frontier models requires three mutually distrustful parties: an AI Lab with private weights, an AI Safety org with a private benchmark, and an Auditor with eval code to cooperate without revealing their secrets to one another. We built (i) a blind-audit harness: a code-to-data flow on OpenMined’s syft-client where weights, benchmark, and code meet inside a sealed enclave that both data owners review and approve, and only a signed scorecard exits; and (ii) a 47-prompt bilingual (EN↔VN) local-harms benchmark, every harmful prompt citing a real Vietnamese source. Running the harness across four models (qwen2.5-0.5b/3b, phogpt-4b, seallm-v3-7b), we found the Vietnamesespecialized model the least safe in Vietnamese: it refused only 14% of harmful prompts in VN versus 38% in English — the worst gap of any model — while safety otherwise tracked size and alignment, not language coverage. Auditing in English alone would have over-rated exactly the model marketed for Vietnamese. All data and code are at github.com/khoaguin/blindfold.

Hackathon paper: Blindfold - Blindly Auditing the Vietnamese LLM Safety Blind Spot using Secured Enclaves
Code: github.com/khoaguin/blindfold
Slides: Slides
Recording: Presentation recording 1, Presentation recording 2, Presentation recording 3

Theory of Impact

Rigorous safety audits are deadlocked by a trust problem: labs cannot hand over weights, safety orgs cannot publish benchmarks without turning the exam into training data, and auditors are trusted by neither. Blindfold shows a working mechanism to solve this problem: a hardware-attested enclave where all three secret assets meet and only scores come out. No one needs to trust anyone else for the evaluation to happen.

The impact path is twofold. First, the concrete example: Vietnamese-language safety is essentially unmeasured — existing work is hate-speech classification, and Vietnam's own models publish no adversarial safety numbers — and our first audit already found a Vietnamese-tuned model refusing an order of magnitude less than its peers. Second, the general case: the same architecture makes held-out, natively-authored benchmarks viable for every small safety org in all other low-resource languages, because custody stops being a cost of doing the work and becomes a feature of it

People

Khoa Duy Nguyen

Team Member

Track Record

Global South AIS Hackathon (Apart Research), 2026

Grants Received– no grants recorded

Add grant

Funding Asks

Blindfold - Blindly Auditing the Vietnamese LLM Safety Blind Spot using Secured Enclaves

Open

Minimum

$10,000

Ideal

$30,000

What this ask is

Blindfold showed that a three-party zero-trust safety audit where secret weights, secret benchmark, secret eval code can run end-to-end using secure enclaves, and only final scores leave the enclave. It surfaced a real finding: the Vietnamese-specialised model phogpt-4b refuses only 12–15% of coherent harmful Vietnamese requests, versus 46–85% for comparable models.

With the funding, we will first, grow the benchmark from 47 to ~200 prompt pairs, authored by paid Vietnamese safety practitioners rather than translated from English sets, with the benign bucket expanded to more than 20 prompts to measure over-refusal, not just refusal.

Second, improve our experimental results. Our original headline finding (the hackathon project) changed when we re-scored it with a different judge. Hence, we'd extend the audit to more Vietnamese-specialised models and at least one frontier model, to test whether the pattern we found — Vietnamese fine-tuning eroding safety — holds beyond a single model.

Third, we will deploy the full protocol on GCP Confidential Space with hardware attestation, with a strong open-weights model, e.g. a 70B-class Llama or Qwen, running inside the enclave at realistic scale. We would still play all three roles ourselves, but the enclave, the attestation, and the model would be real.

Fourth, scale from one auditor to many: integrate with ScreamingFace (https://github.com/ScreamingFace/screamingface), an open-source ensemble-evaluation stack and public leaderboard, to demonstrate that multiple AI safety organizations can each contribute a held-out benchmark to the same audit. If that integration lands, it also opens a direction we'd further explore: making the audit target an ensemble of models in the enclave, not just a single model.

How the money will be spent

Lead researcher stipend, 6 months part-time (~50%), including benchmark authoring: $18,000
LLM-judge API costs (two-judge scoring, multi-seed, 6+ models): $3,000
GCP Confidential Space + confidential-GPU compute for 70B-class enclave runs: $7,000
Contingency: $1,000

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Private UserPrivate15d?

Suprita ShankarJudge · Global South AIS HackathonEndorsement15d

One of the best entries. It lets a lab, a safety team, and a checker who don't trust each other run the same test without showing each other their private data, and it adds a set of Vietnamese harmful prompts to test with. The main finding stands out: the model made for Vietnamese is the least safe in Vietnamese — a problem an English-only test would never catch. As you note, the numbers are early: one run, a small setup, 47 prompts. Next: run it more times, use a checked way to score answers, and do one full run on the real secure system. Also fix a few typos.