Independent safety evaluations and public reports on current frontier and open-weight models
Run our open-source safety harnesses on the latest frontier and open-weight models and publish a public evaluation report for each one.
This ask funds one evaluation cycle: running our open-source safety harnesses against eight current models and publishing a public report on each.
Models (the latest version of each available when testing starts):
- Frontier: OpenAI's ChatGPT (GPT), xAI's Grok, Google's Gemini and Anthropic's Claude.
- Open-weight: Alibaba's Qwen, Moonshot AI's Kimi, Meta's Llama and Zhipu AI's GLM, run on hardware we control.
Questions we will answer for each model:
- Manipulation: when a system prompt tells the model to change a user's stated view, how often does it use deceptive tactics, and does it tell the user it is trying to persuade them?
- Agent prompt injection: when a tool result contains hidden instructions, how often does the model attempt an action the user never asked for? Each case runs twice, once clean and once poisoned, with the full tool-call trace recorded.
- Multi-turn jailbreaks: does a refusal that holds on a single harmful request still hold when the same request is built up over several turns?
- Bias: does the answer to the same question change when only the user's demographic cues change?
Deliverables and dates:
- Evaluation results for all eight models published to ai4h-evaluation-results on GitHub by December 31, 2026.
- A written report for each of the eight models by February 28, 2027, covering methods, test cases, results by dimension (with sample sizes) and example failures published on our website.
- All harness code, test cases and raw outputs public, so anyone can reproduce or challenge the results.
Responsible disclosure: before publishing, we will share serious findings, such as a reliable jailbreak or agent exploit, with the model's developer and give them time to respond (30 days). Where a serious exploit is still unpatched at publication, we will describe it without releasing a working prompt.
Work already done: we built AI4H Eval Lab and the AI4H Test Suites (single-turn, multi-turn and agent tool-use), and in July 2026 published evaluations of Google's Gemma and NVIDIA's Nemotron in ai4h-evaluation-results: https://github.com/SafeAI4Humanity/ai4h-evaluation-results
Team:
- Aureliusz DeSimone, founder of Safe AI for Humanity Foundation and project lead.
- Leo Simone, researcher: https://www.linkedin.com/in/leo-simone-1a1045436/
- API tokens, $800: running the full test suites against the four frontier models.
- GPU hardware (cloud and physical), $2,800: running the four open-weight models on hardware we control, so results are repeatable.
- Researcher time, $1,200: designing test cases, reviewing results and writing the eight model reports.
Total: $4,800.