Reproducible Safety Evals for LLMs in Medical Reasoning

São Paulo, Brasil

A reproducible evaluation pipeline to audit frontier LLM failure modes, overconfidence, and reliability in medically relevant high-stakes questions.

Project Details

This project aims to answer questions about the risks of using AI systems for medical advice by ordinary users.

I have seen this discussion very frequently on social media. It usually starts with examples and news about the growing number of people who use LLM tools to ask medical questions. This can range from simple questions mentioning symptoms they have, to even sending photos to check whether something may indicate a health problem.

In this type of discussion, two sides always appear. Some people believe in the potential of these tools, especially when comparing them with the quality of current healthcare professionals. Others completely reject this use, pointing to it as a major risk to health.

To try to answer this question, I am developing a project that will evaluate two groups of LLMs. The first group involves the main LLMs currently available on the market, especially frontier models. The second group involves smaller models, including models that can be run locally by ordinary users.

For this, data collection software will be developed from scratch, in order to guarantee reproducibility and data safety. Then, all selected LLMs will be evaluated using questions from the Brazilian national medical exam. These questions are considered complex and non-trivial to answer, which can be confirmed by the results of recent editions, where scores were much lower than desired.

Finally, these data will be used to write a study analyzing failure modes as well as overconfidence. The goal is to provide evidence about safety, performance, and failures in medical diagnosis-related tasks by current LLM models.

The study intends to provide: a pipeline, aggregated data, accuracy analysis, an error taxonomy, and a technical report.

* This text was written by me (Davi Prata) without the help of AI tools. However, due to my current limitations with the English language, I used assistance only to translate the text into English, while preserving the original language, structure, and style.

Theory of Impact

This project contributes to AI safety by building and applying a reproducible evaluation pipeline for frontier LLM behavior in a high-stakes, medically relevant domain. As advanced models become more capable and are increasingly used by non-experts for sensitive decisions, safety work needs better evidence about where models fail, when they appear overconfident, and how their performance changes across languages and local contexts.

The immediate output is not a claim about clinical competence. It is an auditable benchmark workflow and analysis of failure modes using Brazilian national medical education questions as a standardized testbed. This can help the safety community develop more rigorous evaluations for domain-specific reliability, non-English deployment risks, and unsafe overgeneralization from benchmark performance.

In the longer term, better evaluation infrastructure reduces risk by improving how labs, policymakers, researchers, and the public understand the limits of frontier models before they are widely trusted in high-stakes settings.

People

Davi Prata

Team Member

Funding Details

Start Date: -
End Date: Mar 3, 2026
Expected Duration: More 3 months
Funding Raised to Date: -
Annual Budget: -
Monthly Burn Rate: -
Current Runway: -
Funding Goal: -
Funding Stage: Seeking first grant
Fiscal Sponsor: -

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$5,000

Ideal

$10,000

How the money will be spent

Minimum budget ($5,000):

- $1,750 — API credits for frontier model evaluation runs and repeated tests where needed.

- $750 — Cloud infrastructure, storage, logging, backups, and reproducibility tooling.

- $500 — Tooling and documentation work needed to package the pipeline and results coherently.

- $2,000 — Short-term focused work stipend to allow me to complete the pipeline, run evaluations, analyze results, and write the report without pausing the project again.

Ideal budget ($10,000):

- $3,500 — Larger-scale API credits across multiple frontier and frontier-adjacent models, including repeated runs to estimate variability.

- $1,000 — Compute, storage, monitoring, and reproducible artifact hosting.

- $1,000 — Data cleaning, validation, documentation, and preparation of a public/sanitized release where appropriate.

- $4,500 — Focused work stipend over the project period to complete development, analysis, writing, and dissemination.

The minimum version would produce a working evaluation pipeline, a limited but meaningful benchmark run, a structured error analysis, and a technical report. The ideal version would expand the model coverage, improve reproducibility, include more robust error taxonomy work, and produce a stronger publication-ready output.

Discussion

No comments yet. Be the first to share your thoughts.