BioRAG-Guard: Poisoning Evals for Biological AI Agents

Raising $28,000active

An open-source benchmark and defense toolkit for testing whether corrupted biological databases can hijack retrieval-augmented AI agents used in genomics, protein science, and single-cell analysis.

Endorsed by

Ryan

Marcus+1

An open-source benchmark and defense toolkit for testing whether corrupted biological databases can hijack retrieval-augmented AI agents used in genomics, protein science, and single-cell analysis.

Endorsed by

Ryan

Marcus+1

Project Details

In this project we will develop BioRAG-Guard. An open-source benchmark and defense framework which evaluates data-poisoning risks in biological AI systems. Can a small number of malicious or corrupted database records influence what a biological AI agent retrieves and change the agent's conclusions? This is the main research question of the project.

We will evaluate different representative biological retrieval architectures. More specifically, protein embedding retrieval based on models such as ESM-2, single-cell similarity search using SCimilarity, reference mapping and nearest-neighbor label transfer using scArches and finally retrieval-augmented prediction of cellular perturbation responses. These are systems span across different biological data and different ways in which scientific agents rely on external evidence.

For each of the systems we will:

Construct controlled and non-hazardous poisoning scenarios using public or synthetic data. The evaluation will measure whether manipulated records a)entered the highest ranked retrieval results b)change a predicted label or a biological interpretation c) transfer across embedding models / evade quality control procedures. The benchmark will distinguish between retrieval failure (corrupted information returned) or an outcome failure (corrupted information changes the answer of the agent).
2)Implement practical defenses. These will include source and dataset provenance, visioned by database snapshots, anomaly detection across metadata /embeddings, quarantine periods for recently added records. Conclusions must be validated by multiple independent records.

The main outputs of the project will be an open-source evaluation that will include a collection of safe and reproducible poisoning scenarios, baseline results across multiple biological retrieval systems and implementation of several different defenses. Additionally, we will supply documentation for developers and database maintainers and an academic paper of the projected.

The project will NOT involve a) pathogen engineering, harmful biological sequence design or wet lab experimentation.

Theory of Impact

Future biological AI systems may move beyond answering scientific questions and begin autonomously retrieving evidence, comparing biological sequences, classifying experimental samples, prioritizing hypotheses, and recommending laboratory actions. As these systems become more capable, their behavior will depend not only on the underlying AI model but also on the integrity of the databases, vector indexes, and reference collections from which they retrieve information.

A well-trained and well-aligned model can still produce dangerous or misleading outcomes if the evidence presented to it has been manipulated. A corrupted protein annotation, single-cell reference profile, or perturbation record could influence an agent’s reasoning while appearing to be legitimate scientific evidence. This creates a potential pathway through which an external attacker, compromised data source, or systematic database error could redirect the behavior of a more autonomous biological AI system.

Most current biological AI safety work focuses on the capabilities and outputs of the model itself. Examples include whether a model provides harmful biological knowledge, whether it follows unsafe requests, and whether hazardous capabilities can be removed or controlled. These are important questions, but they do not fully address the retrieval layer. As scientific agents become more dependent on external tools and databases, retrieval integrity becomes part of the safety boundary.

People

Michail Patsakis

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$28,000

Ideal

$38,000

What this ask is

Biological AI systems are increasingly being connected to external scientific databases. These systems do not only answer from the information that is contained in a model but also retrieve protein sequences, cell profiles, gene annotations and experimental results before recommending an action to the user. This retrieval can make biological AI a lot more accurate but also creates a huge risk. The model can treat retrieved information as 100% trusted evidence even when that information is incorrect, compromised or even deliberately manipulated!

In this project we will develop BioRAG-Guard. An open-source benchmark and defense framework which evaluates data-poisoning risks in biological AI systems. Can a small number of malicious or corrupted database records influence what a biological AI agent retrieves and change the agent's conclusions? This is the main research question of the project.

We will evaluate different representative biological retrieval architectures. More specifically, protein embedding retrieval based on models such as ESM-2, single-cell similarity search using SCimilarity, reference mapping and nearest-neighbor label transfer using scArches and finally retrieval-augmented prediction of cellular perturbation responses. These are systems span across different biological data and different ways in which scientific agents rely on external evidence.

For each of the systems we will:

Construct controlled and non-hazardous poisoning scenarios using public or synthetic data. The evaluation will measure whether manipulated records a)entered the highest ranked retrieval results b)change a predicted label or a biological interpretation c) transfer across embedding models / evade quality control procedures. The benchmark will distinguish between retrieval failure (corrupted information returned) or an outcome failure (corrupted information changes the answer of the agent).
2)Implement practical defenses. These will include source and dataset provenance, visioned by database snapshots, anomaly detection across metadata /embeddings, quarantine periods for recently added records. Conclusions must be validated by multiple independent records.

The main outputs of the project will be an open-source evaluation that will include a collection of safe and reproducible poisoning scenarios, baseline results across multiple biological retrieval systems and implementation of several different defenses. Additionally, we will supply documentation for developers and database maintainers and an academic paper of the projected.

The project will NOT involve a) pathogen engineering, harmful biological sequence design or wet lab experimentation.

How the money will be spent

Minimum budget:

$21,000: Researcher support and protected research time for system implementation, benchmark design, experiments, analysis, documentation, and preparation of an academic paper.

$5,000: Computing infrastructure such as GPU access, cloud and storage costs, HPC environment.

$2,000: Review by researchers with relevant biological backgrounds.

Ideal budget:

$4,000: Part time research engineering student for experiment automation.

$2,000: Reproducibility and security review, documentation, verification that benchmarks can run independently.

$2,000: Dissemination and maintenance, including publication costs/ conference participation

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Grant ReviewerPrivate5d?

KIMON ANTONIOS PROVATASEndorsement1d

Great idea!