Evaluating Prompt Injection Defenses in LLMs

Somalia

An independent evaluation of current prompt injection defenses in large language models, producing practical recommendations for safer AI deployment.

Project Details

Nowadays, large language models are being used daily for personal, work, education and health reasons, this has caused such models to hold sensitive information as they hold fast data. Prompt injection has emerged as one of the most significant threats to LLM, where attackers want to take advantage how LLM architecture is designed, where LLM is different from tradional software’s and can’t separate who is user and who is developer, these provide attackers an opportunity to pretend developers, try to manipulate the model and bypass security measures. This study would focus on what solutions exist to prevent such attacks happen and safeguard the model.

These research project would explore various prompt injection techniques, explain the current mitigation strategies for it and compare to what extent current mitigation strategies are able to safeguard the model.

The Research project would lead by me, supported by one thematic expert, one research assistant and one data analyst.

The concrete output would be :

1. Would help LLM developers’ ways to reduce risks poised by prompt injection by offering them matchmaking on which mitigation stragies are suitable for each prompt injection technique

2. The study would help AI Security professionals on understand current risk poised by LLM and effectiveness of their mitigation strategies.

3. Provide practical lessons on organizations adopting Advanced AI

Theory of Impact

OWASP recognized Prompt injection number one threat to LLM, as LLMs are increasingly connected to external tools, sensitive data, and autonomous workflows, if such vulnerabilities remain poorly understood, attackers may exploit this weakness and undermine safety

This project reduces AI risk by providing an independent evaluation of existing prompt injection defenses, identifying which approaches are effective, where they fail, and the trade-offs they introduce.

People

Ismail Magan

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$15,000

Ideal

$25,000

How the money will be spent

The project will be led by me (Principal Researcher), supported by one thematic expert, one research assistant, and one data analyst.

Budget allocation:

Principal Researcher (project leadership, literature review, evaluation design, report writing): $7,000
Thematic Expert (technical review and research guidance): $2,500
Research Assistant (data collection, benchmarking, documentation): $2,500
Data Analyst (analysis, visualization, quality assurance): $2,000
Compute, API costs, software, and publication: $1,000

Ideal Budget (USD 25,000)

The additional funding would allow the team to expand the scope of the research by:

Increasing the involvement of the thematic expert and research staff.
Evaluating a larger set of frontier and open-source language models.
Conducting more extensive testing across diverse prompt injection attack scenarios.
Obtaining independent peer review from AI security experts.
Publishing an open-source evaluation toolkit and supporting documentation.
Organizing dissemination activities, including webinars or conference presentations, to maximize the project's impact.

Discussion

No comments yet. Be the first to share your thoughts.