grantmaking.ai Launch Round
We will develop an open-source tool to secure AI agents. The initial idea is to secure tool and computer usage to prevent malicious attackers from executing harmful actions through the agents. It's a middleware that intercepts the agents actions and enforces security policies. This will be available for developers and AI users who want to use agents safely.
The tool includes security policy decision & enforcement, policy definition, best security practices (to prevent a wide range of attacks and vulnerabilities), logging, evals and benchmarks. The security solution will include:
- Process Isolation
- Taint Tracking
- Audit Logging
- Testing and Red Teaming
- Natural Language Policy Definition -> Formal Verification of Policies
- Policy Recommendations/Presets
- SDK support for agent frameworks
- Human-in-the-Loop
It will be compatible with the main AI agent tools, applications, protocols and frameworks.
We will also write and publish a blogpost or open source arxiv publication.
The team consists of 3 developers with experience in cybersecurity, penetration testing, AI applications, cryptography, and ML.
The money will be spend to provide the team with the necessary tools for development, testing, evaluation, and benchmarking, as well as a developer stipend.
The minimum duration for the project is 6 months, and ideally 9 months. The ideal conditions will allow us to more comfortably develop the tool, get feedback from users and make improvements in response, and to develop more extensive use cases and benchmarks.
Spreadsheet: https://docs.google.com/spreadsheets/d/168H6uzkYum01yAlus-uTeEseoXcIVRu2FwmNKx2NIog/edit?usp=sharing
We also accept resources like API credits, etc.. instead of directly money if it satisfies the requirement.