Project summary

We are planning an online AI safety for builders pilot fellowship to build tools, products, etc.

We have observed that there is lot of research & empirical work happening in AI safety, but much less focus on building products and tools.

This gap has been recognized by several organizations. For example:

• The case for AGI safety products (Apollo)
• RFP for safety-tooling founders (Halcyon)
• Design sketches for a more sensible world (Forethought)
• For-profit AI Safety (Apart)
• AI Safety software incubator (Surplus)
• Builders focused fellowships like Generator Residency, MATS & ERA fieldbuilding streams, BlueDot fieldbuilding/ incubator week, GenStream programmes for generalists, etc

How will help in countering risks?

Broadly, in two ways:

1. Bring AI safety work from research orgs / toy implementation to real world faster.
2. It is also a way to engage countries like India where somehow AI safety ecosystem is very small even though it represents a significant share of the world's population and technical talent. Culturally Indian tech has a builders focus & it can be a huge momentum inducing event for them.

What are this project's goals? How will you achieve them?

Our goal is to help promising AI safety ideas cross the gap between research and real world use. Many useful ideas stop at a paper, proposal, or proof of concept. At the same time, technically strong people want to contribute to AI safety but lack a tractable project, domain guidance or a team.

We want to address both bottlenecks through a 2 month, online, part time AI Safety for Builders Fellowship. Small teams of builders will work with AI safety experts to produce tools, evals and prototypes that can be tested and continued after the programme.

See this list of proposals (collated after surveying various AI Safety + builders) for a hackathon we were planning. These may evolve as we evolve from planning a hackathon to a mentor led fellowship.

Also see a detailed fellowship deck - AI kavach.

Outcome for the fellows

1. Turn important safety problems into buildable projects: Each project will begin with a clear usecase, intended user and scoped deliverable with a mentor.
2. Produce useful artifacts: Outputs can include open-source tools, eval suites, benchmarks, datasets, dashboards, standards, implementation guides or tested product specifications.
3. Develop a pipeline of AI safety builders: We will recruit globally while making a deliberate effort to reach strong technical talent in India and other regions underrepresented in AI safety.
4. Create paths for promising work to continue: either adoption by an org, open-source maintenance, further research, further grants, etc.

Areas where AI safety needs builders

Here are a set of main areas / fields we have identified so far.

How will this funding be used?

[below is for the approved funds of $10,000; however if we get more funds, we'll be able to run a 8 weeks pilot cohort of AI safety for builders fellowship, see details in next section]

• Compute, model APIs and software: $5,000

• Organizer time: $2,000

• Programme operations, platform and communications: $1,500

• Accounting, payment and compliance reserve: $1,000

• General contingency: $500

What will the fellowship look like?

We are thinking to run the pilot online for 8 weeks, with approx 12-18 fellows in 6-8 teams, contributing around 8-12 hours per week. Mentor matching will be based on project needs, complementary skills, preferences, time zones, and working style.

Who is on your team? What's your track record on similar projects?

• Ankur Pandey is a 3 time AI/SaaS founder, builder, and AI safety generalist with 15 years of applied AI experience and over a decade of involvement in EA, rationality, and AI safety, beginning with a MIRI workshop in 2013.
  He started one of India’s earliest large freethinker and rationalist communities in 2010 and has since helped grow the country’s AI safety ecosystem through events, ACX meetups, and an AI safety mixer at the India AI Impact Summit 2026. His recent training includes programmes with BlueDot Impact, CAIS, Cambridge Digital Minds, SPAR, AI Safety Camp, and PIBBSS workshop, where he has worked on AI reasoning failures and integrity disorders in AI systems.
  He also created a comprehensive global directory of AI safety organisations, consulted with Alice on AI security, and currently advises Groundless.ai on epistemic and AI-codependency risks.
  As a founder, he has built products used by hundreds of thousands of people, spanning fake-news detection and general-purpose AI agents, and has received an award from OpenAI. His background also includes public policy and the humanities, complementing his technical and entrepreneurial experience.
• Soumya Jain is an AI governance researcher, research manager, and product leader with experience spanning technical safety, governance, enterprise AI, and programme delivery. She currently serves as Research Manager for the Cambridge AI Safety Hub’s MARS 5 programme, where she helps technical-safety and governance teams do good research.
  Previously, as a MARS 4 research fellow, she led a project on compute policy and AI chip export controls with Erich Grunewald (IAPS). She also supports the Jinesis Lab at the University of Toronto and EuroSafe AI with grant development and research management across AI governance research streams.
  Before moving into AI safety, Soumya spent more than five years in product management and data science across enterprise AI, banking, and regulated fintech. At Terrabase, she led product for an agentic product that transforms siloed business data and documents into governed context, insights, and decisions.
  She has also judged Apart Research’s Global South AI Safety Hackathon, mentored through the Paragon Policy Fellowship, and been selected to mentor an agent-governance project through the Sentient Futures Project Incubator. She founded and facilitated Decoding the Mind, a discussion series on morality, phenomenology, self-reference, and suffering.

What are the most likely causes and outcomes if this project fails?

1st two are more urgent

• Teams don't finish. Some teams may not complete their projects or end up with rough, untested prototypes.
  • Scope is too ambitious.
  • Data or infrastructure isn't available.
  • Teams aren't matched well.
  • Participants drop out.
  • Mentorship is inconsistent.
• Projects aren't useful. Teams build something, but it doesn't solve an important problem or isn't practical to use.
  • Projects are chosen for novelty instead of real need.
  • No clear end user.
  • Practical constraints are discovered too late.
• Nothing gets adopted. A good prototype is built, demoed, and then forgotten.
  • No organisation takes ownership.
  • No one maintains or deploys it.
  • There's no clear path to continue after the hackathon.

How much money have you raised in the last 12 months, and from where?

The project has been funded with 10,000 USD by grantmaking.ai, see link to the project page