Project summary

PROBE develops a local pipeline of AI evaluators who can design, run, and critique safety-relevant evaluations of frontier and open-weight AI systems. We plan to start in Asia/greater China region, where capable open-weight models ship fastest and there is a strong need for evaluators.

The PROBE workshop pairs  hands-on evaluation design talk with an interactive audit game, which feeds an open-source practitioner handbook. We are requesting $23,000 for a pilot of two workshops in Asia (December 2026 – Q2 2027), supporting the learning of 40–70 participants looking to upskill evaluation. We created PROBE for the following reason:

• Evals are how labs, auditors and governments see dangerous behavior before deployment, and evals are getting harder to build well.
• China is a major producer of open-weight models with a stated government mandate for AI safety, but the AI safety ecosystem is relatively small. This holds true across greater Asia, where the AI safety hubs are still in development stage.
• There is a lack of support for independent, safety-focused evaluator commons. That is the gap PROBE fills.

Safe deployment of advanced AI is bottlenecked by lack of independent evaluation talent who can turn standards into rigorous, safety-aligned evaluations. Best practices already exist in the US, UK/EU, and China. However, adoption remains limited because implementation is nuanced and the talent is scarce. As capabilities grow, especially in agentic systems, measuring risk needs more care in design and oversight. Frontier evaluators such as METR and Apollo Research increasingly point to the limits of current measurement. There are relatively fewer AI safety aligned hubs in Asia that can support evaluation activities and information exchange. China is shipping increasingly capable open weight models that need corresponding development of “human in the loop” talents for evaluation.  Delegating evaluation and oversight to LLM-based monitors and classifiers needed a stronger human in the loop approach since LLM based judges have known failure modes.  Human judgement and evaluators remains essential at high-stakes checkpoints, which requires skilled and trained evaluators.

What are this project's goals? How will you achieve them?

Each PROBE session takes 20–40 ML engineers, researchers and risk professionals from reading about evals to designing and critiquing one. The format was piloted at PROBE Berlin in October 2026.

The Eval jam in Shenzhen aim to create the same workshop environment in a conference setting in Mushenzhen's Maker City (https://mushenzhen.xyz/), which convenes hundreds of global developers, researchers, and builders. We also plan for another iteration in 2027 pending funding. In the design jam**,** small teams learn about the latest eval tools and research.  Participants have the option of scoping a safety-relevant eval from a real case, covering threat model designs and human-in-the-loop checkpoints. We will also have game instructions and a facilitation guide so future partners can run similar PROBE independently.

Impact Assessment (+ 6 months)

Workshops delivered in Asia

2 (minimum)

Participants reached

40–70

Open source Resources

1 audit game kit and handbook from evaluation materials reviewed

Outcome

• Gain in participant eval design skill and awareness of tools and techniques
• Participants applying eval design or safeguards at work (6-month follow-up): over 75%

How will this funding be used?

Overall Series Budget ($23,000 Total)

PROBE Shenzhen (Dec 3, 2026): $7,500

Venue support, speaker and organizer travel & honoraria compensation, game printing, and documentation

PROBE Iteration 2 (2027) $6,500

Second city deployment (venue, speaker travel, compute, local logistics)

Series Infrastructure $3,500

Support for community contributors for open Facilitation Playbook, interactive game kits, evaluator network support

Working-Group Seed & Content Curation Fund $5,500

Support for community contributors for capturing, structuring, and curating expert workshop inputs for workshop and handbook integration

Total Funding Request. $23,000

Who is on your team? What's your track record on similar projects?

• Min-Si Wang (Program & Coordination): project strategy, overall event execution,  and workshop curriculum development. Background: Product and research lead in fintech and risk analytics. Main curator of www.aisafetynode.com for cross border AI safety development and field building. https://www.linkedin.com/in/min-si-wang-867395a/
• Jaspreet Singh (Program Design): Role: Design workshop format, evaluation methodology, workshop activity structure, and participant experience. Background: experienced product and design lead experience. https://www.jaspreetsingh.net/#projects
• Bryan Chan (Developer & AI Safety Specialist): Role: Design and development of the interactive audit game kit, evaluation design jam exercises, and theory of change alignment. Background: AI Safety researcher with background in evaluation(https://www.linkedin.com/in/cnm13ryan/)

What are the most likely causes and outcomes if this project fails?

Downscaled reach for Evaluation Training: We will reach less participants due to curation of fewer workshops. The breadth and quality of the workshop will also be less comprehensive in scope in terms of the interactive activities and followup resources.

How much money have you raised in the last 12 months, and from where?

We have not raised funding so far.