Quaver: adversarially-verified RL environments

Malaysia

A tool that generates RL environments for AI agents and adversarially attacks each one, so models don't train on tasks they can cheat.

Project Details

In a nutshell, Quaver builds environments for training AI agents and then tries to break them.

An RL environment is a task an agent attempts, with an automatic grader that scores whether it succeeded. Labs run these thousands of times to train their agents, each attempt is scored, and that score is the training signal. The problem is that a bad environment, one an agent can pass without actually doing the task, teaches the model to cheat instead of to work.

Train on enough broken environments and you get a model that's learned to exploit loopholes. Or worse yet, lie, cheat, scam and deceive to get what it wants (see Andon Labs' Vending-Bench experiments, where a frontier model told to maximize profit colluded on prices, lied to suppliers, and told a customer she'd been refunded when she hadn't). And the tools generating these environments usually can't tell when they've made a broken one, because the same model that writes the task doesn't know what's genuinely hard, what's fair, or where it's leaking the answer.

Quaver addresses just that! It generates an environment from a description, then runs an adversarial gate against it, including:

a correct solution
a null agent that does nothing
and a set of probes that actively try to reward-hack the task.

The environment only passes if a cheating agent can't score well on it. The point is to catch gameable environments before anyone trains on them.

It's already built and working. It has generated a dozen environments across domains like financial operations, compliance, and incident response. One of them caught a frontier model making an unsafe economic decision in a simulated gig-economy setting.

Theory of Impact

Quaver attacks RL environments before models train on them, so gameable environments get caught instead of shipped. This matters because models are now trained with RL to achieve goals, and they learn whatever the reward actually pays for.

When the environment is gameable, the reward pays for cheating, and the model learns to cheat at training scale which eventually gets baked into weights.

And scarily, this is already happening!

Grok hard-coded the example test ROMs into its emulator to fake passing GBA Eval.
Datacurve audited SWE-bench Pro and found its verifiers misgrade about a third of trials, with frontier models passing by reading the answer out of git history instead of solving the task.
And again, Andon Labs told Opus 4.6 to maximize profit on Vending-Bench and it colluded on prices, lied to suppliers, and told a customer she'd been refunded when she hadn't.

This is misalignment! (just at smaller scale, for now)

Nobody checks environments for this systematically.

Researchers find gameability by hand, one environment at a time, usually after something goes wrong.

People

Faw Ali

Team Member

Funding Details

Start Date: Apr 24, 2026
End Date: -
Expected Duration: 1 year
Funding Raised to Date: -
Annual Budget: -
Monthly Burn Rate: -
Current Runway: -
Funding Goal: -
Funding Stage: Seeking first grant
Fiscal Sponsor: -

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$25,000

Ideal

$75,000

How the money will be spent

Minimum

Mostly compute. Testing environments against the strongest models costs money per run, and development needs hundreds of runs. An environment is only useful if the best models fail it, so every test cycle has to include them. I recently tried to save money by calibrating against free models instead, and ended up with an environment that every frontier model passes. The rest of the compute goes to generating new environments and running the cheat-detection gate on each one.

The remainder buys me more time. I cut contracting hours and work on this consistently for about six moths instead of fitting it around a full workload.

Ideal

Bigger scale! More test runs, way more repeated runs per model so results are much more reliable, and longer experiments over long-horizon tasks. Also a lot more of my time on this and less on contracting, closer to treating it as my main job for nine months to a year.

Discussion

No comments yet. Be the first to share your thoughts.