Measuring reward hacking against a physics verifier

Raising $5,000active

Train a model against a real physics checker and measure whether it learns designs that genuinely hold, or exploits in what the checker can't see, on ground truth that costs seconds instead of expert judgment.

Project Details

This project is a testbed for reward hacking, which is when an AI trained against a checker learns to beat the checker instead of actually doing the task. What makes this one different is the domain. It is physical engineering, one of the few places where the checker is cheap and can actually be trusted.

Starting with the setup, it is the same verification loop as my other application in this round. A model proposes a hardware design, and real physics solvers check it for strength, interference, and manufacturability. Anything the solvers cannot verify gets flagged UNCHECKED instead of asserted. My other application measures how reliable that loop is when a model is prompted at it. This application asks a different question, which is what happens when a model is trained against it.

The experiment works like this:

I train a small open model using the checker's verdict as the reward, but I only hand the reward part of the verifier.
The verifier is four separate signals kept independent on purpose, so the rest can be held back as an audit the reward never saw.
I audit every training checkpoint against those held-out signals, against higher-fidelity re-solves, and against eleven design families the system has never seen.

The gap between passing the reward and passing the audit, tracked across training, is the actual measurement of reward hacking.

In addition, a lot of this already works and reproduces. The training environment is built and packaged, and the reward part is validated end to end on a free Colab GPU, giving real 0 or 1 solver-graded rewards at about nine seconds per rollout. The verifier matches textbook physics to about one percent. Not to mention, a model has already driven the untrained loop to a working 44-piece engine on about $100 of rented compute.

Finally, what does not exist yet is the training run itself. No model has ever been trained against the checker. This grant funds the first one, and I publish the divergence curve and the list of exploits either way. Solo, me.

Theory of Impact

Reward hacking is when a model optimizes against a checker and wins by finding exploits instead of actually solving the task. It is the mechanism underneath specification gaming and most concrete scheming stories. The problem is that it is normally studied where the ground truth is expensive, like a human judging the answer or another model standing in as the judge. Because of this the measurement itself is contested, since you cannot cheaply tell whether the optimizer beat the task or just beat the judge.

Physical engineering removes that wall. The ground truth is a deterministic solver verdict that costs seconds and agrees with textbook physics to about one percent. This makes it one of the only places where you can put real optimization pressure on an honest but incomplete checker and actually know, cheaply and objectively, which one won.

The checker here is also fail-closed, meaning a claim that was never measured cannot be shown as passed. The open question is whether that holds up under training. Does training against the gate lead to designs that genuinely hold, or to exploits in the dimensions the verifier does not measure? If gaming shows up even here, against a validated physics checker, that number tells us how much to trust the phrase "it passed the checker" everywhere the ground truth is more expensive, for example code, biology, and alignment research itself. But if gaming does not show up, then the things that prevented it, which are fail-closed gating and a held-out audit, become a tested recipe instead of just a hope. Either way I publish the result.

People

Peter Boctor

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$5,000

Ideal

$25,000

How the money will be spent

$5,000 (minimum): rent the GPU hours for one complete RL training run of a small open code model against the verifier reward, audit every checkpoint against the held-out signals, and publish the divergence curve and exploit taxonomy, flattering or not. The environment and audit set are already built, so this is compute plus my time.

$25,000 (ideal): the same experiment done properly. That means three model scales, multiple seeds, and the ablation that matters, which is training with and without the fail-closed gate to see whether fail-closed verification suppresses gaming or just hides it. RL is hundreds of runs, so the real constraint is iteration time, and rented GPUs meter it since every run costs money and I ration them. If the workstation in my other application is funded, these iterations cost electricity instead and this money shifts to more scales and seeds. Plus a published dataset of every exploit found, so other groups can study the failure mode without rebuilding the rig.

Discussion

No comments yet. Be the first to share your thoughts.