Train a model against a real physics checker and measure whether it learns designs that genuinely hold, or exploits in what the checker can't see, on ground truth that costs seconds instead of expert judgment.
Train a model against a real physics checker and measure whether it learns designs that genuinely hold, or exploits in what the checker can't see, on ground truth that costs seconds instead of expert judgment.
Project Details
Updated 07/08/26 · Provided via application · VerifiedThis project is a testbed for reward hacking, which is when an AI trained against a checker learns to beat the checker instead of actually doing the task. What makes this one different is the domain. It is physical engineering, one of the few places where the checker is cheap and can actually be trusted.
Starting with the setup, it is the same verification loop as my other application in this round. A model proposes a hardware design, and real physics solvers check it for strength, interference, and manufacturability. Anything the solvers cannot verify gets flagged UNCHECKED instead of asserted. My other application measures how reliable that loop is when a model is prompted at it. This application asks a different question, which is what happens when a model is trained against it.
The experiment works like this:
- I train a small open model using the checker's verdict as the reward, but I only hand the reward part of the verifier.
- The verifier is four separate signals kept independent on purpose, so the rest can be held back as an audit the reward never saw.
- I audit every training checkpoint against those held-out signals, against higher-fidelity re-solves, and against eleven design families the system has never seen.
The gap between passing the reward and passing the audit, tracked across training, is the actual measurement of reward hacking.
In addition, a lot of this already works and reproduces. The training environment is built and packaged, and the reward part is validated end to end on a free Colab GPU, giving real 0 or 1 solver-graded rewards at about nine seconds per rollout. The verifier matches textbook physics to about one percent. Not to mention, a model has already driven the untrained loop to a working 44-piece engine on about $100 of rented compute.
Finally, what does not exist yet is the training run itself. No model has ever been trained against the checker. This grant funds the first one, and I publish the divergence curve and the list of exploits either way. Solo, me.
Theory of Impact
Updated 07/19/26 · By grantmaking.aiReward hacking is when a model optimizes against a checker and wins by finding exploits instead of actually solving the task. It is the mechanism underneath specification gaming and most concrete scheming stories. The problem is that it is normally studied where the ground truth is expensive, like a human judging the answer or another model standing in as the judge. Because of this the measurement itself is contested, since you cannot cheaply tell whether the optimizer beat the task or just beat the judge.
Physical engineering removes that wall. The ground truth is a deterministic solver verdict that costs seconds and agrees with textbook physics to about one percent. This makes it one of the only places where you can put real optimization pressure on an honest but incomplete checker and actually know, cheaply and objectively, which one won.
The checker here is also fail-closed, meaning a claim that was never measured cannot be shown as passed. The open question is whether that holds up under training. Does training against the gate lead to designs that genuinely hold, or to exploits in the dimensions the verifier does not measure? If gaming shows up even here, against a validated physics checker, that number tells us how much to trust the phrase "it passed the checker" everywhere the ground truth is more expensive, for example code, biology, and alignment research itself. But if gaming does not show up, then the things that prevented it, which are fail-closed gating and a held-out audit, become a tested recipe instead of just a hope. Either way I publish the result.
People
Updated 07/19/26 · By grantmaking.aiTeam Member
Discussion
No comments yet. Be the first to share your thoughts.