Reward hacking vs. capability in security RL

Raising $5,000active

Does RL-from-verifier-reward get less safe as models get stronger? A de-confounded measurement using leaky security-patch verifiers and a paired held-out-trigger metric for reward hacking.

Project Details

This is a solo research effort into how self-play and verifier-driven reinforcement learning behave when you point them at security tasks. It continues my MTech capstone directly: I built tinker-rl-lab to scale GRPO fine-tuning experiments and strip out the confounds that usually make their results hard to trust, and I want to apply that same rigor to automated security benchmarks.

The task is security patch generation on reproducible crashes (OSS-Fuzz / ARVO-style). The training verifier is deliberately cheap and leaky — a held-out proof-of-concept no longer crashes and the regression tests still pass — because the gap between "the PoC is suppressed" and "the vulnerability is actually fixed" is exactly the reward-hacking surface I want to study. I measure spec-gaming with a paired, unseen trigger for the same bug: whether the model's "fix" also defeats a trigger it never saw during training turns "did it reward-hack" into a number.

The compute funds a capability-scaling experiment. GRPO fine-tuning runs across a Qwen2.5-Coder-Instruct size ladder, using model size as a clean capability axis, against a matrix of reward structures and seeds (multiple seeds are non-negotiable — de-confounding is the whole point). Two funding tiers:

• Minimum — $5,000 (~2,500 H100-hours, spot). The full cross-validated sweep: Qwen2.5-Coder 1.5B / 3B / 7B across three reward structures and three seeds, plus Llama-3.1-8B-Instruct as a cross-architecture control so the finding can't be dismissed as a Qwen artifact. Delivers the headline result — the proxy-true verifier gap plotted against capability, cross-validated across seeds and architectures. Below this the de-confounding isn't defensible. (H100 80GB currently rents at ~$2–2.5/hr spot/community-cloud; A100 80GB covers the ≤3B runs more cheaply.)

• Ideal — $7,000 (~3,500 H100-hours). Adds a 14B point to extend the scaling curve and extra seeds for tighter confidence intervals — the difference between a clear trend and a publishable one.

Everything ships as one public, fully reproducible GitHub repo: (1) an environment and eval harness pairing open coder LLMs with executable, verifiable security tasks and the paired-trigger reward-hacking metric; (2) training scripts and full logs across reward structures, model sizes, and seeds; (3) a writeup of the empirical findings on verifier robustness and training stability.

Theory of Impact

RL post-training doesn't just occasionally reward-hack. It systematically searches for the gap between what a verifier checks and what we actually want — that's what the optimizer is for. Frontier labs are leaning harder on automated verifiers (RLVR) precisely because human labels don't scale, which makes this gap-finding a dominant force shaping how frontier models behave.

My concrete prediction is that the proxy-true gap widens with capability: a stronger policy is a better verifier-exploiter, so the same RLVR setup drifts further from the true objective as the base model improves, not closer. If that holds, it points at a scaling anti-safety trend — the "just add more automated environments" recipe gets less safe as models get stronger, exactly when we're relying on it most. I test this directly by measuring the gap across the Qwen2.5-Coder size ladder under fixed verifiers.

Security is the right place to measure it because security verifiers are inherently leaky proxies. "The PoC no longer crashes" is cheap to check; "the vulnerability is actually fixed" is expensive and often undecidable. That gap is large, consequential, and — through paired held-out triggers — measurable. A math verifier is near-ground-truth and you'd have to manufacture spec-gaming; here it emerges on its own.

People

Arvind C R

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$5,000

Ideal

$7,000

How the money will be spent

Compute only — this is a solo project with no salary or overhead ask; every dollar buys GPU time.

Minimum ($5,000 ≈ 2,500 H100-hours): the full cross-validated sweep — Qwen2.5-Coder 1.5B / 3B / 7B across three reward structures and three seeds, plus a Llama-3.1-8B-Instruct cross-architecture control — on spot / community-cloud H100 80GB at ~$2/hr, with the ≤3B runs shifted to cheaper A100 80GB. Produces the headline result: the cross-validated proxy–true verifier gap plotted against model capability.

Ideal ($7,000 ≈ 3,500 H100-hours): everything above plus a 14B point to extend the scaling curve and extra seeds for tighter confidence intervals. Includes a small allowance for storage and egress of the full run logs and artifacts that ship with the public repo.

Discussion

No comments yet. Be the first to share your thoughts.