Fixing flaky AI benchmark scorers with eval-invariance-engine

Fixing flaky AI benchmark scorers with eval-invariance-engine

I kept seeing the same thing — shuffle MMLU answer choices from A/B/C/D to D/C/B/A, or add a trailing newline, and the same model jumps 3-6 points. Same harness, same seed. That's not measuring reasoning, it's measuring my regex.

I built eval-invariance-engine because national safety harnesses (like UK AISI Inspect) decide ASL-3 triggers on these scores. If the scorer isn't invariant, the leaderboard isn't real.

It's a small local CLI. You point it at a scorer function, it runs 1,000+ Monte Carlo perturbations (whitespace, case, choice order, newlines) and tells you if variance is actually zero. I've been using it on my own evals — 31 property tests with Hypothesis, 97% coverage, green CI on 3.12.

Repo is already built and public: https://github.com/j-arndt/eval-invariance-engine — not a pitch deck, it's working.

What I'll ship in 45 days

I've got the core working. Funding pays for the boring integration work that makes it usable for others.

First, a one-flag adapter for Inspect so anyone can do inspect eval --invariance-check. Then I'll audit 5 open suites (starting MMLU and GSM8K) and publish where they break — I've already found a regex that fails on trailing spaces. If we hit scale, I'll run 100k+ trials across 20 frontier + open models and put the dataset on Hugging Face, plus a short arXiv note on the common failure patterns. I want this to be a pre-flight check you run before you trust a number.

How funding will be used — $38k target

$8k minimum gets Inspect adapter + 2 suite audits shipped. $20k gets 5 suites + 10 models. $38k gets full 20-model, 100k trial dataset + paper.

I flipped the budget from salary-heavy to compute-heavy after seeing what actually got funded here:

Engineering: $10,500 for 30 days focused work

Model API and compute pool: $15,000 for 100k+ trials across 20 models — this is the real cost

Cloud runners and storage: $2,500

Hugging Face and arXiv: $1,000

Infrastructure buffer: $9,000 for extra API for stretch audits

Total: $38,000

Previous draft was $25k with 63% salary. This version is compute-first, like the Bayesian modelling project that got $32k and unlocked the interesting results.

Proof of work

Full Python 3.11+ implementation, Pydantic v2, typed. Perturbation engine does whitespace, case, newline, and choice reorder — all seeded and reproducible. 31 unit/property/stress tests, 97% coverage, stress tested to 10k iterations. Click CLI with audit, demo, and report commands, plus Dockerfile and GitHub Actions. Every audit gives a tamper-evident SHA256 receipt.

If this fails

Most likely is no one runs the check — eval authors keep their regex. I'm mitigating by building inside Inspect so there's zero new tool to install. Second risk is API flakiness for the multi-model dataset — I cache every permutation locally so reruns are free. Either way the audits and receipts stay public and MIT-licensed.

About me

I'm Justin — independent systems/architecture background, built this self-funded. No prior Manifund. Link to proof above. I've talked to a couple Inspect users who want this — aiming to get a comment from them here.