In today’s world the LLM judges are more and more present and are here to stay and they decide from a multitude of options, witch outputs are worth keeping and witch are not(for example the scoring step in RLAIF, Constitutional AI, and most automated evals). How can you make sure that they are not pushing forward fabricated truths because they sound good? That was the question that started this project.
I took 200 outputs from my own production pipeline, each mechanically verified to contain fabricated claims, and tested the judges on them across four model gpt-oss-120b, Llama-3.3-70b, grok-3-mini, and claude-haiku-4.5. We tested 2,000 judgments total (10 runs × 200 items). The technology I have created makes sure that every result I get is sealed on the Bitcoin blockchain (OpenTimestamps), and also that the decisive follow-up tests were pre-registered. The interpretation thresholds are anchored before a single call was made, so the methodology couldn't be tweaked later on.
What I have found is that the judges aprove a huge amount of fabricated information. For example when they do not have a source text to double-check, judge models approved between 43% and 100% of confirmed fabrications and every single model failed, some just failed worse than others. When providing a source text to double check, the LLM judges aproval becomes like a lottery – for example when responding to the same instructions one model had 0% leakes, while another had 65%.
The next step was that I forced the judge (using gpt-oss) to break down the text line-by-line while citing the exact supporting quote or explicitly marking it missing. The result was the failure rate dropped from 11% (that same judge's with-source baseline) down to 0.5% and no single fabrication survived every defense we tested.
I didn't rely on another AI's opinion to label what was fake but instead, every claim was broken down and checked against source material, then validated against a separate human-anchored dataset.
I first came across this problem while building autonomous AI systems that deliver a huge amount of outputs and creating an open-source auditing tool for them (BIJOTEL). This project aims to those findings into a citable research paper alongside open and verifiable datasets as well as a practical test harness so anyone can measure their own AI judge before taking their word for it.
Project Goals
A few recent papers have been noticing the same problem: LLM judges reward writing that sounds right over factual acuracy. "More Convincing, Not More Correct" (arXiv:2607.05904) caught reward-hacking emerging in self-play whenever the judge had nothing to check answers against. "Faithful or Fabricated?" (arXiv:2605.23970) found a similar rationalization bias, though they only looked at static summaries.
What nobody has answered yet, as far as I can tell, is this: how badly do the standard grounding fixes fall apart once you swap out the judge? Is there any protocol that actually holds up when the output is flat-out fabricated? Nobody has a good answer to that yet, and it's what I want to find out. My approach is pretty direct: first measure how much the verdicts change when you swap one judge model for another, then test a counting-based protocol against that variance. So far, in my own runs, the counting protocol closes the gap. I want to stress "so far" — that's exactly what the funded validation is meant to confirm or kill. Either way, everything I publish comes with provenance anyone can check without trusting me.
Concretely, the project produces a preprint (target venue is a scalable-oversight workshop, and the cross-model matrix is the centerpiece of it) plus two datasets and a test harness.
The datasets are MIT/CC licensed. The first is the 200-item cross-model matrix — 2,000 judgments total, since each item gets 10 runs. The second is a historical corpus of 2,091 pairs I've been accumulating. I timestamp both through OpenTimestamps on Bitcoin, and the extraction rule is frozen and public. There's an annoying copyright wrinkle: some source texts I legally can't redistribute, so for those I publish a link and a hash instead. Not ideal, but it keeps the whole pipeline verifiable end to end, which is the point.
The harness takes any OpenAI-compatible endpoint and runs the judge through three baselines (blind, grounded, forced-counting), sealed so runs can be reproduced exactly. You shouldn't need to fight with setup to use it.
Now, what the money is actually for. Mostly validation work I can't fit into spare evenings. Two pieces: extending the counting protocol to more model families — right now gpt-oss is the only one I've tested, which is obviously not enough to claim generality — and expanding the human-labeled held-out set that everything gets scored against.