A working benchmark that tests whether frontier models can compute legally binding procurement deadlines under amendments — and catches models that give the right verdict from the wrong clause.
A working benchmark that tests whether frontier models can compute legally binding procurement deadlines under amendments — and catches models that give the right verdict from the wrong clause.
Project Details
Updated 07/04/26 · Provided via application · VerifiedI am building a benchmark that checks one thing. Can AI models calculate legal deadlines correctly, and for the correct reason.
The main trap is simple. A tender says bids are due June 10. Then Addendum 2 changes it to June 14. Bid security must stay valid for 120 days after the deadline, plus 28 more days. If a model uses the old June 10 date, it gets November 5, sees the bidder covered till November 6, and says VALID. The correct answer from the new deadline is November 9, so the bid is actually INVALID. The model sounds confident and quotes clauses, but it is wrong. Companies are already using LLMs to check tenders. Nobody is testing if they can really do this.
What I built so far is public on GitHub with MIT license. 6 benchmark items. A calculator that computes the correct answers from the rules directly, so there are no hand written answers to trust, and CI checks every answer against the math. A strict grader that fails a model even when the verdict is right but the clause behind it is wrong. I tested two real models. The weaker one fell into the June 14 trap. The stronger one escaped it but still made smaller mistakes. So the benchmark can already tell models apart.
I am working solo. With 3 months of funding I will deliver 100+ harder items reviewed by an independent procurement legal expert, a private held out test set, a public leaderboard across frontier models, and a short report. The pipeline is already done. The money goes to content, expert review, and evaluations only.
Theory of Impact
Updated 07/19/26 · By grantmaking.aiThe failure this benchmark identifies goes beyond procurement. It involves models that give the right answer for the wrong reason. In our results, a model might return the correct compliance verdict while relying on a deadline that was changed two addenda ago. Regular accuracy metrics can't detect this issue. Our main metric, right-reason accuracy, can.
As AI systems begin to make legally binding decisions like compliance checks, filings, and contracts the difference between "sounds correct" and "correct for the right reason" becomes a significant oversight problem. If our evaluations only focus on final answers, we end up training the entire system to reward confident pattern-matching.
This project offers two important contributions to AI safety. First, it provides a concrete tool for assessing whether models can be trusted with procedural authority. Second, it introduces a reusable evaluation methodology: generating gold answers with a reliable oracle instead of relying on human labels, using paired controls to reveal pattern-matching, and implementing failure attribution to identify which reasoning step went wrong. I've already used this workflow to spot and fix a bug in my own grader when a cutting-edge model revealed it. Grading the grader is the standard I aim for this benchmark to establish.
People
Updated 07/19/26 · By grantmaking.aiTeam Member
Discussion
No comments yet. Be the first to share your thoughts.