TENDER-TIMELINE: A verifiable benchmark for procedural legal reasoning

Raising $9,000active

A working benchmark that tests whether frontier models can compute legally binding procurement deadlines under amendments — and catches models that give the right verdict from the wrong clause.

Project Details

I am building a benchmark that checks one thing. Can AI models calculate legal deadlines correctly, and for the correct reason.

The main trap is simple. A tender says bids are due June 10. Then Addendum 2 changes it to June 14. Bid security must stay valid for 120 days after the deadline, plus 28 more days. If a model uses the old June 10 date, it gets November 5, sees the bidder covered till November 6, and says VALID. The correct answer from the new deadline is November 9, so the bid is actually INVALID. The model sounds confident and quotes clauses, but it is wrong. Companies are already using LLMs to check tenders. Nobody is testing if they can really do this.

What I built so far is public on GitHub with MIT license. 6 benchmark items. A calculator that computes the correct answers from the rules directly, so there are no hand written answers to trust, and CI checks every answer against the math. A strict grader that fails a model even when the verdict is right but the clause behind it is wrong. I tested two real models. The weaker one fell into the June 14 trap. The stronger one escaped it but still made smaller mistakes. So the benchmark can already tell models apart.

I am working solo. With 3 months of funding I will deliver 100+ harder items reviewed by an independent procurement legal expert, a private held out test set, a public leaderboard across frontier models, and a short report. The pipeline is already done. The money goes to content, expert review, and evaluations only.

Theory of Impact

The failure this benchmark identifies goes beyond procurement. It involves models that give the right answer for the wrong reason. In our results, a model might return the correct compliance verdict while relying on a deadline that was changed two addenda ago. Regular accuracy metrics can't detect this issue. Our main metric, right-reason accuracy, can.

As AI systems begin to make legally binding decisions like compliance checks, filings, and contracts the difference between "sounds correct" and "correct for the right reason" becomes a significant oversight problem. If our evaluations only focus on final answers, we end up training the entire system to reward confident pattern-matching.

This project offers two important contributions to AI safety. First, it provides a concrete tool for assessing whether models can be trusted with procedural authority. Second, it introduces a reusable evaluation methodology: generating gold answers with a reliable oracle instead of relying on human labels, using paired controls to reveal pattern-matching, and implementing failure attribution to identify which reasoning step went wrong. I've already used this workflow to spot and fix a bug in my own grader when a cutting-edge model revealed it. Grading the grader is the standard I aim for this benchmark to establish.

People

Gudavalli Teja

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$9,000

Ideal

$18,000

How the money will be spent

Solo researcher, 3 months. The evaluation pipeline is already built and open-source (MIT), so funding goes entirely to what code can't do: authoring harder items, expert legal review, and frontier-model evaluation.

Minimum ($9k): $6k part-time stipend to create ~50 harder items and a private held-out split; $1.5k for procurement/legal expert review; $1.2k for API costs to evaluate 3–4 frontier models; $300 for hosting/CI.

Ideal ($18k): $13k near full-time stipend to create 100+ items with full expert review; $2.5k for expert QA; $2k for repeated API evaluations to improve confidence in the results; $500 for infrastructure.

Same deliverables either way: a public benchmark, held-out split, leaderboard, and report. The ideal budget simply delivers more items and more reliable evaluation results.

Discussion

No comments yet. Be the first to share your thoughts.