CrucibleBench: Using MUDs for Long-Term, Persistent AI Evaluation

This is Phase 2 of an existing experiment, CrucibleBench https://zenodo.org/records/21386663 Phase 1 asked whether a MUD could produce useful behavioral differences across models. Results indicated yes but a larger study would be required. Phase 1 also showed that LLM-based judges distort scores (sometimes significantly) and so Phase 2 will seek to improve the scoring instrument and examine weaknesses in LLM-based judges while providing a human baseline.

Phase 2 moves CrucibleBench from a proof-of-concept towards platform validation. It will test model reliability, test environmental discrimination between models, examine whether the intended model abilities are actually being measured (social adaptation, planning, memory, deception, etc.), configuration sweeps, and whether the MUD environment provides the information needed to evaluate models.

Who's working on it includes myself and my co-author from the whitepaper, as well as Dr. Matthew Morris, also named in the whitepaper. Additional funding will allow me to contract additional statistical and engineering support.

Concrete outputs include: Both public and hidden evaluation suites, preregistered evaluation protocols, a calibrated scoring package, MUD-specific ablation results, a Phase 2 validation corpus, benchmark validation report, reproducibility release, and an explicit go/no-go decision for future development.