Scale Up: AI Development Index & Model Evals for Fabrication Benchmark

Raising $12,600active

Expanding a continuously updated database covering global panel AI data as a ground-truth benchmark which allows stress-testing frontier LLMs’ fabrication rates in global contexts (lower-income and upper-income countries’ contexts

Project Details

In November 2025, I founded the milestone-based Global AI Dataset (GAID) Project. I set up the site AI in Society to release subsequent GAID Project outputs. I carried out data-engineering work to harmonise, compile, curate and document the GAID w1 (v1 and v2) datasets (deposited on Harvard Dataverse; which GAID dataset is updated annually at the end of each year). GAID w1 v2 dataset contains over 24,000 AI indicators, covering 227 countries/territories across 20 AI domains from 1998 to 2025. Recently, I finished building Phases 3 and 4 of the GAID Project upon my GAID w1 v2 dataset. Phase 3 is to programmatically present the GAID AI Development Index ranking global countries’ AI capacity that covers multiple thematic pillars. Phase 4 is a model-evaluation benchmark that scores 11 frontier LLMs’ (both open-weight and proprietary) responses against GAID ground-truth data. The goal is to evaluate which LLMs are subject to higher fabrication rates, and also whether and which LLMs are more likely to fabricate in the contexts of lower-income countries relative to their upper-income counterparts (so as to address AI-driven digital colonisation concerns). I built Phase 4 upon two peer-reviewed pilots, one published on Apart Research and the other accepted by IEEE IES for conference presentation at the IEEE IRAI 2026 in Melbourne this September.

I would like to apply for 5-month funding to scale up both Phases 3 and 4:

Phase 3: Harmonise and map GAID with global panel socioeconomic data (such as World Bank Open Data), and publish the compiled and documented dataset on Harvard Dataverse for public download and/or API extraction, so as to facilitate worldwide research on how AI capacity, adoption, safety, readiness and beyond relate to socioeconomic development in global contexts
Phase 4: Publish the GAID w2 dataset in late 2026 (expected to cover data across 30+ AI domains); extract more indicators from GAID w2 dataset across the thematic pillars (AI accountability, adoption, ethics, fairness, regulation, safety, security, transparency); stress-test a wider set of Western and Chinese frontier LLMs (both open-weight and proprietary)
Additional Outputs: Currently in discussion about collaboration with senior Cambridge academics within my academic network to co-author two research papers, one introducing the open-access GAID methodology and AI Development Index and the other presenting the open-source model evaluation methodology and empirical findings

Note: I only cover publicly accessible, non-paywalled data in this GAID Project. As always, all code is and will be open-sourced.

Theory of Impact

LLMs are increasingly deployed as monitors in AI control protocols and to advise and facilitate international governance. My peer-reviewed pilots already show that frontier LLMs fabricate verified facts (i.e. GAID data), with the presence of geographical hallucination working against the interest of the Global South. LLMs’ fabrication, especially in Global South contexts, results in the failure of AI control protocols. My GAID Project measures frontier LLMs’ propensity to fabricate across global contexts. Open-sourcing the methodology and publicly sharing the empirical outputs expose the issues and risks of untrustworthiness/unreliability of LLMs, especially in less-developed countries’ contexts, and identify the corresponding AI-facilitated international governance concerns.

People

Jason Hung

Team Member

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$12,600

Ideal

$25,000

How the money will be spent

Research stipend: US$10,000 (US$2,000/month for 5 months at 0.4 FTE)
API expenses for continual/expanded model evaluation/stress-testing: US$2000
Note: I spent ~US$400 on the first round of model evaluations (Phase 4) via the OpenRouter API. As I am scaling up my project, more models, especially more advanced models, will be stress-tested. More advanced models, especially the proprietary ones, are disproportionately costly.
Other compute expenses: Claude Max 5x (US$100/month) and Cursor Pro (US$20/month) for 5 months = US$600
Note: Compute expenses will be used for the engineering work necessary for the scale-up of Phases 3 and 4 as proposed above

Note: I have been working with senior Cambridge academics on AI publications and funding submissions. The additional cost for the ideal amount will cover the research stipends (at ~0.1 to 0.15 FTE) of my collaborators at Cambridge.

Discussion

No comments yet. Be the first to share your thoughts.