Distinguishing progress in data and algorithms

Raising $10,000active

Dataset curation, synthetic data generation, and LLM training, fine-tuning, and evals to distinguish and quantify the effects of data improvements, separately from progress in algorithms and architectures, on AI capabilities.

Endorsed by

Endorsed by

Project Details

Figure 5 of Algorithmic Progress in Language Models illustrates that there was a 4*10^10× scale-up of LMs from 2012 to 2023, of which 1.7*10^7× was from physical compute scaling and 2.2*10^4× was from algorithmic progress (including algorithms, optimizers, architectures, and training data quality improvements in this second category). The goal is to isolate the effects of data quality improvements and estimate the data-only compute-equivalent gain, by doing experiments such as the following:

Fixed recipe, swap only the data
1. Hold constant: architecture, tokenizer, optimizer, hyperparameters, training code, numeric precision, evaluation procedure
2. Vary dataset: raw web, deduplicated web, filtered web, curated or enriched datasets
Test at several scales
Data time-leap
1. {old, new} data x {old, new} algorithms+architectures
2. Then do a Shapley decomposition to find compute savings from each.
Ablations on modern data pipeline
Longitudinal experiment: same recipe; dataset from each year 2014-2026
Generate datasets from models of different intelligence; measure student model quality

Theory of Impact

Informs x-risk reduction plans, especially those calling for slowdown of AI capabilities. Specifically, it affects plans that involve restrictions on algorithmic progress or AI R&D. If data-only CEG is large, then data governance is more important.

People

Robi Rahman

Data Scientist

Grants Received– no grants recorded

Add grant

Funding Asks

grantmaking.ai Launch Round

Applied

Minimum

$10,000

Ideal

$30,000

How the money will be spent

Compute for synthetic data generation, model training and fine-tuning, and evaluations.

Discussion

Private comments in this thread are only shown to approved funders and grant reviewers. Email hi@grantmaking.ai or to request access.

Grant ReviewerPrivate8d?

Private UserPrivate7d?

Loek Verdonk2d

Hi Robi,

Your fixed-recipe, data-swap design seems like a strong way to isolate data-related gains. The boundary I am curious about is that deduplication, filtering, curation, enrichment, and synthetic-data generation are themselves algorithmic processes.

What attribution rule and artifact chain will allow an independent reviewer to verify that a measured gain should be assigned to the resulting dataset rather than to a changed data-production algorithm?

Would one bounded pilot include immutable dataset manifests and transformation lineage, training-recipe hashes, seeds, checkpoints, preregistered metrics, and uncertainty estimates?

Greetings :D