Public model launches can look sudden, but labs estimate large-model performance before committing to a full training run.
Yusuf (@_yusufknl) argued that scaling laws explain much of this apparent surprise. Labs fit power laws on smaller runs and use them to estimate the loss of larger models. On log-log axes, the fitted power law is a straight line.
The interactive section below is the useful part for me: it separates five ideas that are often collapsed into “bigger model.” The curves change with compute, parameter count, training data, evaluation rules, and inference-time compute.
Interactive scaling model
These are toy curves with a realistic shallow exponent, not a reproduction of any single paper. The first tab compares the same power law on ordinary and log-log axes.
How I read the curves
The line predicts loss, not a product
In January 2020, Kaplan, McCandlish, and colleagues at OpenAI published Scaling Laws for Neural Language Models. Cross-entropy loss fell as a smooth power law in parameters, training tokens, and compute — across more than seven orders of magnitude. A power law is curved on ordinary axes and straight on log-log axes. The fitted slope can be estimated on smaller models and extrapolated, within the range where the relationship continues to hold.
The exponents are small, around 0.05–0.10. Each doubling of scale therefore produces a modest, predictable reduction in loss. Large improvements require much larger increases in compute, data, or parameters.
This is the distinction I want the interactive model to keep visible. A fitted loss curve can help plan a training run. It does not predict whether a release will follow instructions well, pass a particular benchmark, or feel useful in a workflow.
Estimating training compute with 6ND
A common approximation is C ≈ 6ND: two FLOPs per parameter on the forward
pass and about four on the backward pass. For GPT-3, with 175 billion parameters and 300 billion tokens,
the estimate is ~3.15 × 10²³ operations, matching the public figure almost to the digit.
Epoch AI found that since 2010, training compute has doubled roughly every six months, about four times faster than Moore’s law, for over a decade. Four months after the scaling-laws paper, many of its authors shipped GPT-3. Several later founded Anthropic.
The recipe changed
Kaplan’s original advice had a bias: spend most of your compute on more parameters. MT-NLG hit 530 billion. Then DeepMind’s March 2022 paper Training Compute-Optimal Large Language Models trained 400+ models and found the industry had often been allocating too much compute to parameters and too little to training data.
Model size and data should scale together, both as the square root of compute. The popular rule of thumb: about 20 tokens per parameter.
Chinchilla, with 70 billion parameters and 1.4 trillion tokens, beat Gopher (4× the size) on the same compute, and beat GPT-3 and MT-NLG too. Run GPT-3’s own budget through the rule and the compute-optimal model was smaller and better-read than what OpenAI shipped.
Chinchilla optimizes the training budget, not the lifetime cost of inference. Meta deliberately over-trained Llama 3 on 15 trillion tokens because a small model is cheap to run repeatedly. This trade favors smaller models trained on more tokens when inference cost matters.
The scorecard can create a cliff
Wei et al. (2022) argued some skills are absent in small models and appear past a scale threshold, a phase transition. Schaeffer, Miranda, and Koyejo (NeurIPS 2023 Outstanding Paper) fired back: with an all-or-nothing metric, per-token accuracy can rise smoothly while the score stays at zero until enough tokens line up.
Both can be right about different things. Underlying loss scales smoothly. Whether a human-meaningful ability looks like a jump depends partly on the metric. Tab 4 shows this effect as k changes.
I left claims about a fixed stock of public text, a date when it will be exhausted, and ARC-AGI scores out of this note. The references below do not document the assumptions and evaluation conditions needed to state those figures responsibly.