How much model does a task need, and what changes when it has more?
Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.
Does a bigger model learn differently, or just better? And what does the extra capacity buy?
Where this stands
The abrupt jump is what spare capacity buys -- it fades smoothly as the model shrinks, long before the model stops working. Capacity and task difficulty act separately, not as a ratio.
It is the closest this project has come to a scaling-shaped statement, and it reframes the central phenomenon as something oversized models do rather than something tasks force.
Still open · 4 published results bear on this question.
What this does not settle yet
Stated plainly, because the gaps are as much a part of the record as the answers:
- Everything here is tiny models on a laptop. The scale question is the standing limit on the entire project.
- Architecture and objective are perfectly correlated across every result (I5).
How these results fit together
Four records asking what size of model a task needs, and how that changes what we see. The finding that shapes the rest is that the sharpness of the jump depends on having capacity to spare: bigger models transition more abruptly, smaller ones more gradually, and the effect is strong enough to beat seed noise several times over. That reframes the whole phenomenon. The sudden jump is not a universal fact about how neural networks learn -- it is what a model with room to spare does on a task that does not need all of it. The phenomenon is also architecture-dependent: it appears reliably in gated recurrent models and does not appear in the ungated and attention-based models tested, even when those models learn the task perfectly well. We also tried to compress size and difficulty into one number, on the theory that only the ratio matters, and that failed cleanly: three settings at an identical ratio spanned a factor of two, because the two axes have genuinely different strengths. Difficulty is not simply capacity in reverse.
The results
Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.
The Jump Needs Room To Spare
The abrupt improvement is not something the task forces. It is what a model does when it has more capacity than the job needs, and it fades smoothly as that capacity is taken away.
Read the recordSpare Capacity Is Not One Number
Size and difficulty both matter and they do not cancel out, because difficulty counts about twice as much. The way we measure difficulty did survive the test, and that half had no adjustable numbers in it.
Read the recordLooking Is Not Using
The attention model relies on exactly the right word from the very first check, then settles into a habit the exercise is built to punish.
Read the recordThe Phenomenon Map: Which Architectures Show It
Nine of nine gated recurrent cells show the effect; zero of three others do. The decisive cell is a transformer that learns the task, takes the longest of anything in the grid, and still shows nothing.
Read the record