What decides when a model learns, and can you change it?
Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.
A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.
Two models given the same task learn at different moments. What sets the timing, and can you move it deliberately?
Where this stands
Settings dominate, data barely matters, and there is a brief window before the jump in which interrupting the model is unusually costly. Timing can be delayed but not brought forward.
A critical window that closes as models grow is the kind of thing that matters when scheduling real training, and it is measurable here in a way it is not at scale.
Still open · 14 published results bear on this question.
What this does not settle yet
Stated plainly, because the gaps are as much a part of the record as the answers:
- Nothing found so far makes the transition arrive EARLIER. Every lever delays it.
- The critical window has been measured on one component; whether other components have their own windows is untested.
How these results fit together
Eight records, and they line up into a clear and slightly deflating conclusion: the timing of the jump is mostly decided by ordinary things, and it is not steerable. Ordinary training settings -- learning rate, batch size -- move the moment by hundreds of steps. Changing the entire training dataset moves it by about seven, and changing the order of the data by about three. So the exotic explanations lose to the boring ones by two orders of magnitude. We then tried to move the moment deliberately, and found the response to a nudge is discontinuous: tiny and enormous perturbations both delay learning by about the same amount, and nothing we tried made it arrive sooner. That closed a whole line of work about steering training, and it is why this project stopped trying to build a controller. Alongside that sits a genuinely causal result -- there is a window of about forty steps just before the jump where interfering with the model's memory delays it much more than interfering earlier or later. The window is real, but it narrows as models get bigger and had vanished at the largest size we tested, which is a reason to be careful about building anything on it.
The results
Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.
The Optimizer Follows
The optimizer's internal state changes after learning, not before, and wiping it afterwards costs nothing: it follows, it does not lead.
Read the recordLonger Memory, Lower Limit
With a finer search, the usable learning rate falls steadily with memory span, from 0.072 at 2 steps to 0.020 at 8.
Read the recordOne Step of the Ladder
A published prediction that the usable learning rate falls with memory length: true in direction here, carried by a single step, not yet a law.
Read the recordLate, Not Delayed
Given 30,000 steps the model learned from only 384 examples, slowly and all at once, never memorising first. The grokking search closes.
Read the recordNo Delay at the Edge
Looking for grokking at the edge of enough data: weight decay lowered the data needed four-fold, but learning never lagged memorising.
Read the recordMemorised, Never Generalised
Looking for the famous 'grokking' delay: small training sets were memorised at once and never generalised, with or without weight decay.
Read the recordThe Forty Steps That Matter
Freezing the model's memory just before it learns costs 10.6 steps more than freezing it earlier. Forty steps after, the same freeze costs 0.1 steps. A peak, not a ramp.
Read the recordThe Window Closes as Models Grow
The effect shrinks steadily with size and vanishes at the largest we tested. Our first three attempts to measure it were wrong, and one check caught all three.
Read the recordWe Had Been Calling It the Right Thing
The effect appears in both gated designs and not in the ungated one, which learns the task faster and better. The name held.
Read the recordIs Transition Timing Controllable?
A step, not a dial: any interference costs about 21 steps and far more interference costs barely more. Nothing makes learning arrive sooner.
Read the recordWhat Actually Decides When a Model Learns
Data is worth about 2.5% of what the settings are worth. The ratio test we built to decide this landed at 0.49 against a 0.50 threshold, and its confidence interval reached 1.58, so it settled nothing.
Read the recordIs Learning a Lucky Accident?
Noise moves the transition 1.70x later, monotonically, and doubles its width. The batch-size sweep that seemed to show the opposite was measuring update count.
Read the recordDoes the Effect Survive the Optimizer?
The effect is within 8% across three optimizers, so it is not an artifact of adaptive scaling. Its coincidence with the moment of learning is an artifact of one of them.
Read the recordThe Fix That Sharpened It
Cycling the learning rate makes an interruption at an arbitrary moment much cheaper and one at the critical moment no cheaper, so the gap between them widens.
Read the recordBack to all research questions
Get new results as we publish them
Roughly monthly, one finding per email, in plain English first. Including the approaches that turned out not to work, which are usually the useful ones. No sales email.
Double opt-in: we send a confirmation link and add nobody who does not click it. One-click unsubscribe on every email.