Is the task we are studying actually hard?
Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means.
A note on the language in these records. This is a working laboratory notebook for research into training AI models more cheaply and efficiently, so you will read that an approach did not work, that a result did not hold up, or that one method was worse than another. That is the research doing its job, not a verdict on the engineering we deliver to clients. Ruling an approach out is how the search narrows, and these are the pages that teach us the most: nearly every technique we now rely on came from understanding why something else fell short. Testing our own ideas at least as hard as anyone else's is the point of publishing them. More about this programme and why we run it.
Before asking how a model solves something, it is worth asking whether the something is difficult at all -- and what the cheapest thing that also works would score.
Where this stands
Often it is not. A rule from 1990 with no parameters beats the trained model on the task most of these results were measured on, and what an intervention costs is set by the task's own structure.
This is the check that reframes everything else, and almost nobody runs it. If you are choosing a benchmark, run the dullest possible method on it first -- it takes minutes.
Still open · 64 published results bear on this question.
What this does not settle yet
Stated plainly, because the gaps are as much a part of the record as the answers:
- Is there any task in the battery a GRU beats every classical baseline AND a transformer on?
- Does the cost-class ordering survive at other lags? N12 gives reason to doubt it.
How these results fit together
Four records, and the reason this question exists at all is uncomfortable: a rule with no learned parameters, of a kind known since about 1990, solves the main task here perfectly, while the network spends substantial compute rediscovering it. That does not invalidate the other results -- they honestly describe how these networks find that mechanism -- but it sets a limit on what they can be about. So we built a battery of harder tasks with deliberate room above the ceiling, because a task the model already scores 99% on cannot show you that something made it slightly worse. Two records here are more interesting than the framing suggests. The cost of denying a model part of its task turns out to be set by *which* part of the task's repeating structure is removed, and those cost groupings move when the structure moves -- which means the task is telling us something about the model rather than the reverse. And the parts learned earliest are the most expensive to disrupt, which is where the idea of a prerequisite structure inside learning came from.
The results
Each of these is a self-contained record: what we asked, what would have proved us wrong, what we found, and what it does not show. They open with a plain-language summary before any of the technical detail.
A Quarter of the Rule Gone
The easy start's saving vanishes once a quarter of the rule is broken: a cliff, not a slope.
Read the recordThe Sum's Baseline, Tuned Both Ways
With ordinary training tuned in both directions, the easy start's halving of the steps on the sum stands.
Read the recordMatched, and the Sum Still Halves
Same symbols, same model: the sum is sped up and random tables are slowed, so a shared rule is what decides it.
Read the recordA Harder Table, and Still Slower
Without a rule shared between easy and hard versions, the easy start costs time even when the task is hard.
Read the recordThe Sum Crosses Earlier
The easy start's switch from cost to saving appears on the sum too, at a different difficulty than on the maximum.
Read the recordThe Sum Never Crosses
The easy start was ahead on every version of the sum, but none was easy enough to test where it stops paying.
Read the recordThe Harder the Task, the Bigger the Saving
The easy start costs time on easy versions of a task and saves up to 40% on hard ones, crossing at a definite difficulty.
Read the recordBoth Tuned, and Still Slower
Both recipes at their best rate, and the easy start is still about 40% slower on a task with no rule.
Read the recordA Faster Rate, and Still Slower
The easy start's slow-down on a random table is not its learning rate: tuned, it is still slower than ordinary training.
Read the recordA Random Table, and It Slows
With no rule shared between the easy and hard versions, the easy start cost time even at matched difficulty.
Read the recordXOR, Tuned, and It Still Halves the Steps
Tuning ordinary training's learning rate moved it a few hundred steps; the easy start's lead of about five thousand stayed.
Read the recordXOR Halves the Steps Too
The easy-start speed-up holds for bitwise XOR, a different kind of operation from the sums it was found on.
Read the recordA Harder Maximum, and It Helps Again
The easy start helps when the hard task is hard for ordinary training; our earlier arithmetic-only reading is withdrawn.
Read the recordThe LSTM's Best Rate, and the Speed-Up Holds
Ordinary LSTM training at its best rate still loses to the easy start: the speed-up holds on a second recurrent network.
Read the recordFor the Maximum, It Slows Learning
For an easy maximum the easy start slowed learning; corrected by a harder version, where it helped. Difficulty, not the operation, decided.
Read the recordOn the LSTM, Not Settled
On the LSTM the easy-sum lead shrank by half once ordinary training was tuned, and the tuning is not finished; the result there is not settled.
Read the recordSubtraction Too
The easy-start speed-up is not tied to addition: with a hard subtraction task it saves the same share of steps.
Read the recordOn an LSTM It Holds
The easy-sum speed-up holds, and is larger, on a second recurrent architecture; it is absent on a transformer.
Read the recordOn a Transformer, No Speed-Up
The easy-sum speed-up that holds in the recurrent model does not appear on a small transformer; the earlier lean was noise.
Read the recordAt Width 24, Undecided
The easy-sum speed-up is clear at two model widths and leans the same way, undecided, at a third, smaller one.
Read the recordAt Width 96 It Holds
The easy-sum speed-up is the same size at twice the model width, on a task matched to the larger model.
Read the recordOn a Transformer, Not Established
The easy-sum speed-up is established on the recurrent model; on a small transformer it leans the same way but is not yet outside the noise.
Read the recordDifferences Help Too
What carries over from the easy warm-up may be the shape of the task, combining two remembered symbols, rather than addition itself.
Read the recordRe-Tuned for Decay, Plain Still Trails
With ordinary training re-tuned for a decaying learning rate, the easy-sum mix still wins clearly; tuning closed about a third of the gap.
Read the recordKept In, They Get in the Way
Easy examples help as an early phase and hinder when kept in, even with the hard task seen just as often; timing, not exposure.
Read the recordThe Operation and the Component
About half the easy-sum benefit comes from practising the operation itself, and half from practising a piece of the actual task.
Read the recordUnder Decay the Lead Widens
A decaying learning rate barely affects the easy-sum mix and costs slower ordinary training most of its accuracy, though ordinary training was not re-tuned for it.
Read the recordIt Must Be an Early Phase
On the adding task, easy examples help as an early phase and stop learning when kept in the mix, though the hard sum was also seen less.
Read the recordPlain Gets There Eventually
On the harder sum, too, the easy-sum mix is a speed-up: ordinary training solves it eventually, in roughly three times as many steps.
Read the recordForty Percent Fewer Steps
Measured as steps to solve, fixed in advance, the easy-sum mix reaches the solution in about three-fifths of the steps of tuned ordinary training.
Read the recordTuned on Its Own Target, Plain Never Solves It
The replication did not depend on borrowed settings: tuned on the second sum itself, ordinary training still could not learn it in the budget.
Read the recordIt Replicates on a Second Target
The easy-sum mix's advantage over tuned ordinary training holds on a second hard sum, on runs never used for tuning.
Read the recordThe Mixture Gets There Sooner
The easy-sum mix does not reach a better final answer; it reaches the same answer in roughly a third of the training steps.
Read the recordSlower Than Tuned, Not Hotter
The last untested result's task tunes to a higher rate than it used; it was run slow, not hot, and its conclusion does not move.
Read the recordOn Fresh Seeds, the Mixture Wins
Against properly tuned ordinary training with a warm-up schedule, on runs never used for tuning, the easy-sum mix solved the task on 26 of 32 runs against 12.
Read the recordA Delay by Most Measures
Run longer, the warm-up that seemed to stop learning mostly slowed it down; where you stop counting decides which story the numbers tell.
Read the recordTwo Easy Sums Are Traps
On their own, two of the easy warm-up tasks leave the model unable to learn the hard one; shuffled together with the third, they help.
Read the recordAgainst the Best Plain Run, Not Established
Against properly tuned ordinary training, the easy-first mixture leans ahead on the second task but has not yet cleared the noise; a larger test follows.
Read the recordOrder Can Make the Warm-Up Harmful
On the second task the order of the easier examples decided between help and harm; ending on the easiest was worse than no warm-up.
Read the recordNo Other Claim Ran Hot
An audit of every standing speed-up result found none outside the warm-up experiments trained at a learning rate above their task's best.
Read the recordEasier Data Adds Nothing to a Rate Warm-Up
Once training has an ordinary learning-rate warm-up, starting on easier data brings no further benefit on this task.
Read the recordOn Modular Sum the Mixture Holds Up
On the second task the easy-first mixture was not tuned away: it stays ahead at every rate, though plain training's best setting was not yet found.
Read the recordWith Both Tuned, the Warm-Up Is Within Noise
Compared fairly, each at its own best setting, the warm-up adds nothing measurable; it changes which learning rate works best.
Read the recordA Learning-Rate Warm-Up Does the Job
A standard learning-rate warm-up, which most training recipes already use, matches the best data warm-up.
Read the recordThe Burst Wins Only at a Higher Rate
The best data warm-up helps only by making a larger learning rate usable, which an ordinary warm-up schedule does equally well.
Read the recordOne Hundred Steps of One Easier Version
The simplest warm-up is also the strongest; whether it beats properly tuned ordinary training is being tested next.
Read the recordThe Window Is the First Hundred Steps
Whatever the early burst prevents happens in the first hundred steps of training, and a slower learning rate prevents it too.
Read the recordHalf the Rate Matches the Warm-Up
Training at half the learning rate matched the warm-up's benefit on its own; eleven earlier results are qualified and our outside write-up is on hold.
Read the recordA Sweet Spot, Not the Nearest
The most useful warm-up is easier than the target by a margin; nearer versions help less and much easier ones barely help. An earlier reading of nearness is corrected.
Read the recordAt Width 96 the Gap Holds Steady
At the larger width the warm-up's lead is small and steady rather than widening; the larger model also needed a slower learning rate, which is tested next.
Read the recordThe Burst Works Only at the Start
Early training has a window in which a little of the right data changes which solution the model finds; the same data later does nothing.
Read the recordOne Nearby Version Does Most of It
Variety in the warm-up is not what matters; one easier version close to the target does most of the work, and one further away does little.
Read the recordOne Hundred Steps Is Enough
Under 2 percent of training spent on easier versions of the task gives the whole benefit, and the benefit shows up thousands of steps later.
Read the recordThe Warm-Up Must Be Related
Only easier versions of the target task help as a warm-up; a different easy task does nothing, and training on meaningless answers costs more than the time it takes.
Read the recordThe Warm-Up Holds at a Larger Width
The warm-up's benefit survives a larger model on a harder task, at a quarter of the size so far, with the gap still opening when the run ended.
Read the recordA Short Warm-Up Is Enough
A few hundred steps of varied, easier material at the start is as good as any longer warm-up; past a point, more of it costs.
Read the recordA Higher Ceiling, Not a Head Start
The early easy-task mix leads to a better final solution, not just a faster start; keeping the easy tasks mixed in all along does worse.
Read the recordMixing First Makes It Learnable
Mixing easy versions in early made a hard task learnable in six runs of eight instead of one, at the same compute, on a second task family.
Read the recordMixing First Lifts the Ceiling
A random mix of easy tasks early made a model 21 points better at the hard task, at the same compute, confirmed on new runs.
Read the recordOrder Did Not Help, Mixing Might
An easy-to-hard curriculum was no faster than practising the hard task throughout; a random mix of easy tasks may raise the ceiling.
Read the recordA Rule From 1990
A rule with no parameters, invented before neural language models existed, solves our main task outright. That does not make our results wrong; it changes what they are about.
Read the recordA Task Battery With Designed Headroom
Two of four candidate tasks accepted. The rejections are the informative part: one is all-or-nothing per seed, the other is trivial for a recurrent model.
Read the recordPredicting The Cost From The Task
The first thing we have found here that can be predicted from the exercise alone, without looking at the model at all.
Read the recordThe First Piece Is The Load-Bearing One
The part a model learns first costs the most to remove, and the part it struggles longest with costs the least.
Read the recordBack to all research questions
Get new results as we publish them
Roughly monthly, one finding per email, in plain English first. Including the approaches that turned out not to work, which are usually the useful ones. No sales email.
Double opt-in: we send a confirmation link and add nobody who does not click it. One-click unsubscribe on every email.