The questions we are asking
Ongoing research. This is an experimental result from active work, not a settled conclusion. The numbers are what we measured and the method is described so you can judge it, but the programme is still running and later experiments may revise what it means. More about this programme.
Our research is one project asked as 10 questions. Each page below states the question in plain language, says where it currently stands, explains how its results fit together, and then lists them. Between them they cover all 76 published records.
Does reshaping a model's internals make training cheaper?
No. The technique does exactly what it claims and the claim does not translate into an efficiency gain -- ordinary training reaches the same place, and the overhead is 23x the measured benefit.
See the resultsCan you tell in advance that a model is about to improve?
Yes, repeatedly and by several independent routes -- a probe on the hidden state, gradient statistics, held-out loss, and which earlier word the prediction leans on. The warnings are real; how far ahead they fire varies a great deal.
See the resultsIs seeing it coming actually worth anything?
So far, no. Both attempts to act on a warning were matched by acting at a random moment, and the detection cost more than the manoeuvre saved. What the warning is worth is a separate question from what makes training cheaper, which has its own page.
See the resultsWhat actually happens at the moment a model learns?
It builds machinery rather than selecting it, working through the task in a reproducible order and trying a simpler wrong rule on the way. The visible training curve cannot tell you which is happening.
See the resultsWhat decides when a model learns, and can you change it?
Settings dominate, data barely matters, and there is a brief window before the jump in which interrupting the model is unusually costly. Timing can be delayed but not brought forward.
See the resultsWhich parts of a model actually matter?
The parts that move most are not the parts that matter, and no single component is required -- the model routes around every freeze. What a part is worth shows up only when you remove it.
See the resultsHow much model does a task need, and what changes when it has more?
The abrupt jump is what spare capacity buys -- it fades smoothly as the model shrinks, long before the model stops working. Capacity and task difficulty act separately, not as a ratio.
See the resultsIs the task we are studying actually hard?
Often it is not. A rule from 1990 with no parameters beats the trained model on the task most of these results were measured on, and what an intervention costs is set by the task's own structure.
See the resultsHow do we know our own results are real?
Repeatedly, the control rather than the measurement decided the result -- and several striking findings dissolved when the right comparison was finally run.
See the resultsWhat actually makes training cheaper?
One thing has worked: stopping part of the training early saved about 7% with no loss of quality. Everything else tested has been matched by a simpler or cheaper method -- and in two cases the clever method was only winning because it was quietly being given more.
See the results