https://awarenesssoftwaregroup.com/research/ Open research on how AI models actually learn. One finding per entry, in plain English first and in full technical detail underneath, including the approaches that turned out not to work. en Sun, 27 Sep 2026 22:40:39 +0000 https://awarenesssoftwaregroup.com/research/a-critical-amount-per-step/ https://awarenesssoftwaregroup.com/research/a-critical-amount-per-step/ Learning time follows one curve in material per training step, and the standard critical batch size formula fits it within 2.2 percent across an 86-fold Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/a-rise-not-a-peak/ https://awarenesssoftwaregroup.com/research/a-rise-not-a-peak/ Five independently trained copies of a public language model all reorganise their internal state as they learn to copy, then level off. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/a-speed-accuracy-frontier/ https://awarenesssoftwaregroup.com/research/a-speed-accuracy-frontier/ Faster learning rates made small models learn sooner but end less accurate. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/a-third-was-model-size/ https://awarenesssoftwaregroup.com/research/a-third-was-model-size/ With every model built the same size, a bigger vocabulary still slows learning, but about a third less than an earlier estimate that let model size change Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/batch-and-length-are-substitutes/ https://awarenesssoftwaregroup.com/research/batch-and-length-are-substitutes/ Bigger batches and longer sequences both speed learning, but each helps less when the other is large. A law that treats them as independent misses that. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/flat-through-width-256/ https://awarenesssoftwaregroup.com/research/flat-through-width-256/ Across a fourfold range of model width, an unneeded skill costs the other skills the same share of their learning time. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/four-agree-one-took-a-shortcut/ https://awarenesssoftwaregroup.com/research/four-agree-one-took-a-shortcut/ Four ways of packing the same material into a training step took the same time to learn. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/it-sees-the-damage-not-the-repair/ https://awarenesssoftwaregroup.com/research/it-sees-the-damage-not-the-repair/ Rounding a model's weights to fewer bits damages it. Measuring its internal shape spots the damage, but a repaired model works again with a changed shape. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/it-was-the-rate-not-the-width/ https://awarenesssoftwaregroup.com/research/it-was-the-rate-not-the-width/ The easing of competition between skills that seemed to come with a wider model came from its slower learning rate. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/less-competition-at-width-512/ https://awarenesssoftwaregroup.com/research/less-competition-at-width-512/ With its learning rate tuned, the widest model paid a smaller share for an unneeded skill, but its rate changed along with its width, so which one Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/longer-sequences-did-not-help/ https://awarenesssoftwaregroup.com/research/longer-sequences-did-not-help/ Training on sequences eight times longer barely changed when a small model learned, because the extra length only repeated what each sequence already held. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/most-of-it-is-pull/ https://awarenesssoftwaregroup.com/research/most-of-it-is-pull/ Teaching an unneeded skill half as often removed about half of what it cost a small model's other skills: the competition is mostly the pull of training, Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/one-number-forecasts-the-grid/ https://awarenesssoftwaregroup.com/research/one-number-forecasts-the-grid/ A simple law in material per training step, fitted on one experiment, forecast another experiment's results within 7.5 percent, until the material per Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/one-run-predicts-the-sweep/ https://awarenesssoftwaregroup.com/research/one-run-predicts-the-sweep/ Measuring how noisy the learning signal is, during one ordinary training run, predicted the critical amount of material per step within a factor of two, Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/predicting-it-without-watching/ https://awarenesssoftwaregroup.com/research/predicting-it-without-watching/ Our only working early-warning instrument costs 55% extra compute to buy about 23 steps of notice. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/questions/ https://awarenesssoftwaregroup.com/research/questions/ The 10 questions our public AI training research is organised around, each with its current answer and the published results behind it. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/representation-geometry-is-not-load-bearing/ https://awarenesssoftwaregroup.com/research/representation-geometry-is-not-load-bearing/ The white paper closing our first research programme. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/same-material-same-time/ https://awarenesssoftwaregroup.com/research/same-material-same-time/ Giving a model the same amount of material per training step, packaged as a few long sequences or many short ones, gave the same learning time within 2.4 Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/the-archive-was-too-coarse/ https://awarenesssoftwaregroup.com/research/the-archive-was-too-coarse/ We measured every published checkpoint of two public language models. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/the-batch-law-holds-at-a-fixed-rate/ https://awarenesssoftwaregroup.com/research/the-batch-law-holds-at-a-fixed-rate/ A published law for how batch size sets when a model learns fits our tiny model closely, as long as the learning rate stays fixed. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/the-edge-shrank-with-more-models/ https://awarenesssoftwaregroup.com/research/the-edge-shrank-with-more-models/ We had found a way of looking inside compressed models that spotted damage three times better than a simpler check. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/the-head-start-closes/ https://awarenesssoftwaregroup.com/research/the-head-start-closes/ We published a break-even calculation one day and tested it across model sizes the next. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/the-level-was-not-the-problem/ https://awarenesssoftwaregroup.com/research/the-level-was-not-the-problem/ Measuring the loss signal against a level set by the task did not make it work across vocabularies: loss and accuracy genuinely track different things Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/the-method-that-looked-fastest-was-cheating/ https://awarenesssoftwaregroup.com/research/the-method-that-looked-fastest-was-cheating/ One way of choosing training data reached the finish line 26 steps sooner than the others. It got there by quietly skipping the hard half of the task. Sun, 27 Sep 2026 00:00:00 +0000 https://awarenesssoftwaregroup.com/research/the-one-that-was-not-a-ruler/ https://awarenesssoftwaregroup.com/research/the-one-that-was-not-a-ruler/ We once showed a striking internal change was an artifact of how we measured it. This measurement cannot have that problem, and it shows the same shape. Sun, 27 Sep 2026 00:00:00 +0000