The first words
First words, at step 250
“It was on that it had said”
16 July 2026
The day a model trained from nothing on a laptop stopped producing noise and started producing English. Small, honest, and entirely its own.
The record
9.78 to 3.89Training loss, first step to last
First words, at step 250
“It was on that it had said”
The claim
A 34-million-parameter model is tiny. What mattered was proving that the project’s own trainer, its own tokeniser and public-domain text could take a model from nothing to word-shaped English on an ordinary laptop.
The first-contact tooling was built for this: a supervisor script that steps the batch size down when the GPU runs out of memory and resumes from the last checkpoint after a crash, so a run can be started once and left overnight. A local playground then let the same prompt run across checkpoints: at step 1,000 the model babbles, and by step 20,000 it writes sentences in the style of the old books it read.
The project’s identity document says this plainly, and so do we:
For the same reason the playground is a text-completion page, not a chat window. A chat box would invite questions the model cannot answer and make a correct result look broken. A chat interface becomes honest only after instruction tuning.
The first corpus was 8 books, about 1.34 million tokens. For a model this size the usual compute-optimal budget is around 680 million tokens, so training on 8 books would have meant roughly 98 passes over the same text: a memoriser, not a language model. The corpus was rebuilt from 500 public-domain books, fetched politely one at a time, with the Project Gutenberg licence header and footer stripped so the model would not learn to recite them.
The first-contact weights are proof of life, not a product, and they are not published. The next rung is a roughly 200-million-parameter run on a rented GPU, trained on FineWeb-Edu, with a target of perplexity under 30 on English text.