Synthetic Data Teaches a Model to Learn Languages

A transformer trained only on synthetic sequences learned to predict six real languages from context, without changing its weights or seeing natural language during training.

Lennart Carstens-Behrens and Holger Fröhlich built the 300-million-parameter Prior-Fitted Language Model, or PFLM, at the byte level. Each training sequence came from a newly sampled causal process with its own rules, forcing the model to infer the current sequence rather than memorise a recurring synthetic language.

Why it matters

A researcher studying in-context learning rarely knows whether a model is learning a new rule or recalling something close to its training data. PFLM creates a cleaner separation: its frozen weights contain a general strategy learned from artificial processes, while the real language arrives only in the prompt.

The synthetic generator was designed to reproduce broad statistical properties found in natural text, including frequent symbols, long-range dependence and slowly improving predictability. It did not contain words, grammar or examples from Wikipedia. The model’s task during pretraining was simply to predict the next byte in a sequence from a different generator each time.

When the authors supplied long Wikipedia streams in English, Chinese, Hindi, Arabic, Japanese and Korean, prediction improved throughout the context. Loss began near the uniform baseline of eight bits per byte and fell to between 0.9 and 2.4 after one million bytes, depending on the language. The model weights remained frozen.

The experiment tests adaptation, not fluency

PFLM does not generate polished prose or demonstrate an understanding of meaning. It estimates what byte comes next after observing enough of an unfamiliar source. That narrower ability still matters because it shows that a sequence learner can acquire a useful adaptation procedure without first absorbing natural language.

The model reads raw bytes rather than a fixed vocabulary of word pieces. This removes a tokenizer trained on human text as another possible route by which linguistic structure could leak into the system. The public repository provides the model integration and an API for scoring streams; the released weights allow others to test new domains.

The authors also compared PFLM with classic general-purpose compression methods. They report that it compressed six non-text sources, including source code and speech data, below gzip and PPMd after enough context. Compression is a direct test of prediction: a model that assigns better probabilities needs fewer bits to encode the same data.

The study also found

Given streams of numerals, PFLM learned approximate addition, counting and magnitude comparison in context. It predicted deterministic sequences including prime-number indicators and Rudin-Shapiro patterns. Its language results varied substantially across the six Wikipedia languages, leaving clear room between adaptation and the performance of models trained directly on text.

The work is a preprint and the headline result comes from one 300-million-parameter system built around one synthetic prior. Its generator deliberately shares high-level statistics with natural data, so the experiment does not show that any arbitrary synthetic curriculum produces language learning. It shows that carefully chosen structural diversity can train a reusable learning procedure.

The code and weights are public, making the immediate test straightforward: apply the frozen model to genuinely new structured sources and measure when its context-based improvement holds or breaks.

Verification