Watching a Model Fail on Purpose: Tokenization and the Decoding Tradeoff

Post 2 of 7. Post 1 covered why I built a playground instead of training a model. This post is about what happens before and after the model itself: how text becomes tokens, and how probability distributions become text.

---

The first output I ever got from a language model running on my own machine was this. Prompt: "How do I boil an egg?" Model: GPT-2, default settings.

How do I boil an egg?

You can boil an egg in a pot of water. You can boil an egg in a pot of water.

How do I boil

It answered, sort of. Then it repeated itself verbatim. Then it started asking my question back at me. My instinct was that the model was broken or too small to be useful, and both guesses were wrong in instructive ways. The repetition wasn't the model, it was the decoding strategy, a setting I didn't know existed. And the failure to answer like an assistant wasn't a bug at all, it was a base model doing precisely what base models do, which is the subject of Post 3.

This post unpacks that one output, layer by layer. By the end, you'll be able to reproduce it, fix it, and break it in a different direction, all in the live playground.

Layer 1: The model never sees words

Neural networks operate on numbers. Before any text reaches the model, a tokenizer converts it into a sequence of integers, and the scheme it uses shapes everything downstream.

The two obvious schemes both fail. One number per character makes sequences enormously long and forces the model to learn spelling before semantics. One number per word explodes the vocabulary, every inflection, typo, and coinage needs its own slot, and anything unseen at training time becomes an unknown blank. GPT-2 uses Byte Pair Encoding (BPE), the standard compromise: start from characters, repeatedly merge the most frequent adjacent pairs in a training corpus, and stop at a vocabulary of ~50,257 tokens. Common words end up as single tokens; rare words decompose into familiar fragments.

You can watch this happen. "How do you boil an egg?" tokenizes to seven clean pieces, one per word, plus the question mark:

["How", " do", " you", " boil", " an", " egg", "?"]
→ [2437, 466, 345, 20667, 281, 5935, 30]

Three things in that output taught me more than any explanation had. First, the leading space is part of the token. " boil" and "boil" are different tokens with different IDs, GPT-2 learned "word after a space" and "word at start of text" as separate units. Second, the IDs are arbitrary indices into the vocabulary; 2437 means nothing except "the slot that decodes to How." Third, this sentence is the easy case. Feed it something rarer and BPE shows its actual behavior:

"GPT"           → ["G", "PT"]
"tokenization"  → ["token", "ization"]

"GPT" wasn't frequent enough in 2019-era web text to earn a single token, so it fragments. "tokenization" splits into two productive morphemes the tokenizer saw constantly. The rarer and longer the word, the more pieces, my own name shatters into fragments of other words.

The tokenization tax

Then I fed it other languages, and the picture stopped being cute.

GPT-2's tokenizer was trained overwhelmingly on English web text. Hindi, Japanese, and Arabic text fragments toward individual bytes, a short greeting that would be five or six tokens in English becomes dozens of near-meaningless byte-level fragments. The consequences are not academic. Models have finite context windows measured in tokens, and API pricing is per token. If your language tokenizes at 4x the density of English, you get a quarter of the effective context and pay four times the price for the same meaning. The tokenizer is trained before the model and inherits the biases of its corpus; every downstream user of a non-dominant language pays a permanent tax for that decision.

This is reproducible in the playground's Sandbox tab in ten seconds, and I'd argue it's the fastest way to viscerally understand that tokenizers are not neutral infrastructure.

Layer 2: The model outputs a distribution, not an answer

Here is the mental model that unlocked everything else for me. At each generation step, the model does not produce the next token. It produces a probability for every token in the vocabulary, 50,257 numbers summing to one. Something has to collapse that distribution into a single choice, and that something, the decoding strategy, is a separate component from the model, with its own knobs, capable of ruining or rescuing the output on its own.

This separation is why "the same model" can produce wildly different text. The weights were frozen in 2019. The character of the output is substantially decided at sampling time.

Layer 3: Breaking and fixing generation, on purpose

The playground's /compare endpoint runs one prompt through several decoding configurations simultaneously. Here's what each one did, with real outputs.

Greedy decoding (do_sample=False): at every step, take the single highest-probability token. Deterministic, and the source of my opening loop. The mechanism is worth spelling out: once the model emits "You can boil an egg in a pot of water," that exact phrasing becomes strong evidence, statistically, that the same phrasing continues. The most probable continuation of a sentence is often that sentence again, so greedy decoding falls into an attractor and cannot leave, because leaving would require choosing a lower-probability token, which is the one thing greedy never does.

A detail from a later run made the commitment property vivid. Generating from a Lorem Ipsum prompt, greedy produced "tool for printing and typeetting", typo and all, and then repeated the typo identically in every loop iteration. Greedy doesn't just commit to phrases; it commits to errors, because they were locally most-probable once and the path never gets re-evaluated.

Temperature reshapes the distribution before sampling. Mechanically, the model's raw scores (logits) are divided by the temperature value before the softmax converts them to probabilities:

p(token_i) = softmax(logit_i / T)

T below 1 sharpens the distribution, the top choice becomes even more dominant, converging on greedy as T→0. T above 1 flattens it, probability mass shifts toward the tail. At T=0.7 with sampling on, my Lorem Ipsum prompt produced coherent, on-topic, non-repeating prose about typesetting. At T=2.0 it produced grammatical wreckage: "Ipsum has been using as a replacement 'non printable documents'...", novel, non-repetitive, and semantically unmoored. Small models can't hold a thread when you sample that deep into their uncertainty.

Top-k truncates the distribution to the k highest-probability tokens before sampling, a fixed shortlist. Top-p (nucleus sampling) makes the shortlist adaptive: take the smallest set of tokens whose cumulative probability exceeds p. When the model is confident, the nucleus is tiny; when it's uncertain, it widens. These stack with temperature, production systems typically apply all three.

The instructive accident: I once set top_p=0.35 with a long generation length, expecting reasonable output since sampling was on. I got the repetition loop back, full "the man who had been shot had been lying on the ground" recursion, five paragraphs of it. Over-restricting the nucleus had collapsed sampling back into near-greedy behavior. The lesson generalizes: these parameters interact, and restriction anywhere in the pipeline pushes you toward the deterministic attractor.

Plotted as a curve, the tradeoff looks like this:

temperature →   low            mid             high
character   →   safe, loops    coherent, varied    novel, incoherent

The sweet spot for these small models sits around 0.7–1.0, which is also, not coincidentally, where most production APIs set their defaults.

The part where the corpus shows through

One more output, because it belongs in this post even though it's uncomfortable. Prompting GPT-2 with "Once upon a time", story opener, default-ish settings, produced a continuation about two people with gunshot wounds, faces covered in blood, looping the imagery for paragraphs.

Nothing malfunctioned. A base model is a mirror of its corpus statistics, and violent narrative is common in web text, so violent narrative is a high-probability continuation of a story opener. The model has no concept of appropriate, because appropriateness was never part of the training objective, only likelihood was. Every safety behavior you've seen in a deployed chatbot was installed after pre-training, by people, on purpose. That installation process, what it changes, and what it fails to change, is the next post.

Quantifying what I'd been eyeballing

Before moving on, I wanted numbers for the things I'd been judging by sight. Two metrics, both built in an afternoon: repetition rate (the fraction of 3-word sequences that are repeats of earlier ones, my greedy loop scores catastrophically high) and perplexity (how surprised the model is by a text, computed from its own probabilities, with the counterintuitive property that degenerate looping text scores low, because the model is maximally unsurprised by its own repetition). Both have a blind spot big enough to drive a truck through, which Post 4 does.

Try it yourself: In the live playground, the Sandbox tab tokenizes anything you type, try your name, then try a non-English sentence. The Swing Set tab has a do_sample toggle; turn it off and reproduce the loop. The Merry-Go-Round tab runs greedy vs. T=0.7 vs. T=1.5 side by side on your prompt.

Caveats: GPT-2 is a 2019 model at 124M parameters; modern base models fail in the same categories but far less flagrantly, and some of the loop behavior shown here is mitigated by repetition penalties in modern serving stacks. Every demonstration in this post is a single prompt, not a sweep, the mechanisms are real, but don't read the specific outputs as effect-size estimates.