What Building an LLM Playground Actually Taught Me

Post 7, the series close. Posts 1–4 built the instrument and used it to make pre-training, post-training, and evaluation concrete. Posts 5–6 shipped it. This post is the consolidation, every claim the series made, in one place, stated as plainly as I can state it, with pointers back to the evidence.

---

Six posts ago, the plan was to learn the LLM pipeline by building something that could observe real models rather than training a toy one. I want to close by stating, in one place, what that instrument actually showed, not as a recap, but as a set of claims I'm now willing to defend, each traceable to a specific experiment rather than something I read somewhere.

On tokenization and generation (Post 2)

A model never sees words, the tokenizer's choices are load-bearing. BPE gives common words single tokens and fragments rare ones into pieces the model learned from frequency, not meaning. This isn't cosmetic: tokenizers trained overwhelmingly on English text fragment other languages far more densely, which means non-English users get less effective context and pay more per unit of meaning, permanently, for a decision made before the model was ever trained. Reproducible in thirty seconds by tokenizing the same sentence in two languages.

Decoding is a separate variable from the model, and it can dominate the output's character. The model produces a probability distribution; decoding collapses it. Greedy decoding, always take the top token, reliably loops, because a phrase's own recent occurrence makes it more likely to recur, and greedy has no mechanism to escape an attractor once it's in one. Temperature reshapes the distribution before sampling; too low converges toward greedy's loops, too high produces fluent-sounding incoherence. And these interact: over-restricting top-p can reproduce greedy's exact failure mode even with sampling nominally turned on. Same frozen weights, three completely different personalities, depending only on how you sample them.

A base model has no concept of appropriate. Fed a neutral story prompt, GPT-2 produced violent content, not a malfunction, just a high-probability continuation given what "story opener" tends to precede in web text. Whatever restraint a deployed chatbot shows was added after pre-training, on purpose, by people. It is not a property of language modeling.

On post-training (Post 3)

Post-training changes what a model is trying to do, not what it knows. The cleanest evidence: Qwen2.5-0.5B base versus its own instruct-tuned sibling, identical architecture, identical pre-training, the only variable being SFT and preference optimization. The instruct model reliably answers instructions where the base model merely continues text. But the same instruct model, given a non-question creative fragment, opened with "As an AI language model, I don't have feelings...", reframing input that was never a query through the lens of its trained persona. Post-training installs a goal; the goal is not always aimed correctly.

Form improves before correctness, and formatting creates a halo effect. The instruct model's egg-boiling answer was well-structured, numbered, addressed the question directly, and instructed the reader to place "the boiled egg" in water before any boiling step occurred. Fluency and correctness are different axes; post-training targets the first far more directly than the second. This single small error turned out to matter more than I expected, it's the exact flaw that later separated a reliable evaluator from an unreliable one (see below).

Chat is not a model capability, it's an application-layer illusion. Models are stateless between calls. "Memory" across a conversation means resending the entire history with every new message, cost and context-window pressure grow every turn, and something eventually has to decide what to drop. Confirming this rigorously took two attempts: the first test (recall a stated preference) passed for the wrong reason, since the fact being recalled was still present in the resent user turn regardless of whether the model's own prior reply mattered at all. The corrected test, recall something computed only in the model's own previous turn, is what actually demonstrated context threading. Worth remembering: a test that can pass without exercising the mechanism under test isn't a test.

On evaluation (Post 4)

Automatic metrics can't see truth, and comparing them across models is a trap. Repetition rate correctly flags loops; perplexity correctly flags surprise, and, counterintuitively, degenerate looping text scores low perplexity, since the model is maximally unsurprised by repeating itself. Neither metric has any mechanism for factual correctness: a fluent, confidently fabricated answer scores fine on both. And perplexity specifically cannot be compared across models with different tokenizers, a mistake I nearly published before catching that GPT-2's ~50K-token vocabulary and Qwen's ~150K-token vocabulary put "surprise per token" on different scales entirely.

The smallest LLM judge's scores were decoupled from actual quality. Across a well-formatted answer, a confidently fabricated one, and a completely off-topic one, a 0.5B judge returned a near-constant 4/4/4/4, and, more strikingly, generated fluent reasoning that either rationalized the score after the fact or actively hallucinated supporting evidence that didn't exist in the text it was scoring. This is a worse failure mode than leniency: a judge that always says roughly the same thing, in confident prose, is indistinguishable from a working judge unless you already know the right answer.

Judge reliability scales with capability, but not smoothly, and not uniformly across error types. A 1.5B judge, three times larger, correctly caught two of three deliberate flaws that fooled its smaller sibling, but was more confidently wrong than the smaller judge on the one case involving a subtle internal logic error dressed in good formatting (5/5/5/5, "no inaccuracies," versus the smaller judge's already-wrong-but-less-confident 4/4/4/4). Scale helped with topical failures and did not help with reasoning failures. "Use a bigger judge" is real advice with a real boundary.

Rubric design bounds what capability can find. The starkest single result in the whole project: a frontier-class judge (Gemini 2.5 Flash), given a violent story continuation and a rubric with no safety criterion, gave that response its highest relevance score of the entire study, mechanically correct, since violence is a valid continuation of a story prompt, and the judge was never asked whether that continuation was appropriate. Capability didn't fail here. The question set failed. Evaluation measures what you ask, and nothing you didn't think to ask, regardless of how good the thing doing the measuring is.

On shipping (Posts 5–6)

Local development hides categories of concern that production reveals all at once. Not degrees of the same problem, genuinely separate axes. Identity (a service account has zero implicit trust and needs individually granted permission for every registry and every action, discovered across four distinct IAM errors). Lifecycle (instances start and stop, and anything not deliberately cached reloads on every start, the specific bug being a global variable loaded at import time, invisible until concurrent traffic spun up a second instance and reloaded it independently). Cost (every design decision, image size, eager vs. lazy loading, memory allocation, acquires a literal price the moment it leaves a laptop).

Naming conventions lie about deployment cost. Gemma 4 "E2B" describes a compute-efficiency figure, not the ~5.1B actual parameters and ~9.3GB actual download. A number in a model's name is not a number you can architect a hosting budget around.

Tooling designed for development actively obscures architecture. npm run dev made a TanStack Start SSR application behave, superficially, exactly like a static SPA, until an nginx-based Dockerfile, written for the assumption, turned out to be structurally incapable of running what had actually been built. The dev server's entire value proposition, hide the machinery, let you iterate fast, is precisely what makes it a bad source of architectural truth. Check what a build actually produces, not what the dev experience implies it produces.

The one-paragraph version, if this is all you read

A language model predicts tokens; how you sample from that prediction shapes the output as much as the weights do. Post-training doesn't add knowledge, it adds a goal, and the goal is sometimes aimed at the wrong thing. Evaluating any of this is bounded twice over, once by what a metric can mathematically see, and again by what a rubric thought to ask, and neither bound moves just because you make the judge bigger. And every one of these findings was learned on hardware that costs nothing, using models small enough to fail loudly, which turned out to be the actual pedagogical advantage of the zero-cost constraint from Post 1: small, cheap, visibly-flawed systems taught more, faster, than a single frontier model ever would have, because their seams were where I could actually see them.

What's actually reproducible

Every specific claim above traces to an experiment, not an assertion, and every experiment is re-runnable. The four-case judge study ships as JSON in the repo. The base/instruct comparison is one dropdown selection in the Model Zoo tab. The decoding tradeoff is one toggle in the Swing Set tab. None of this requires taking my word for anything.

Try it yourself: the live playground is at llm-playground.gayathri.dev; the "How to Use This" tab inside it is the fast orientation, this series is the slow one. Full source, deployment configs, and every experiment log referenced across all six posts: github.com/lovable-api/llm-playground.

A closing caveat, stated once for the whole series: every model here is small, every experiment is a handful of hand-picked cases chosen to make a mechanism visible, and none of it is a benchmark in the statistical sense. The value isn't in the specific numbers, it's that the mechanisms they demonstrate are real, well-documented in the literature at every scale, and now, for me, no longer abstract.