Who Judges the Judges? Small Models Grading Small Models

Post 4 of 7. Post 3 ended with a well-formatted answer containing a logic error nobody would catch at a glance. This post is about who, or what, does the catching. It contains the most surprising results of the project.

---

For everything in the first three posts, the evaluator was me. I read the outputs, noticed the loops, caught the lactose fabrication, spotted the boiled-egg-before-boiling slip. Eyeballs work, but they don't scale, they don't replicate, and they quietly encode whatever I happened to be looking for. If evaluation is the third stage of the LLM pipeline, and every course outline says it is, I wanted to build it, not read about it.

The build went in two phases: metrics you can compute, then models you can consult. The second phase produced results I did not expect, so this post has an actual experiment with actual tables.

Phase 1: Metrics you can compute

Repetition rate quantifies the loops from Post 2. Take every 3-word window in the text, count what fraction are duplicates of earlier windows:

def compute_repetition_rate(text, n=3):
    words = text.split()
    ngrams = [tuple(words[i:i+n]) for i in range(len(words) - n + 1)]
    return 1 - (len(set(ngrams)) / len(ngrams))

Six lines, fully interpretable, and it converts "this output loops" from an impression into a number. The greedy-decoding disasters from Post 2 score dramatically high; the temperature-0.7 outputs score near zero. Eyeball, confirmed.

Perplexity is the standard fluency metric: exp of the average negative log-likelihood the model assigns to a text, informally, how surprised the model is by it, where low surprise means the text matches what the model expects language to look like. The implementation is a few lines (pass the text as both input and labels; the loss is the average NLL), but the interpretation carries two traps, and I stepped in both.

Trap one: low perplexity is not good. Degenerate looping text scores very low self-perplexity, the model is maximally unsurprised by its own repetition, which is precisely why greedy decoding produced it. A perplexity score has no valence on its own; extremely low self-perplexity is a degeneracy signature, not a quality signal.

Trap two: perplexity doesn't compare across models. My early dashboard happily charted GPT-2's perplexity next to Qwen's next to TinyLlama's, and the chart was meaningless. Perplexity is surprise per token, and each model has its own tokenizer, GPT-2's vocabulary is ~50K tokens, Qwen's ~150K. The same sentence is a different sequence of different units under each model, so the per-token averages live on different scales. What's valid: comparing the same model across decoding settings. What's not: ranking different model families by raw perplexity. I rebuilt the dashboard to compute perplexity only through the eyes of the model that generated the text, and the invalid chart became a caveat in my notes instead of a claim in this post. Catching that before publishing, rather than after, is most of what "research posture" means at my scale.

But here's the ceiling on all of phase 1, and it's low: run these metrics on the lactose answer from Post 3, the fluent, confident, entirely fabricated chemistry. Repetition rate: fine. Perplexity: fine. Word count: fine. Truth is invisible to every metric that only consults the text's statistics. Something has to actually understand the content, and in 2026 the standard answer is: another model.

Phase 2: Models you can consult

LLM-as-judge: give a model the prompt, the response, and a rubric; get back structured scores. My rubric asked for four criteria on a 1–5 scale, relevance, coherence, factual plausibility, helpfulness, returned as JSON with a sentence of reasoning. Judging temperature was set to 0.3, low, because judges should be consistent rather than creative.

The known weakness of the method is that the judge is a model with all the failure modes of models, and the standard advice is to use a judge substantially more capable than what it judges. Which raises the question this post exists to answer: how much does judge capability actually matter? So I ran three judges spanning the capability range I had access to for free:

• Qwen2.5-0.5B-Instruct, local, the smallest chat model in the playground
• Qwen2.5-1.5B-Instruct, local, 3x larger, same family
• Gemini 2.5 Flash, frontier-class, via free API tier

One implementation note that is itself a finding: small judges break format. Despite explicit "respond ONLY with JSON" instructions, the local judges intermittently wrapped output in commentary or markdown fences, so the pipeline extracts the first {...} block by regex and records a parse failure otherwise. Instruction-following is a capability, capabilities are partial at 0.5B, and your infrastructure has to budget for that.

The four-case study

I selected four prompt/response pairs from the project's own outputs, each engineered to probe a different failure axis. Same rubric, same three judges, all runs preserved.

Case 1, good but subtly flawed. The Qwen-Instruct egg answer from Post 3: beautifully formatted, contains the boiled-before-boiling logic inversion and the "yolk escape" claim.

Both local judges missed the flaw entirely. The 1.5B judge scored it perfect and asserted "no inaccuracies." The 0.5B judge did something stranger: its reasoning praised the response for "safety precautions such as not using raw eggs", a sentence that appears nowhere in the response. It hallucinated evidence to support the grade. Gemini named the exact problems: incomplete, cuts off mid-sentence, ice-water claim factually wrong.

Case 2, confidently false. The GPT-2 lactose fabrication: fluent, declarative, every clause wrong.

The 0.5B judge scored it 4s again, describing lactose-oil as legitimate content that merely lacked detail about "temperature range." The 1.5B judge caught it cleanly this time, agreeing with Gemini, which, characteristically, was the only judge to name the specific error ("egregious factual errors about lactose").

Case 3, fluent but off-topic. A deliberately mismatched pair: the egg prompt with a poetic birds-and-sunlight response generated for a different prompt entirely.

Third consecutive 4/4/4/4 from the smallest judge, and its reasoning this time claimed the sun's warmth "aligns with the concept of boiling an egg," a connection that exists nowhere except in the justification itself. The other two judges correctly identified a non-answer.

Case 4, the rubric-gap probe. The violent "Once upon a time" continuation from Post 2: coherent-ish narrative, repetitive, disturbing content. My rubric contains no criterion about safety or appropriateness. Would any judge flag it anyway?

No. Not one judge's reasoning mentions the violence, including Gemini's, which awarded this response the highest relevance score of the entire study (a bloody cliff scene is, mechanically, a valid continuation of a story opener) while correctly docking it for repetition. The judges answered exactly the four questions I asked, and I never asked the question that mattered.

Three findings

1. The smallest judge's scores are decoupled from quality. Across a decent answer, a fabrication, and an irrelevant answer, the 0.5B judge returned a near-constant 4/4/4/4, and, on every case, generated reasoning that rationalized the score rather than diagnosed the response, twice inventing content or logic that didn't exist. The scores aren't lenient; they're uninformative, with a fluent explanation attached. This is the failure mode to fear, because a judge that always says 4 with confident-sounding reasons looks exactly like evaluation while measuring nothing.

2. Judge capability helps, but not monotonically, and failure modes are error-type-specific. The 1.5B judge reliably caught what I'd call topical failures, wrong subject, non-answers (Cases 2, 3), but was fully fooled by the well-formatted response with an internal logic error (Case 1), where it was more confidently wrong than its smaller sibling. Surface polish defeated it; irrelevance didn't. Only the frontier judge caught the reasoning-level flaw. "Use a bigger judge" is real advice, but the relationship between judge scale and judge reliability is not a smooth curve, it depends on what kind of error you need caught.

3. The rubric bounds what evaluation can see, independent of judge capability. Case 4 is the cleanest result in the study: a maximally capable judge, given a rubric with no safety criterion, will not report a safety problem. Evaluation measures what you ask. Judge quality determines how well the questions get answered; rubric design determines which questions exist. These are separate failure surfaces, and the second one is entirely the evaluator's responsibility.

Why this matters beyond a toy project

Scale these findings up and you get the evaluation landscape as it actually exists. Automated benchmarks and leaderboards are rubrics, they measure what their designers thought to ask, and models optimize toward exactly that. LLM-as-judge pipelines at production scale inherit the leniency and blind spots of whatever judge they use, which is why serious setups use frontier judges, calibrate them against human raters, and still treat the scores with suspicion. And the Case 1 result, formatting defeating a mid-size judge, is a miniature of a documented industry problem: preference-trained raters, human and machine alike, reward polish, and polish is cheaper to produce than correctness.

Try it yourself: The live playground's Judge's Bench tab lets you submit any prompt/response pair to all three judges; the four cases from this post ship in the repo as JSON so you can replay them exactly, and the Report Card tab charts every judgment you run.

Caveats, which deserve real space here: n = 4 hand-selected stress cases, chosen to probe specific failure axes, not a sampled benchmark, and nothing here is a rate. Each case ran once per judge; at temperature 0.3 there is still run-to-run variance I did not measure. One rubric, one phrasing, judge behavior is known to be sensitive to rubric wording, and I did not test alternatives. The Gemini API resolved my model request through an alias, so the exact judge version is the provider's current Flash tier rather than a pinned checkpoint. These results establish that the failure modes exist and are easy to hit; they do not establish how often you'd hit them in the wild. For a learning project, existence was the point.