Do AI Models Ever Go Obsolete?

Every few months a new model shows up and the previous one quietly drops out of the conversation. GPT-3 gave way to GPT-4, then GPT-4o, and the cycle keeps going. Three questions are worth sitting with: what happens to the old models, how does a lab actually build the next one, and what's really changing under the hood when it does.

Old models don't disappear, they change jobs

GPT-2 isn't competing with GPT-4 on any task. But it's still useful, just for different reasons:

• It's small enough to study. Open-source and lightweight, it's the version people use to actually see how a transformer works, without needing a cluster to run it.
• It runs anywhere. No data center required, a laptop, sometimes a phone, no internet connection needed.
• It's cheap to fine-tune. Training GPT-2-small on a narrow task (a specific voice, a specific format) is something you can do on a single consumer GPU in under an hour. Doing the same to a frontier model is a different budget entirely.

Commercially, older models retire fast. As teaching tools and cheap fine-tuning targets, they stick around.

New generations start from zero, not from the old model

You can't take GPT-4's weights and grow them into GPT-5. A new generation usually means a new architecture, different layer structure, longer context window, different training objective, and none of that transfers from a random initialization.

What does carry over is the training data. The old models don't hand down their weights; they hand down the material:

• Synthetic data. High-quality human text is a limited resource at this point, so labs increasingly use their current best model to generate large volumes of reasoning traces, code problems, and structured examples to train the next one.
• Distillation, worth being precise here, since this term gets used loosely. It means a larger, already-trained model teaches a smaller model how to approximate its outputs, cutting training cost for that smaller model. It's not really how frontier models train each other generation to generation; it's a separate technique for producing efficient smaller models from large ones.

Worth a caveat: not every model update is a from-scratch rebuild. Point releases and mid-generation updates sometimes continue training from an existing checkpoint rather than reinitializing. The "brand new brain" story applies most cleanly to actual generational jumps, not every version bump.

The token dictionary changes, here's what that actually looks like

Models don't read words. They read tokens, chunks of text mapped to integers by a tokenizer. Whether those integers stay stable across model releases depends on whether it's a minor update or a new generation.

Same generation, same dictionary. GPT-4 and GPT-3.5-turbo both use cl100k_base, same ~100K-token vocabulary, same IDs. That's why swapping models mid-project doesn't break your tokenization logic.

New generation, full re-index. GPT-4o moved to o200k_base, a ~200K-token vocabulary built from different training data. Every ID shifts.

The reason to rebuild the vocabulary at all: non-English text gets tokenized far less efficiently under the older, English-heavy vocabularies. I ran the same sentence through both tokenizers directly to check, rather than taking that on faith:

"The weather is nice today and I plan to go for a walk", and its Hindi, Japanese, and Arabic equivalents:

English barely moves. Hindi drops by more than half. That's not a rounding difference, it's the difference between a sentence costing 3x as much to process as its English equivalent versus roughly the same. Non-English text was getting chopped into small, expensive fragments under the older vocabulary; the newer one gives whole words and common substrings their own token far more often.

The same expansion helps with code and math notation, more symbols and patterns get single tokens instead of being split character by character.

Where that leaves things

Old models don't vanish, they specialize into teaching and low-stakes fine-tuning. New generations aren't built by editing the old model, they're built by starting over and feeding the new one better material. And the tokenizer isn't cosmetic, it's a real efficiency decision that shows up as a measurable cost difference the moment you step outside English.