Deep Diving into LLMs, Part 3: GPT-2, Today's Models, and What a Base Model Really Is

This is the final part of my notes from Karpathy's deep dive video. Part 1 covered tokenization, part 2 covered training and inference. This part covers the scale comparison that put everything in perspective for me, and the base model demo that was honestly my favorite moment in the whole hour.

GPT-2 vs Today: How Fast the Bar Moved

This comparison genuinely put things in perspective for me. GPT-2, released back in 2019, had 1.6 billion parameters. At the time, that number was mind blowing. Today, frontier models are sitting somewhere around hundreds of billions to over a trillion parameters, and even open weight 7B or 8B models that people casually run on their own laptops barely register as impressive anymore. We've normalized a scale of model that would have seemed like science fiction a few years ago.

Context length tells the same story. GPT-2 could only look at 1,024 tokens at a time when predicting the next token, meaning its entire "memory" of the conversation or document was capped at roughly that many tokens. Modern models routinely handle context windows in the hundreds of thousands of tokens, some pushing toward a million. That's the difference between a model that can barely remember the start of a long email and one that can hold an entire codebase or book in its context at once.

Training data scaled up just as dramatically. GPT-2 was trained on roughly 100 billion tokens. Modern datasets like FineWeb sit around 15 trillion tokens, which is roughly 150 times more text. And the cost of training has actually gone down over that same period, not up, because data pipelines got more refined and hardware and software got dramatically faster. What cost an estimated $40,000 to train in 2019 could reportedly be reproduced for somewhere close to $100 today.

Base Models: Expensive Autocomplete That Can Be Tricked Into Being Useful

This was probably my favorite part of the whole hour. A "base model," the raw output of all that pretraining, isn't an assistant. It has no concept of answering your questions. All it knows how to do is continue a sequence of tokens based on the statistical patterns it absorbed from its training data. Ask it "what is 2 + 2" and it won't answer, it'll just keep generating text that could plausibly follow that string, sometimes drifting into something totally unrelated.

But here's the clever part: since it's just pattern completing, you can trick it into acting useful by shaping your prompt cleverly. Two examples stood out:

• Give it several examples of a pattern (like English to Korean word pairs) and then leave the last one incomplete. The model picks up on the structure through what's called in-context learning and completes the pattern correctly, without ever being "taught" translation directly.
• Write a prompt that looks like a script of a conversation between a "helpful AI assistant" and a "human," complete with a few example exchanges. The base model, just trying to continue the text naturally, starts playing the role of the assistant and generates surprisingly coherent, helpful sounding answers, purely because that's the most statistically likely continuation of a document shaped like that.

That last trick was basically a preview of what post-training (the next stage, turning a base model into something like ChatGPT) actually formalizes. But seeing it work with nothing but clever prompting on a raw autocomplete engine was a great "oh, that's why prompting matters so much" moment.