Deep Diving into LLMs, Part 2: Training and Inference
In part 1, I covered how raw internet text gets filtered and turned into tokens. This part picks up right after that: once you have a giant sequence of tokens, how does a neural network actually learn from it, and what does it look like when you finally talk to the trained result?
Training the Neural Network: Predicting One Token at a Time
Once text is turned into a long sequence of tokens, training works like this: take a window of tokens (the "context"), and ask the network to predict what token comes next.
Here's the part that actually made this click for me. The network doesn't just predict "the next token", it outputs a full probability distribution across every single token in its vocabulary. So if your vocabulary has 100,000 possible tokens, the model spits out 100,000 numbers, each one representing how likely that specific token is to come next. In the very beginning, before any training has happened, the network's parameters are random, so these probabilities are basically noise. Every token looks about as likely as any other.
Training is the process of nudging those probabilities in the right direction. You already know the correct next token because you sampled it straight from real text. So you compare what the model guessed against what actually came next, and you adjust the model's internal parameters (there can be billions of them) just slightly, so that next time it sees that same context, it assigns a bit more probability to the correct token and a bit less to everything else.
This single adjustment happens across huge batches of tokens simultaneously, over and over, millions of times. That's the whole training loop: predict, compare to the real answer, nudge the parameters, repeat.
The number you watch as this happens is called the loss. It's a single number that tells you how wrong the model currently is, and lower is better. Watching a real training run, you literally see loss drop step by step, and as it drops, the sample text the model generates goes from complete gibberish to something that starts resembling coherent language, then eventually to fluent, grammatically correct sentences. It's slow and incremental. Nobody flips a switch and gets a working model, it's thousands of tiny corrections stacking up over days of compute.
Inference: Watching the Model Guess, One Token at a Time
Inference is what happens when you actually talk to a trained model. No more learning is happening here, the parameters are frozen. The model just takes whatever tokens you've given it, produces a probability distribution over what could come next, and then samples one token from that distribution, kind of like flipping a biased coin where more likely tokens are more likely to get picked. That new token gets appended, and the whole process repeats for the next token, and the next, and so on.
Because it's genuinely sampling rather than always picking the single most likely option, the same prompt can produce different completions every time you run it.
One of the wildest demos in this part of the video was feeding the model the opening line of a Wikipedia article, almost word for word, and watching it continue by reciting the rest of the article nearly verbatim, sentence by sentence, purely from memory. That happens because the model saw high quality sources like Wikipedia multiple times during training (sort of like re-reading the same page ten times), so it can genuinely memorize certain passages. But if you let it keep generating, eventually it drifts. It runs out of exact memorized text and starts blending in its own remix, generating something that sounds similar in style but isn't a direct copy of anything it saw. That transition, from "reciting" to "improvising", is a nice visual for what these models are actually doing under the hood: part memorization, part statistical pattern completion.
Next up in part 3: comparing GPT-2 to today's models to see just how much the bar has moved, and what a "base model" actually is before it becomes something like ChatGPT.