Jul 13, 20265 min read

Deep Diving into LLMs, Part 1: How the Internet Becomes Tokens

I've been watching Andrej Karpathy's deep dive into how models like ChatGPT actually get built, and I wanted to break down what I learned into a short series instead of one giant post. This first part covers the very beginning of the pipeline: how raw internet text gets filtered down into a clean dataset, and how that text gets turned into the tokens a neural network can actually work with.

Turning Words into Tokens (and why Byte Pair Encoding exists)

Before any of the "intelligence" stuff happens, a language model needs a way to turn text into numbers it can actually compute with. That sounds trivial until you realize there are real tradeoffs involved.

The most obvious approach is to just use raw bytes. Every character in "hello" can be represented as UTF-8 bits, which then get grouped into bytes. A byte can only take 256 possible values, so now you have a vocabulary of 256 symbols. Simple, but there's a catch: your sequences get really long. A short sentence turns into dozens of individual byte-tokens, and sequence length is expensive for a neural network to process. The longer the sequence, the more compute you burn just to read the input.

So the question becomes: can we shrink the sequence length by using a bigger vocabulary of symbols? That's exactly what Byte Pair Encoding (BPE) does.

Here's the intuition with a toy example. Say your training text has the words:

low low low lower lowest

At the byte/character level this is a long string of individual letters: l-o-w-space-l-o-w-space-l-o-w-space-l-o-w-e-r-space-l-o-w-e-s-t.

BPE works by repeatedly finding the most frequent adjacent pair of symbols and merging them into a single new symbol.

Step 1: Look through the sequence and count pairs. "l" followed by "o" shows up constantly. Merge it into a new symbol, call it "lo".
Step 2: Now scan again. "lo" followed by "w" is very common. Merge that into "low".
Step 3: Keep going. Maybe "low" followed by "e" starts showing up (from "lower" and "lowest"), so that becomes another merged symbol if it's frequent enough.

Every merge does two things at once: it shrinks the length of the sequence (because two symbols become one), and it grows the size of the vocabulary (because you just minted a brand new symbol). You keep repeating this merging process as many times as you want, and each round trades vocabulary size for shorter sequences.

In production, this isn't run on a five word toy sentence, it's run on enormous amounts of text, and the merging continues until you hit a target vocabulary size. GPT-4, for example, lands on 100,277 distinct tokens after this process. That's the sweet spot between "too few symbols, sequences too long" and "too many symbols, no real compression benefit."

A fun side effect of this is that tokenization gets weirdly picky. "hello world" gets tokenized differently than "Hello World", and even adding an extra space between two words changes which tokens get produced. The model doesn't see letters or words the way we do, it sees whatever chunks the BPE merges happened to produce.

Next up in part 2: what actually happens once that data is tokenized, how a neural network learns to predict the next token, and what it looks like to watch a model go from generating gibberish to fluent text.