How LLMs Work: Tokens, Embeddings & Next-Token Prediction

Open WhatsApp and type "Happy". Your keyboard instantly suggests birthday. Type "Good" — it offers morning. Your phone has a tiny model that predicts the next word from the last one or two. Now imagine that same idea, but the model has read a huge chunk of the internet, remembers the entire conversation so far, and its guesses are so good they form essays, code and legal drafts. Congratulations — you already understand LLMs. This lesson just makes that understanding precise, because interviews will test the precision.

Tokens — the model's alphabet

First surprise: an LLM doesn't read words, and it doesn't read letters. It reads tokens — pieces of text, each mapped to an id number from a fixed dictionary (the vocabulary, typically 50,000–200,000 entries). Common words are one token. Rarer or longer words get split into pieces:

Rules of thumb worth memorizing: in English, 1 token ≈ 4 characters ≈ ¾ of a word, so 1,000 tokens ≈ 750 words. Hindi/Hinglish and code often cost more tokens per sentence than plain English, because the tokenizer's dictionary was built mostly from English-heavy data. You can count tokens yourself:

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("Generative AI is powerful")
print(tokens)
print(len(tokens), "tokens")

[5831, 48622, 15592, 374, 2410, 1285]
6 tokens

Why should you care so much about tokens? Because everything in the LLM world is measured and billed in tokens — the context window (how much fits), the price (per 1K tokens), the speed (tokens/second). An engineer who thinks in tokens designs cheaper, faster systems.

Embeddings — meaning as coordinates

Second surprise: after tokenization, each token becomes a vector — a long list of numbers (hundreds or thousands of them) called an embedding. And these numbers aren't random: they're learned so that similar meanings land near each other in this number-space. "Doctor" and "nurse" are neighbours; "doctor" and "cricket" live in different pin codes.

Think of it as a gigantic map of meaning. On a city map, nearby points are physically close; on the embedding map, nearby points are semantically close. Because meaning is now geometry, a computer can do math on it: "find me sentences similar to this question" becomes "find nearby vectors" — one distance formula. Park this thought carefully: it is the entire engine of semantic search and RAG (lesson 9), and the Practice Zone below makes you compute one by hand.

Next-token prediction — the only trick

Now the core. Given all tokens so far, the model produces one thing: a probability for every token in its vocabulary being next. Then one token is picked, appended, and the whole thing runs again. That's the loop. There is no step where it "looks up the answer" — prediction IS the answer.

Watch ChatGPT answer and you can literally see the loop — words appearing one at a time. That's not a loading animation; that's generation happening, one predicted token per step. A 300-word answer is roughly 400 runs of the loop.

Selection-round radar: "How does ChatGPT actually work?" is asked at every level. The winning shape: text → tokens → embeddings → the model predicts next-token probabilities → sampled token is appended → repeat. Five arrows, no math. Most candidates say "it's trained on data" and stop — the loop is what separates you.

How it got so good — training, in three acts

Act 1 — Pre-training. Show the model trillions of tokens of internet text, hiding the next token each time: "guess." Wrong guess → adjust the billions of internal numbers (weights) slightly → repeat, for months, on thousands of GPUs. To get good at this game, the model is forced to learn grammar, facts that repeat across sources, styles, even reasoning-shaped patterns — because they all improve next-token guesses.

Act 2 — Instruction tuning. Raw pre-trained models complete text; they don't follow orders. So they're further trained on examples of instructions with good responses — teaching the format "when asked, answer helpfully."

Act 3 — Alignment (RLHF and friends). Humans rank model answers; the model is tuned toward the preferred ones — more helpful, less harmful. That's the difference you feel between a raw model and a polished assistant.

What got stored after all this? Not the documents — just the weights: billions of numbers encoding the patterns of language and knowledge. The model is a compression of patterns, not a library of pages. Keep this; it explains the next section.

Wait — it writes poetry but can't count the r's in "strawberry"?

Ask a model "how many r's in strawberry?" and it may confidently say two (it's three). The reason isn't stupidity — it's the alphabet. The model sees tokens, and "strawberry" might be a single token — one opaque id number. The letters inside are invisible to it, the way you can't count the bricks in a house from a satellite photo. Character-level tasks (counting letters, reversing strings, strict spelling) fight the model's own input format.

This is a beautiful interview answer because it demonstrates understanding: the failure is a direct consequence of tokenization — the very first thing you learned today.

One idea, four mysteries solved

Next-token prediction isn't trivia — it's the master key. Why do models hallucinate? Because they generate plausible continuations, and plausible isn't always true (lesson 7). Why do prompts matter so much? Because the prompt is the "text so far" that all predictions continue from (lesson 6). Why is there a context window? Because the model can only attend to a limited number of tokens per prediction (lesson 5). Why does RAG work? Because if you paste the right facts into the "text so far," the most plausible continuation becomes the truth (lesson 9).

Common mistakes

• "A token is a word" — tokens are pieces; rare words split, spaces attach to tokens, and non-English text costs more tokens.
• "The model stores documents and searches them" — it stores weights (patterns). No lookup, no live internet.
• "Embeddings are encryption/compression of text" — they're coordinates of meaning; similar meaning = nearby vectors. Recovery of exact text isn't the point.
• Expecting reliable character-level work (letter counts, exact string reversal) — tokenization hides letters from the model.
• Thinking the model "knows" when it's unsure — the loop always produces something; confidence is in the probabilities, not the prose.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and three hands-on tasks, including computing a cosine similarity yourself — the exact math RAG runs a million times a day. Attempt first, reveal second.

Q1How LLMs workeasy

In LLM terminology, a token is:

Asked in

aAlways exactly one wordbA piece of text — often a word, sub-word or punctuation — that the model processes as one unitcOne character of textdA unit of GPU memory

Q2How LLMs workeasy

At its core, the ONE task a GPT-style LLM is trained to do is:

Asked in

aLook up answers in a built-in database of factsbPredict the next token, given all the tokens so farcTranslate text into a secret internal languagedSearch the internet in real time

Q3How LLMs workmedium

An embedding is:

Asked in

aA compressed ZIP file of the training databA list of numbers (a vector) representing a piece of text such that similar meanings get nearby vectorscThe final answer the model printsdA type of GPU

Q4How LLMs workmedium

GPT-style text generation is called autoregressive because:

Asked in

aThe model generates the whole answer in one single stepbEach newly generated token is appended to the input, and the model predicts the next token from that longer sequence — one token at a timecThe model automatically retrains itself after each answerdIt uses linear regression internally

Q5How LLMs workmedium

Why does an LLM often fail simple questions like "how many r's are in strawberry"?

Asked in

aThe model is too small to know the alphabetbThe model sees tokens, not letters — 'strawberry' may be one or two tokens, so it never directly observes individual characterscCounting is mathematically impossible for neural networksdThe training data contained no fruit names

Q6How LLMs workhard

During training, an LLM "learned" from trillions of tokens. What exactly got stored?

Asked in

aA copy of every document, retrievable on demandbBillions of numeric weights that encode statistical patterns of language — not the documents themselvescOnly the most recent one million documentsdURLs pointing to the original sources

Now the hands-on tasks:

import numpy as np

doctor  = np.array([0.9, 0.8, 0.1, 0.0])
nurse   = np.array([0.85, 0.75, 0.15, 0.05])
cricket = np.array([0.05, 0.1, 0.9, 0.85])

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

FAQ

Next lesson: the architecture that made all this possible — Lesson 3: Transformers, Without the Math →