In 2017, eight researchers at Google published a paper with a rockstar title — "Attention Is All You Need" — and quietly ended an era. Every headline AI you know — GPT, Claude, Gemini, Llama — is built on the architecture from that paper: the transformer (the T in GPT). Interviewers love asking about it, and here's the good news: at placement level they want intuition, not equations. So that's exactly what we'll build — with two diagrams and zero calculus.
The relay-race problem (life before transformers)
Before 2017, language models were mostly RNNs — recurrent neural networks. An RNN reads a sentence like a relay race: word 1 processes and passes a baton (a summary) to word 2, word 2 passes to word 3… By word 200, whatever word 3 contributed has been squeezed through 197 hand-offs. Long-range information faded, like a message in a game of Chinese whispers. And because each step needed the previous one, you couldn't parallelize — GPUs, which love doing thousands of things at once, sat half idle.
Two problems, then: forgetting across distance, and one-word-at-a-time processing. The transformer killed both with a single idea.
Self-attention — the group discussion
Replace the relay race with a group discussion: every word sits in one room and can look directly at every other word — no baton, no hand-offs. For each word, the model computes how much attention to pay to every other word, and then blends their information according to those weights. That computation is self-attention.
Look at what the word "it" does in the diagram. To predict anything sensible after "it", the model must know what "it" refers to. Attention solves this by weighing: "cat" gets a heavy weight (thick line), "milk" a light one. Nobody hand-coded that rule — the attention weights are learned from data, and they let any word connect to any other word directly, however far apart. Word #3 talking to word #900: one step, not 897 hand-offs.
In reality each layer runs many attention "heads" in parallel — one head might track pronoun references, another subject–verb pairs, another topic words. Multi-head attention just means several of these lookups running side by side, each learning to notice different relationships.
The full pipeline in one picture
Read it left to right. Tokens become embeddings (lesson 2's meaning-vectors). Then a stack of identical layers, each doing two things: self-attention (gather — every token looks around and collects what's relevant) and a feed-forward network (process — think about what was gathered). Stack this attention-then-think unit N times — modern models stack it dozens of times — and finish with the next-token probability scores you met in lesson 2. Attention gathers, feed-forward digests, repeat.
Wait — if everyone talks at once, who remembers the seating order?
Here's the catch the group discussion creates. The relay race knew word order for free (it read one by one). But if all words are processed simultaneously, then "dog bites man" and "man bites dog" look identical — same words, just a bag of them. Order has to be added back explicitly: positional encoding stamps each token's position (1st, 2nd, 3rd…) into its embedding before the discussion starts. Meaning + position go in together; word order is saved.
Selection-round radar: "Why do transformers need positional encoding?" is a favourite because it checks real understanding: the honest answer starts with "because attention itself has no notion of order — it processes all tokens in parallel...". If you can say that cause-and-effect, you've outscored most of the queue.
Encoder vs decoder — BERT, GPT and friends
The original transformer had two halves: an encoder that reads the input (seeing the whole sentence at once, both directions) and a decoder that generates output left-to-right. The famous models each kept the half they needed:
| Family | Keeps | Sees | Best at |
|---|---|---|---|
| BERT-style | encoder only | whole text, both directions | understanding: classification, search relevance |
| GPT-style (Claude, Gemini, Llama) | decoder only | only tokens before the current one (causal mask) | generating: chat, code, essays |
| T5-style | both halves | encoder reads all; decoder generates | input→output transforms: translation, summarization |
Memory hook: BERT = Both directions (understand), GPT = Generates (left-to-right). And the twist worth saying out loud in 2026: large decoder-only models have become so capable that they now handle understanding tasks too — which is why "GPT-style" dominates the landscape you'll meet in lesson 4.
Why transformers actually won: parallelism → scale
The quiet superpower isn't attention's elegance — it's that attention removed the sequential bottleneck. All tokens process at once, which is exactly the workload GPUs are built for. Suddenly you could train on vastly more data with vastly bigger models, and researchers discovered that quality kept climbing as you scaled up. That scaling race — more data, more parameters, more GPUs — is the direct line from a 2017 paper to ChatGPT. Transformers won because they made scale possible; scale is what made LLMs feel intelligent.
Common mistakes
- Diving into query/key/value matrix math in a placement interview — give intuition first; add math only if explicitly asked.
- Saying transformers "remember better than RNNs because they have more memory" — the win is direct connections (any-to-any attention), not a bigger baton.
- Mixing up the families: BERT can't write essays (never trained to generate); GPT reads only left-to-right while generating.
- Forgetting positional encoding exists — then "dog bites man" = "man bites dog".
- Calling attention "the model searching Google" — attention looks at the input text itself, nothing external.
Quick recap
| Concept | One-liner |
|---|---|
| RNN problem | relay race: forgets across distance, can't parallelize |
| Self-attention | every token weighs every other token and blends what's relevant |
| Multi-head | several attention lookups in parallel, noticing different relations |
| Layer | attention (gather) + feed-forward (digest), stacked N times |
| Positional encoding | stamps order back in, since parallel processing loses it |
| Families | BERT = encoder/understand · GPT = decoder/generate · T5 = both |
Practice Zone — PYQs from real selection rounds
Six MCQs plus hands-on tasks — including the classic trophy-suitcase sentence and a 60-second explanation drill. Attempt first, reveal second.
The transformer architecture was introduced in which famous 2017 paper?
Asked in
The main problem with RNNs that transformers solved was:
Asked in
Self-attention computes, for each token:
Asked in
GPT-style models are decoder-only transformers. This means:
Asked in
Why do transformers need positional encoding?
Asked in
BERT vs GPT — the correct pairing is:
Asked in
Hands-on tasks:
Classic test sentence: "The trophy didn't fit in the suitcase because it was too big." Which earlier word should "it" attend to most strongly? Now change "big" to "small" — what should happen to the attention, and why is this pair of sentences hard for machines?
Asked in
You have three tasks: (1) classify support tickets by sentiment, (2) write personalized marketing emails, (3) translate documentation English→Hindi. For each, say which transformer flavour fits naturally: encoder-only (BERT-style), decoder-only (GPT-style), or encoder-decoder (T5-style).
Asked in
Interviewers at service companies often just say: "Explain the transformer architecture." You have ~60 seconds. Draft your answer (no math allowed), then compare.
Asked in
FAQ
Do I need to know the attention formula (QKV) for placements?
Usually not for freshers rounds — intuition suffices. If asked, one sentence works: each token makes a query, compares it against every token's key to get weights, and blends their values accordingly — like searching "who here is relevant to me?" and averaging the best answers.
What does the 'GPT' abbreviation actually mean?
Generative Pre-trained Transformer — generative (creates text), pre-trained (learned from huge data before you ever prompt it), transformer (this lesson's architecture). Three lessons of this course hiding in one acronym.
Are transformers only for text?
No — the same attention machinery now powers image models (Vision Transformers), audio, video and multimodal models. Anything you can chop into a sequence of tokens/patches, a transformer can process.
Next lesson: meet the actual models — Lesson 4: The LLM Landscape →