Type "something to keep my coffee hot on the drive" into an old e-commerce search box and you get zero results — no product is literally called that. Type it into a modern one and you get travel mugs. Nobody added those words to the product page. The search engine understood what you meant. This lesson is how that works, where it fails, and why every serious system ends up running two kinds of search at once.
How keyword search works (and why it survived)
Classic search builds an inverted index: for every word, a list of the documents containing it. Search "casual leave" and it intersects two lists — fast, exact, decades-proven. Ranking uses BM25, which scores a document higher when a query term appears often in it, when that term is rare across the corpus (so "leave" counts more than "the"), and it normalizes for document length so long documents don't win by accident.
It has one fatal weakness — it matches spellings, not meanings — and one superpower people forget: it nails exact rare terms like RBI/2024/17, HRA-104 or a SKU, which is precisely where embeddings get vague. Hold on to that; it's why this lesson ends in hybrid search rather than "embeddings won".
How semantic search works
Straight from lesson 2: embed every chunk once at indexing time, embed the query at search time, return the chunks whose vectors are nearest. Three lines of numpy is a working (tiny) semantic search engine:
import numpy as np
# indexed once
chunks = ["Employees get 12 casual leaves per calendar year.",
"Laptops must be returned on the last working day.",
"Notice period for confirmed employees is 60 days."]
matrix = np.array([embed(c) for c in chunks]) # shape (3, 1536)
# per query
q = np.array(embed("how many CLs do I get?"))
scores = matrix @ q / (np.linalg.norm(matrix, axis=1) * np.linalg.norm(q))
print(chunks[int(np.argmax(scores))])Result
Employees get 12 casual leaves per calendar year.
One matrix multiply scores every chunk at once — that's the whole search engine.
Dense vs sparse — the vocabulary interviewers use
Two words you should be able to define instantly. Dense retrieval uses embeddings: a compact vector (say 768 numbers) where nearly every number carries learned meaning — good at paraphrase, weak at exact identifiers. Sparse retrieval is the keyword family (BM25, TF-IDF): conceptually one dimension per vocabulary word, almost all zeros for any given document — good at exact terms, blind to synonyms.
Same job, opposite strengths. Which is exactly why the answer to "which should we use?" is usually "both".
Wait — comparing against every chunk? For ten million chunks?
The numpy version above compares the query to every vector. At 300 chunks that's instant. At 10 million it's a multi-second stall per query, and your bot is unusable.
The fix is ANN — approximate nearest neighbour search. Instead of scanning, you build an index that lets the searchnavigate toward the right neighbourhood: HNSW builds a layered graph and hops greedily toward closer vectors (fly to the city, then drive to the street); IVF clusters vectors and only searches the nearest clusters.
The word doing the work is approximate: you might occasionally miss a true nearest neighbour. Typical recall sits around 95–99%, tunable — search more of the graph for better recall and worse latency. That trade — a tiny chance of a miss for a thousand-fold speed-up — is what makes billion-scale vector search possible, and it's the substance of lesson 4.
Metadata filters — the underrated fix
A user searches "dog collar" and gets dog food. Not a bug: those phrases genuinely live near each other in meaning space. No embedding model will cleanly separate them, because semantically they're neighbours.
The fix isn't a better model — it's structure. Store metadata with each chunk (category, product line, language, department, date, permissions) and filter before or during the vector search:
results = col.query(
query_texts=["dog collar"],
n_results=5,
where={"category": "accessories"}, # structural, not semantic
)Filters express rules embeddings can only approximate — and they do double duty as access control: an intern's query filtered to visibility: all_staff simply cannot retrieve the compensation sheet. Security belongs in the filter, never in the prompt — an instruction can be argued around; a query filter cannot.
Why production search is hybrid
Real query logs are a mix: intent queries ("something for a friend who cooks"), identifier queries ("MZ-4471-B"), and attribute queries ("under ₹5,000"). Dense handles the first, sparse the second, filters the third.
Hybrid search runs dense and sparse in parallel and merges their rankings. The catch: cosine sits around 0–1 while BM25 is unbounded, so you can't just add the scores. The common fix is Reciprocal Rank Fusion — score by each document's rank, never its raw score, so the scales never need reconciling and documents appearing in both lists rise to the top. (You'll implement RRF in the Practice Zone, and lesson 6 builds the full funnel around it.)
Selection-round radar: "keyword vs semantic search" is a standard warm-up; the follow-up that separates candidates is "so which would you use?". Answer: both, plus metadata filters — then name one query type each handles that the other can't. Concrete examples beat definitions here.
Semantic search isn't only for chatbots
The same index earns its keep in plenty of non-LLM products: deduplicating support tickets ("is this the same issue we solved last week?"), matching resumes to job descriptions, recommending products from descriptions, clustering feedback into themes, and flagging near-duplicate content. Worth mentioning in interviews — it shows you see the infrastructure, not just the chatbot.
Common mistakes
- Assuming semantic search replaces keyword search — it loses badly on exact codes and identifiers.
- Brute-force scanning at scale instead of using an ANN index.
- Skipping metadata, then trying to fix category confusion with a bigger embedding model.
- Enforcing permissions in the prompt rather than in the query filter.
- Adding raw cosine and BM25 scores together — incomparable scales; use RRF or normalize.
- Forgetting that ANN is approximate, then being surprised by an occasional missed chunk.
Quick recap
| Concept | One-liner |
|---|---|
| Keyword / BM25 | matches words; rare terms score higher; unbeatable on exact codes |
| Semantic search | matches meaning via embeddings; handles paraphrase and synonyms |
| Dense vs sparse | compact learned vectors vs huge mostly-zero word vectors |
| ANN (HNSW, IVF) | navigate instead of scan — huge speed-up for ~95–99% recall |
| Metadata filters | structural rules embeddings can't express — and where security lives |
| Hybrid + RRF | run both searches, fuse by rank, not by incomparable scores |
Practice Zone — PYQs from real selection rounds
Six MCQs and two tasks — route four real queries to the right search type, then implement Reciprocal Rank Fusion yourself.
The core difference between keyword search and semantic search is:
Asked in
What is BM25?
Asked in
'Dense' vs 'sparse' retrieval refers to:
Asked in
In vector search, what does ANN (approximate nearest neighbour) buy you, and what does it cost?
Asked in
A search over 5 million product descriptions returns semantically related but *wrong-category* items (dog food for a 'dog collar' query). The cheapest structural fix is:
Asked in
Semantic search is useful beyond RAG. Which of these is NOT a natural application?
Asked in
Hands-on tasks:
For each query over a product catalogue, say whether dense (vector), sparse (BM25) or hybrid retrieval serves it best: 1) "something to keep my coffee hot on the drive", 2) "SKU MZ-4471-B", 3) "noise cancelling headphones under 5000", 4) "gift for someone who loves cooking".
Asked in
Hybrid search gives you two ranked lists (vector and BM25) with incomparable score scales. Implement Reciprocal Rank Fusion (RRF) to merge them, and explain why RRF avoids the score-scale problem.
Asked in
FAQ
Is Elasticsearch keyword or semantic search?
Historically keyword (BM25), but modern versions support vector fields too — so it can serve as a hybrid engine. That pattern is common: mature search systems added vectors, and vector databases added keyword search, converging from both directions.
How do I know whether my ANN index is losing recall?
Measure it: run a sample of queries with brute-force exact search, then with the ANN index, and compare overlap. If recall is unacceptable, tune the index parameters (more graph exploration, more clusters probed) and pay the latency.
Should filters run before or after the vector search?
Ideally during — good vector databases apply filters inside the search so you still get k results. Filtering after retrieval can leave you with two results when you asked for five, and filtering before can be slow if it scans everything. It's an implementation detail worth checking in your database's docs.
Next lesson: the system that stores and searches all these vectors — Lesson 4: Vector Databases Explained →