You now know how to turn text into vectors and how to compare them. So why does a whole category of database exist for this? Because of one number: ten million. Comparing a query against ten million vectors, on every request, from a Python loop, is the difference between a demo and a product. A vector database is infrastructure built around one operation — find the nearest vectors, fast — plus the boring, essential things a demo forgets: filters, updates, deletes, persistence and permissions.
What a vector database actually stores
Not just vectors. Every record holds three things, and forgetting the second one is a beginner classic:
| Part | Why it must be there |
|---|---|
| The embedding vector | what similarity search actually compares |
| The original chunk text | the LLM reads text, not numbers — this is what goes into the prompt |
| Metadata | source file, section, date, language, permissions — powers citations, filters and access control |
You search by vector, but you prompt with text and you govern with metadata. A store that keeps only vectors would return row ids you couldn't show a user or feed a model.
How HNSW makes the impossible fast
Brute force is O(n) per query: ten million vectors, ten million comparisons, every single time. HNSW (Hierarchical Navigable Small World) — the index most vector databases default to — avoids scanning entirely.
Think of travelling across India. You don't inspect every street in the country: you take a flight to the right city (a sparse top layer with long-range links), then a train across town (a denser middle layer), then walk the last few hundred metres (the bottom layer, where every vector lives). At each layer you greedily step toward whichever neighbour is closer to your target, then drop a level and refine.
The result: roughly logarithmic search instead of linear — milliseconds over millions of vectors. The cost is memory (the graph's links) and approximation: occasionally the greedy walk misses the true nearest neighbour. Tunable knobs trade recall against latency, and that trade is the entire reason billion-scale semantic search exists at all.
The landscape — and how to choose without hype
| Option | What it is | Good when |
|---|---|---|
| FAISS | a library you run inside your own process | prototypes, notebooks, full control, no service to operate |
| Chroma | lightweight, embeddable, developer-friendly | learning, small/medium apps, fast local iteration |
| pgvector | a Postgres extension | your data already lives in Postgres; you want vectors, joins and transactions in one system |
| Pinecone | fully managed cloud service | you want scale and reliability without operating anything |
| Weaviate / Milvus / Qdrant | full-featured vector databases (self-host or managed) | large scale, hybrid search, filtering, multi-tenancy |
The honest selection criteria, in order: scale (thousands vs billions of vectors), filtering needs (rich metadata queries are not equal across products), operational appetite (who gets paged at 2 a.m.?), data-residency rules, and what you already run — "one less system to operate" is a legitimate architectural win, which is exactly the pgvector argument.
💡 Tip: if an interviewer asks "which vector database would you pick?", don't name a favourite. Ask about scale, filtering, and who operates it — then choose. The reasoning is the answer; the product name is a detail.
Wait — how do I stop an intern retrieving the salary sheet?
This is where a vector database stops being a maths toy and becomes infrastructure. The wrong answer is to retrieve everything and instruct the model to ignore what the user shouldn't see — a prompt is not an access-control layer. Anything in the context window can leak, through clever questioning or a plain bug.
The right answer: store permissions as metadata and enforce them inside retrieval, server-side.
# permissions live with the chunk, applied at query time
results = col.query(
query_texts=[user_question],
n_results=5,
where={"$and": [
{"visibility": {"$in": user_scopes}}, # from the session, never the client
{"status": "current"}, # withdrawn documents excluded
]},
)The model cannot leak what it never received. For SaaS with multiple customers, go further: separate namespaces or indexes per tenant, so cross-tenant retrieval is impossible by construction rather than by correctness of a filter.
Keeping the index fresh — the delete nobody writes
Documents change. The maintenance loop is: re-chunk the changed document, re-embed its chunks, upsert them by stable ids… and delete the chunks that no longer exist.
That last step is the one teams forget, and it produces one of the nastiest RAG bugs there is: a policy is withdrawn, its chunks stay in the index, and the bot keeps citing it — confidently, with a citation, for months. Give every chunk a deterministic id derived from its source (policy_v3::sec4::chunk2), reconcile per document on each run, and alert when the indexed chunk count drifts from the source. Lesson 10 turns this into a full postmortem.
Sizing and cost — napkin maths you should be able to do
Interviewers like this because it's quick and reveals whether you've thought about scale:
vectors = 2_000_000 # 500k documents x 4 chunks
dims, bytes_per_float = 1536, 4
raw_gb = vectors * dims * bytes_per_float / 1024**3
print(f"{raw_gb:.1f} GB of raw vectors")Result
11.4 GB of raw vectors
Plus the HNSW graph, the chunk text, metadata, and any replicas — the real footprint is meaningfully larger.
Two levers follow immediately: a smaller embedding dimension (768 or 384 halves or quarters this) and quantization — storing vectors at reduced precision, trading a little recall for a large memory saving. Both are worth naming when asked "how would you handle 100 million chunks?".
Common mistakes
- Storing vectors without the chunk text — nothing to put in the prompt, nothing to cite.
- Upserting without deleting — withdrawn content stays retrievable forever.
- Enforcing permissions in the prompt instead of the query filter.
- Choosing a database by popularity rather than by scale, filtering needs and who operates it.
- Ignoring that ANN is approximate, then treating a missed chunk as an embedding failure.
- Skipping the sizing calculation until the RAM bill arrives.
Quick recap
| Concept | One-liner |
|---|---|
| Stores | vector + chunk text + metadata — all three are load-bearing |
| HNSW | layered graph; navigate instead of scan; approximate but ~log time |
| Library vs service | FAISS/Chroma in-process; Pinecone/Weaviate/Milvus as systems |
| pgvector | vectors inside the Postgres you already run |
| Security | permissions as metadata filters (or per-tenant namespaces), never prompt rules |
| Maintenance | stable chunk ids · upsert AND delete · alert on drift |
Practice Zone — PYQs from real selection rounds
Six MCQs and two tasks — build a filtered index in fifteen lines, then size one for two million chunks.
Besides the vectors themselves, what must a vector database store for RAG to work?
Asked in
HNSW, the most common ANN index, works by:
Asked in
When would pgvector (Postgres) be a better choice than a dedicated vector database?
Asked in
FAISS differs from Pinecone mainly in that:
Asked in
Your index must serve documents with per-user access rights. What's the correct approach?
Asked in
A document is edited. What's the correct index-maintenance action?
Asked in
Hands-on tasks:
Using Chroma (in-memory), index four chunks with metadata and query them with a filter. Predict which chunk comes back for the query 'how do I claim medical bills?' restricted to department='HR'.
Asked in
You must index 500,000 documents averaging 4 chunks each, with 1536-dimensional float32 embeddings. Estimate raw vector storage, then say two things that make the real footprint larger.
Asked in
FAQ
Do I need a vector database, or is Postgres enough?
For many enterprise apps — millions, not billions, of chunks — pgvector is genuinely enough and saves you an entire system to operate, back up and secure. Dedicated stores earn their place at very large scale, with heavy filtering, or when you need features like built-in hybrid search and multi-tenancy.
What happens if the vector database goes down?
Design for it: retries with backoff, replicas, and a degraded mode (keyword-only search, or an honest "search is temporarily unavailable"). A RAG system whose only behaviour on failure is a stack trace isn't production software — lesson 10 covers this properly.
Can one index hold multiple embedding models?
Not meaningfully in one searchable space — vectors from different models are incomparable (lesson 2). Use separate collections per model, and when you upgrade a model, re-embed the corpus into a new collection and switch over once it's verified.
Next lesson: the decision that quietly determines your retrieval quality — Lesson 5: Chunking Strategies →