Agents + RAG + MCP: How the Modern AI Stack Fits Together

You now know four things that keep showing up in the same job descriptions: LLMs, RAG, agents and MCP. Candidates who list them sound like they read a newsletter. Candidates who can say which layer owns which problem sound like they've built something. This lesson is that map — and the debugging instinct that comes with it, because in a real system the first useful question is always "which layer is failing?"

Five concerns, five homes

Five concerns, five homes — and almost every production problem belongs to exactly one of them. Memorize this table; it turns vague debugging into a two-minute triage.

RAG stops being a pipeline and becomes a tool

In the RAG course, retrieval was a fixed stage: always once, always before generation. Give it to an agent and it becomes a tool the model may call zero, one or several times:

@tool
def search_policies(query: str) -> str:
    """Search company policy documents. You MUST call this for any question
    about company rules, benefits or procedures - never answer such
    questions from your own knowledge. Returns passages with sources."""
    return format_with_sources(rag.retrieve(query, k=4))

That unlocks multi-hop questions (search, read, search again with what you learned) and mixed questions ("compare our policy with this contract" — two retrievals from two sources). It also unlocks a new failure mode, which is the next section.

What MCP actually solves

Suppose your agent needs your ticketing system, your CRM, your document store and your CI. You write four integrations. Then another team builds a different assistant needing the same four — and writes them again. Five apps × ten systems = fifty bespoke integrations, each maintained separately.

MCP (Model Context Protocol) standardizes the connection: each system exposes one server, and any MCP-capable client can use it. Ten servers, any client — the "USB-C for AI tools" framing. MCP doesn't replace tool calling or frameworks; it replaces integration work. Same instinct one level up as A2A protocols for agent-to-agent tasks. Its own course covers the architecture, primitives and security properly (MCP course).

Putting it together

A realistic internal engineering assistant, layer by layer: Knowledge — RAG over docs, runbooks and past incidents, exposed as a search tool with citations. Capabilities — read-only tools (CI status, ticket search, log query) available freely; write tools constrained; a "propose config change" tool that emits a diff instead of applying it. Where systems already publish MCP servers, connect through MCP rather than writing custom clients. Control flow — a graph with a tools cycle, an approval interrupt before writes, a step limit and a cost budget. Behaviour — a system prompt requiring retrieval for internal questions, citations in answers, escalation on uncertainty. Safety — permission checks inside every tool, audit logging, tool output treated as untrusted data.

Notice how little of that is about the model. Choosing a model is one decision; the other twenty are architecture — which is exactly what design rounds test.

Wait — my agent has a retrieval tool and still answers from memory

A perfect example of layer triage. The retrieval works, the tool exists — so this isn't a RAG problem or a capability problem. It's a behaviour problem: given a choice, the model skips retrieval for questions it feels confident about, and confidence about general topics is not knowledge of yourdata.

The fix lives in the prompt layer — an explicit rule ("for any question about company policy you MUST call search_policies"), reinforced in the tool description — plus an eval that asserts the call actually happened for policy questions. Fixing this by tuning embeddings or swapping frameworks would waste a week.

Run the same triage on other symptoms: answers cite outdated documents → knowledge layer (index freshness). Agent loops forever → control flow (step limit). It refunded the wrong customer's order → safety layer (missing ownership check in the tool), not "the model made a mistake".

The cost of the upgrade

Turning a fixed RAG pipeline into an agentic one is not free: more LLM calls per question, higher latency, less predictable behaviour, and harder testing. A common production shape is therefore hybrid — route simple questions through the cheap fixed pipeline, and only send genuinely open-ended ones to the agent.

That routing decision is worth mentioning in interviews, because it shows the instinct that runs through this entire course: upgrade the cases that need it, measured, rather than upgrading everything because the architecture sounds better.

Selection-round radar: "How do agents, RAG and MCP fit together?" is now a standard architecture question. Answer with the five-layer table, then one sentence each: RAG is a tool the agent chooses to call; MCP standardizes how tools are exposed; frameworks orchestrate; safety lives in tool code. Finish with the triage instinct — "when something breaks, I first ask which layer owns it."

Common mistakes

• Treating MCP as a framework or a replacement for tool calling.
• Assuming an agent will use its retrieval tool without being told to.
• Debugging the wrong layer — tuning embeddings for a prompt problem.
• Making everything agentic when most traffic is simple Q&A.
• Putting safety rules in prompts rather than tool code.
• Losing citations when RAG becomes a tool — pass sources through the tool result.

Quick recap

Practice Zone — PYQs from real selection rounds

Six MCQs and a full stack-design task for an internal engineering assistant.

Q1Agent stackmedium

In an agentic system, RAG is best understood as:

Asked in

aA replacement for the agent loopbOne tool among several — the agent decides whether and how often to call retrieval, rather than retrieval being a fixed pipeline stagecA type of memory inside the modeldA framework for orchestration

Q2Agent stackmedium

What problem does MCP solve in an agent stack?

Asked in

aIt makes models reason betterbN×M integration sprawl — instead of every AI app writing custom connectors for every tool/data source, both sides implement one protocolcIt stores embeddingsdIt replaces tool calling entirely

Q3Agent stackmedium

In a well-designed stack, where does each concern live?

Asked in

aAll in the promptbKnowledge → retrieval; capabilities → tools (often via MCP); control flow → the agent loop/graph; behaviour and rules → prompts; safety limits → tool code and approval gatescAll in the model's weightsdAll in the vector database

Q4Agent stackhard

An agent with a retrieval tool keeps answering from model memory instead of retrieving. What's the likely cause?

Asked in

aThe vector database is too fastbThe prompt doesn't require grounding, and the tool description doesn't make clear when retrieval is mandatory — so the model 'knows' enough to skip itcThe embedding dimension is too smalldToo many tools are registered

Q5Agent stackmedium

What's the cost of turning a fixed RAG pipeline into an agentic one?

Asked in

aNo cost — it's strictly betterbMore LLM calls, higher latency and cost, less predictable behaviour, and harder testing — justified only for question types that single-shot retrieval can't servecIt requires a different vector databasedIt disables citations

Q6Agent stackmedium

Why does an agent's system prompt usually need to be tighter than a chatbot's?

Asked in

aBecause agents use more tokensbBecause ambiguity turns into actions, not just words — vague rules produce wrong tool calls with real-world effectscBecause agents ignore system promptsdBecause prompts must fit in the tool schema

Hands-on task:

FAQ

Final lesson: turning all of this into a design-round answer — Lesson 12: Agent System Design →