The Index Agent

Where RAG came from: one system, trained end to end

The version of RAG everyone runs today drops most of what the original design did, and knowing that reframes the retriever you are about to wire in.

When Lewis and colleagues named retrieval-augmented generation in 2020, it was not a model with a search box bolted on. The retriever and the generator were trained together on the task they had to perform. Read the figure from left to right before looking at the arrow across the top.

The design lives in the arrow labelled "End-to-End Backprop through q and p_θ". One loss updated both halves at once. The query encoder learned which passages the generator could use, and the generator learned to lean on retrieved evidence over its own memory.

The retriever began as part of the model, shaped by the task it had to perform, well before anyone described it as a search tool placed in front of one.

Why today's RAG drops the joint training

Almost nobody trains RAG that way now for two practical reasons. You cannot backpropagate into a frontier model you reach through an API, and even self-hosted end-to-end retrieval training is expensive and easy to destabilize.

Practice settled on a simpler assembly. You take a model that already exists, freeze it, and put a retriever in front of it so the prompt arrives pre-loaded with evidence and nothing is trained at all. The parts were built separately and bolted together, and over time "RAG" came to mean that bolt-on rather than the jointly optimized system the paper described.

Naming the context bundle before you pick a store

Before picking a retrieval primitive, you should first define a bundle. The bundle is the complete and bounded set of context that you intend your pipeline to pass around.

• For a simple single-turn question-answering system, the bundle can stay small: the user question, the few chunks retrieved to answer it, and the answer itself.
• For an autonomous agent, the bundle is the working memory it carries across many turns. That memory stores the agent's constraints, prior decisions, partial results, tool surfaces, and operating rules.
• For an arbitrary Index Agent, the bundle returned by the agent may only house the retrieved context chunks along with their chunk sources, relevance scores, and possibly a summary. A separate internal bundle may be the flat live-request-processing pipeline from above, or a ReAct-style loop that iterates, judges retrieval relevance, and retries until it has accumulated enough context.

The bundle principle is a planning discipline: define the bundle before you pick the database. Picking a vector store first and hoping it covers everything is the origin of most "worked in demo but failed in production" stories. The four-shape taxonomy below checks whether your candidate primitive can deliver the bundle for your task.

Where MCP fits in retrieval systems

MCP gives a retrieval system one boundary for discovering and calling external context sources. The agent still needs a runtime that decides when to retrieve, validates the result, and composes it with the rest of the active task state. Other boundaries can carry other parts of the same system: skills package reusable capability instructions, A2A can route work to remote specialists, and UI protocols can stream the generated state back to an application.

The figure below separates those boundaries. Read it left to right to follow MCP context ingress and A2A task delivery. Then read it bottom to top to follow skill brokerage, runtime execution, and generated UX. Choose the boundary that matches the bundle and the system state it must carry.

Matching a retrieval primitive to the shape of your data

Each data shape calls for its own retrieval primitive. Pick the wrong lane and retrieval keeps returning results that look fine while quietly missing what the query wanted, with no error to warn you. The table below tells you which primitive each shape calls for.

The unstructured-prose example carries a corpus through retrieval and generation to a grounded answer.

Every request walks the same four steps:

• Embed the query into a vector.
• Search the passage index for the top-K nearest passages.
• Concatenate those passages onto the prompt.
• Generate an answer grounded in them.

Index agents that manage other primitives largely take the same shape while varying the search/embed/verification logic, so don't feel intimidated to experiment with shapes that better fit your data.

Searching small chunks but returning the parent document

A bare top-K Index Agent has a recall-precision trade-off no embedding model solves. You must choose between short chunks and long documents:

• Short chunks embed precisely (the vector represents a focused idea) but lack context (one sentence is rarely enough to answer with).
• Long documents have context (every fact lands in its setting) but embed fuzzily (the vector is the average of many ideas).

The production pattern indexes at two levels. You embed the small chunks so retrieval stays precise, and when one matches you return its parent document so the model still sees the full context around it.

The exercise below builds this with three parent docs, each split into short sentence-chunks, and a query that returns the matching parent.

Wrapping the index as a tool the model decides to call

A retrieval index on its own only answers when something queries it. It becomes an Index Agent once you wrap it as a tool the model can choose to call, letting the model decide when retrieval is worth running. The cell below does it in five steps:

1. A corpus is embedded once and held in an in-memory index.
2. A retrieve(question, k) tool is declared with a JSON schema.
3. The agent receives the user's question. It chooses whether to call the tool.
4. If yes, cosine retrieval runs over the index; the top-K passages are appended as a tool message.
5. A second LLM call writes the final answer grounded in the retrieved passages.

The "retriever-as-tool" shape you just wired scales straight to production. Swapping the in-memory cosine function for a server-side FAISS+BM25+rerank pipeline changes only the executor; the tool schema, the agent loop, and the final answer step all stay identical.

How the other three shapes are built

You just built the unstructured-prose lane from corpus to answer. The other three are named here so you can recognise them in the wild and reach for the right primitive. Each one has a well-worn production pattern, sketched below.

• Hierarchical docs. A chunker that preserves the heading hierarchy, plus atomic table extraction, parent pointers, and a dual index for structural and semantic search. You index the small child chunks for recall and fetch the enclosing parent on a hit, so the model reads the matched clause together with the section it belongs to.
• Page-image retrieval for figures and charts. A dual-vector setup that embeds the rendered page image alongside its text, so a query can also pull a figure or a slide when the answer lives there.
• Structured / governed tables. Not built in this course. The standard production pattern is a semantic layer sitting between the agent and the row-level data, exposing only the curated metrics the bundle requires.
• Relational / graph data. The agent calls a graph-traversal tool (Cypher, GraphQL, or custom) in place of a vector store. The bundle it gets back is a sub-graph whose explicit edges carry into the prompt, so the relationships between entities reach the model intact.

When top-K is the wrong tool: GraphRAG and global questions

Every retriever on this page answers a local question whose answer sits in a handful of passages that Top-K similarity can find for the model to read, the way asking what the contract says about termination or which paper introduced HyDE does. A whole different class of question has no answer in any single passage at all. Ask what the main themes are across a thousand documents and you find that the answer is too diffuse to live in any one passage and emerges only from the corpus taken as a whole.

Top-K hands back only its three best matches, so the model summarizes those three chunks while staying completely blind to the other nine hundred and ninety-seven passages in the corpus. GraphRAG (Edge et al. 2024) changes the shape of that problem instead of raising K. It spends an LLM up front at indexing time to:

• read the corpus and extract entities and the relationships between them into a knowledge graph;
• detect the communities in that graph, the clusters of densely connected entities;
• summarize each community on its own.

A global question is then answered at query time, by summarizing across the relevant community summaries and reducing those partial answers into one. The work is a map-reduce over structure rather than a lookup over similarity.

The cell below makes the difference concrete on a small corpus, and it runs the real pipeline rather than playing back a script. It asks one global question two ways: plain top-K retrieval, and GraphRAG's map-reduce over named communities. Watch top-K answer from whichever chunks happen to match the question's words while the community pass recovers the corpus-wide theme. Edit the corpus, the communities, or the question and run it again.

A frozen model with a top-K retriever in front answers local questions well and global ones not at all. Restructuring the corpus into summarized communities buys the global answer, paid for by an expensive indexing pass you run once.