The Index Agent

Where RAG came from: one system, trained end to end

When Lewis and colleagues named retrieval-augmented generation in 2020, they trained the retriever and generator together on the task. Read the figure from left to right before looking at the arrow across the top.

The design lives in the arrow labelled "End-to-End Backprop through q and p_θ". One loss updated both halves at once. The query encoder learned which passages the generator could use, and the generator learned to lean on retrieved evidence over its own memory.

The retriever began as part of the model, shaped by the task it had to perform, well before anyone described it as a search tool placed in front of one.

Using pretrained models for RAG

An inference API does not provide access to update model weights. Jointly training a self-hosted retriever and generator requires the model weights, training data, and compute.

The workflow here uses pretrained models without updating their weights. A retriever selects passages to include in the generation prompt. You can change retrieval and inspect its results without training the generator.

Naming the context bundle before you pick a store

Before picking a retrieval primitive, you should first define a bundle. The bundle is the complete and bounded set of context that you intend your pipeline to pass around.

• For a simple single-turn question-answering system, the bundle can stay small: the user question, the few chunks retrieved to answer it, and the answer itself.
• For an autonomous agent, the bundle is the working memory it carries across many turns. That memory stores the agent's constraints, prior decisions, partial results, tool surfaces, and operating rules.
• For an arbitrary Index Agent, the bundle returned by the agent may only house the retrieved context chunks along with their chunk sources, relevance scores, and possibly a summary. A separate internal bundle may be the flat live-request-processing pipeline from above, or a ReAct-style loop that iterates, judges retrieval relevance, and retries until it has accumulated enough context.

The bundle principle is a planning discipline: define the bundle before you pick the database. The table below helps check whether your candidate primitive can deliver the bundle for your task.

Where MCP fits in retrieval systems

MCP gives a retrieval system one boundary for discovering and calling external context sources. The agent still needs a runtime that decides when to retrieve, validates the result, and composes it with the rest of the active task state. Other boundaries can carry other parts of the same system: skills package reusable capability instructions, A2A can route work to remote specialists, and UI protocols can stream the generated state back to an application.

The figure below separates those boundaries. Read it left to right to follow MCP context ingress and A2A task delivery. Then read it bottom to top to follow skill brokerage, runtime execution, and generated UX. Choose the boundary that matches the bundle and the system state it must carry.

Matching a retrieval primitive to the shape of your data

The table lists retrieval approaches to consider for different data shapes. Choose using representative questions, the evidence each answer needs, and the cost of retrieving it; evaluate whether the returned results cover that evidence.

The unstructured-prose example retrieves passages from a corpus and asks the model to answer from them. Check whether the answer is supported by those passages.

Every request walks the same four steps:

• Embed the query into a vector.
• Search the passage index for the top-K nearest passages.
• Concatenate those passages onto the prompt.
• Generate an answer with an instruction to use those passages.

Other retrievers can use keyword search or structured queries. Choose the retrieval method for your data, then inspect which evidence it supplies to the generator.

Searching small chunks but returning the parent document

A bare top-K Index Agent has a recall-precision trade-off no embedding model solves. You must choose between short chunks and long documents:

• Short chunks focus the embedding on a narrower passage, but may omit context needed to answer the question.
• Long documents include more surrounding context, but covering several topics in one embedding can dilute the match to a specific question.

One approach indexes at two levels. Small chunks are embedded for retrieval; a match selects the parent document to supply more surrounding context to a later generation step.

The exercise below builds this with three parent docs, each split into short sentence-chunks, and a query that returns the matching parent.

Wrapping the index as a tool the model decides to call

A retrieval index on its own only answers when something queries it. It becomes an Index Agent once you wrap it as a tool the model can choose to call, letting the model decide when retrieval is worth running. The cell below does it in five steps:

1. A corpus is embedded once and held in an in-memory index.
2. A retrieve(question, k) tool is declared with a JSON schema.
3. The agent receives the user's question. It chooses whether to call the tool.
4. If yes, cosine retrieval runs over the index; the top-K passages are appended as a tool message.
5. A second LLM call writes the final answer grounded in the retrieved passages.

You can retain the tool interface while replacing the in-memory cosine search with a server-side FAISS+BM25+rerank pipeline executor. Validate its results and failure behavior before using it in the loop.

How the other three shapes are built

You just built the unstructured-prose lane from corpus to answer. The other three are named here so you can recognise them in the wild and reach for the right primitive. Each one has a well-worn production pattern, sketched below.

• Hierarchical docs. A chunker that preserves the heading hierarchy, plus atomic table extraction, parent pointers, and a dual index for structural and semantic search. You index the small child chunks for recall and fetch the enclosing parent on a hit, so the model reads the matched clause together with the section it belongs to.
• Page-image retrieval for figures and charts. A dual-vector setup that embeds the rendered page image alongside its text, so a query can also pull a figure or a slide when the answer lives there.
• Structured / governed tables. Not built in this course. The standard production pattern is a semantic layer sitting between the agent and the row-level data, exposing only the curated metrics the bundle requires.
• Relational / graph data. The agent calls a graph-traversal tool (Cypher, GraphQL, or custom) in place of a vector store. The bundle it gets back is a sub-graph whose explicit edges carry into the prompt, so the relationships between entities reach the model intact.

When top-K is the wrong tool: GraphRAG and global questions

Every retriever on this page answers a local question whose answer sits in a handful of passages that Top-K similarity can find for the model to read, the way asking what the contract says about termination or which paper introduced HyDE does. A whole different class of question has no answer in any single passage at all. Ask what the main themes are across a thousand documents and you find that the answer is too diffuse to live in any one passage and emerges only from the corpus taken as a whole.

With K set to 3, top-K returns three matches from the thousand-passage example. Generation receives those selected passages; they may not cover themes elsewhere in the corpus. GraphRAG (Edge et al. 2024) changes the shape of that problem instead of raising K. It spends an LLM up front at indexing time to:

• read the corpus and extract entities and the relationships between them into a knowledge graph;
• detect the communities in that graph, the clusters of densely connected entities;
• summarize each community on its own.

A global question is then answered at query time, by summarizing across the relevant community summaries and reducing those partial answers into one. The work is a map-reduce over structure rather than a lookup over similarity.

The cell below makes the difference concrete on a small corpus, and it compares top-K retrieval with the community-summary stage used in GraphRAG. The groups below are authored labels, not a graph extracted by the model. The summary route receives every group and its name; top-K receives only its selected passages. Inspect which claims each answer supports, then change K, the groups, or the question.

Small top-K selections can omit evidence needed for corpus-wide questions. Community summaries can broaden coverage, but their grouping and summaries also need evaluation. This example runs summarization each time; a larger system can build and store summaries ahead of queries.