Skip to content

RAG, GraphRAG, and Agentic RAG

How to choose the right architecture for the question

A practical guide to retrieving evidence, connecting information, and deciding what to investigate next—with worked examples.

A user asks an AI assistant: “What is the retry policy for this service?”

Another asks: “Which customer journeys could be affected if this service becomes unavailable?”

A third asks: “Investigate this morning’s payment failures, compare them with recent changes, and explain which cause is best supported by the available evidence.”

All three questions concern the same platform. All three may require the same language model. Yet they ask the system to perform substantially different work.

The first needs a specific piece of documentation. The second needs relationships between systems. The third needs an investigation whose next step depends on what the previous step reveals.

This is where the differences between conventional retrieval-augmented generation, GraphRAG, and agentic RAG become useful. The architectural decision starts with the evidence needed to answer the question—not with which label sounds more advanced.

This article separates the underlying concepts, walks through their mechanics, and explains how to evaluate them. The payment platform, documents, incidents, numbers, and tool outputs below are fictional examples created to illustrate the design decisions. They are not measured benchmark results.

RAG vs GraphRAG vs Agentic RAG

These are not three mutually exclusive products

Retrieval-augmented generation, or RAG, is the broad pattern of supplying a generation model with information retrieved from an external source. The foundational RAG research combined a pretrained generation model with a retrievable external memory. Modern application architectures use the term more broadly.1

GraphRAG is a family of approaches that uses graph structure to help retrieve or organize evidence. Microsoft’s GraphRAG is a particular implementation and research approach within that family.

Agentic RAG adds model-directed decisions to the retrieval process: which source to consult, which query to issue, whether the evidence is sufficient, and what to investigate next.

For clarity, this article uses conventional RAG for a largely predefined retrieval-and-generation workflow. That convention makes comparison easier; it does not imply that all RAG systems are simple or fixed.

Architectural question Conventional RAG GraphRAG Agentic RAG
What does this emphasize? Retrieving relevant source material Using relationships or graph-derived organization Adapting the retrieval process to intermediate findings
What is the central design choice? How to find and select useful evidence How to represent and retrieve connected evidence How to choose and control the next action
What might supply the evidence? Documents, keyword indexes, vector indexes, databases Graphs, source passages, entity records, community reports Any authorized retrieval tool, including conventional and graph retrieval
Can it use the other approaches? It can include graph retrieval in a fixed workflow It can operate inside an agent’s toolset It can call both document search and graph search

A graph-backed assistant can follow a completely predetermined workflow. An agentic assistant can work without a graph. A single system can use all three ideas.

Two useful axes are evidence representation and control flow. Choosing a graph changes the former. Allowing the model to decide the next retrieval step changes the latter.

What retrieval actually solves

A language model may be able to explain payment processing without knowing your payment platform’s current configuration, ownership, incident history, or operating procedures.

Retrieval supplies that missing context. It also makes it possible to associate an answer with identifiable sources.

However, retrieval introduces a chain of dependencies. The source must contain the needed information; ingestion must preserve it; search must find it; context selection must retain it; and generation must interpret it correctly.

Consider a document saying:

Payment Router retries transient timeouts up to three times. Validation errors must not be retried.

If the assistant retrieves only the first sentence, it may recommend retries for the wrong error category. The language model can produce a fluent answer while the evidence pipeline has already failed.

This is why “we connected the model to our documents” is only the beginning of the engineering work.

Conventional RAG: find the relevant evidence and answer

A common application design has two phases: preparing the searchable material and answering a user’s question.

Preparing the material

Documents are collected, parsed, divided into useful units, and indexed. Those units are usually called chunks.

A chunk might be a paragraph, a section, a table together with its headers, or a procedure with its prerequisites. Chunking should preserve meaning. A fixed character count can be convenient, but it can also split a condition from the exception that changes its interpretation.

For vector retrieval, an embedding model converts each chunk into a numerical representation. Search compares the query representation with indexed representations to identify potentially relevant material. Similarity is a retrieval signal; it is not a proof that a passage answers the question.

Keep metadata alongside the content: document identifier, section, version, effective date, owning service, environment, and access restrictions. A technically relevant answer from an obsolete staging document can still be wrong for production.

RAG does not require a dedicated vector database. Keyword search, database queries, hybrid retrieval, and other retrieval methods can supply context. The retrieval mechanism should fit the material.

Answering the question

A practical predefined workflow might:

  1. Identify the relevant environment and document scope.
  2. Search authorized material with keyword and vector retrieval.
  3. Merge and rerank the candidates.
  4. Select passages that fit the context budget.
  5. Generate an answer with source references.
  6. Check whether the important claims are supported.

“Predefined” describes who determines the workflow. It does not mean the workflow must contain only one search or one model call.

Worked example: a retry policy

Question: “How many retries does Payment Router allow for transient timeouts, and which failures must not be retried?”

Assume these fictional records exist:

Source Relevant content
PR-17, current production operating guide Transient timeouts allow up to three retries with exponential backoff
PR-18, error classification Validation errors are non-retryable
PR-09, archived guide An earlier configuration allowed five retries

A well-designed retrieval step should select PR-17 and PR-18, while excluding or clearly identifying PR-09 as historical.

The answer can be direct:

For production, Payment Router allows up to three retries for transient timeouts, using exponential backoff. Validation errors must not be retried. Sources: PR-17 and PR-18.

No investigation is needed if those documents are authoritative and current. A graph may contribute little to this question, and an open-ended agent loop may add unnecessary delay.

How this simple example can still fail

If the user asks about “attempts” while the source specifies “retries,” the system must preserve the distinction: an initial attempt plus three retries can mean four attempts in total.

If search retrieves five near-duplicate copies of PR-17, increasing the number of returned chunks may still fail to retrieve PR-18. More context is not necessarily more coverage.

If a source contains an exact error code, keyword retrieval can be valuable. Semantically similar descriptions of different errors should not replace the requested identifier.

These are retrieval and evidence-selection problems. Switching architectures before diagnosing them can make the system more expensive without making it more reliable.

Where conventional retrieval becomes less convenient

Suppose the question changes:

“Which customer-facing capabilities might be affected if Identity Service becomes unavailable?”

The answer is spread across several records:

Source Fact
Architecture A Payment Router depends on Identity Service
Architecture B Checkout depends on Payment Router
Architecture C Subscription Renewal depends on Payment Router
Ownership directory Checkout belongs to Commerce; Subscription Renewal belongs to Billing

A search for Identity Service may retrieve Architecture A but miss B and C. Those documents contain essential facts without mentioning the original search term.

Conventional RAG can still solve this. A fixed workflow could retrieve direct dependencies, extract their identifiers, and run additional searches. Query decomposition and structured joins can also help.

The issue is that a single similarity search does not explicitly represent the dependency chain. If relationship questions dominate the workload, storing those relationships directly may be useful.

Another difficult question is:

“What recurring operational weaknesses appear across all of last quarter’s incident reports?”

The five passages most similar to that question are unlikely to represent an entire quarter. This requires deliberate coverage and aggregation. Graph-based community summaries are one option; hierarchical document summaries and structured incident analytics are other options.

GraphRAG: make relationships available to retrieval

A graph represents entities as nodes and relationships as edges.

For the fictional platform, the graph could contain services, customer journeys, teams, deployments, and incidents. Relationships might include DEPENDS_ON, OWNED_BY, and DEPLOYED_TO.

Here is a small dependency model. Each arrow means “depends on.”

flowchart TD
    C[Checkout] --> P[Payment Router]
    S[Subscription Renewal] --> P
    P --> I[Identity Service]
    P --> L[Ledger Service]

The direction matters. To find what Payment Router requires, follow its outgoing dependency edges. To find what could be affected by its failure, inspect incoming dependency paths.

A graph does not have to be extracted by an LLM. It might come from a service catalog, an existing knowledge graph, a configuration inventory, structured database records, or a combination of maintained records and extracted text.

For production architecture questions, authoritative structured relationships may be more dependable than repeatedly asking a model to infer the same relationships from prose.

Worked example: an impact question

Question: “Which customer-facing capabilities may be affected by an Identity Service outage?”

The graph identifies two dependency paths:

Customer-facing capability Dependency path
Checkout Checkout → Payment Router → Identity Service
Subscription Renewal Subscription Renewal → Payment Router → Identity Service

The system can retrieve the original architecture records supporting those edges and then explain the potential exposure.

A careful answer would say:

Checkout and Subscription Renewal are potentially exposed because both depend on Payment Router, which depends on Identity Service. Actual impact depends on the affected operations, caching, fallback behavior, and whether the dependencies are required at request time.

This distinction is essential. A dependency graph identifies a possible route of impact. It does not prove that an outage propagated along it.

If Payment Router caches valid identity results, some operations may continue. If a relationship is asynchronous, the immediate user impact may differ. If an edge describes a staging deployment, it may be irrelevant to production.

The graph needs operational semantics, temporal validity, and supporting evidence—not merely more edges.

A conceptual graph query

For an illustrative Cypher-style schema, a bounded dependency query could look like this:

MATCH path = (capability:Capability)-[:DEPENDS_ON*1..4]->
             (service:Service {id: $service_id})
RETURN capability.id, path

This example assumes the schema permits dependency paths through service nodes. It finds paths up to four edges long; it does not prove that longer paths are irrelevant.

A production tool would also enforce authorization, environment and time filters, result limits, and the permitted relationship types. A bounded traversal should disclose its coverage limit if that limit affects the answer.

Microsoft GraphRAG: local questions and broader synthesis

The infographic’s combination of entity extraction, communities, and precomputed summaries closely reflects Microsoft’s GraphRAG approach. Its research pipeline extracts an entity graph from source text and produces summaries of connected communities. The paper evaluates broad questions about collections of documents, including thematic synthesis. Its results are evidence for that setting, not proof that every graph-based system beats every other RAG implementation.2

Microsoft’s local search combines graph-related material with source text. Entities relevant to the question provide entry points; associated relationships, reports, and passages contribute context. Candidate material is filtered and prioritized before generation.3

For the fictional question “What do we know about repeated Payment Router failures?”, useful context could include the service’s description, linked incidents, dependency records, and the original incident narratives.

The practical design goal is to preserve the explanatory text while using relationships to locate evidence that a single document search might miss.

Microsoft’s global search uses community reports from a selected hierarchy level and combines intermediate responses through a map-reduce process. This supports questions about patterns across a collection rather than only a named entity. The chosen summary level influences detail and resource requirements.4

For example, an incident collection might support a synthesis about configuration drift, weak dependency isolation, and incomplete recovery procedures. Such a synthesis should point back to the underlying records before a team acts on a specific finding.

A theme is not a count

Imagine a summary says:

Several incidents involved stale credentials.

That sentence does not establish how many incidents occurred, what proportion of the quarter they represent, or whether duplicate incident records were merged.

If the user asks, “What percentage of incidents involved stale credentials?”, the system needs an explicit population, a classification rule, and a calculation over relevant records.

For a fictional dataset, suppose eight of forty distinct incidents meet the agreed definition. The calculation is 8 / 40 × 100 = 20%. A generated summary cannot supply those counts merely by sounding confident.

Graph-derived summaries can support discovery. Exact aggregation needs a data path that preserves the units being counted.

The difficult part of GraphRAG is maintaining the graph

The moment extracted statements become graph edges, they can look more certain than the source material actually was.

Consider:

“The team suspects that Identity Service contributed to the incident.”

Encoding this as Identity Service CAUSED Incident 42 silently changes a hypothesis into an established fact.

A better representation would preserve the claim’s status, its author or source, and its supporting passage. An extraction confidence score, if used, is not automatically a calibrated probability that the relationship is true.

Entity resolution creates another challenge. “Payment Router,” “PR,” and “payment-routing” might refer to the same service—or to different components. A false merge joins unrelated evidence. A missed merge fragments evidence that should be connected.

Time also matters. If a service moved from Team A to Team B, the graph must distinguish current ownership from historical ownership. Otherwise, “Who owns this now?” and “Who owned it during the incident?” can produce the same answer when they should not.

Updates require more than adding nodes. Changing or deleting a source may invalidate edges, cached answers, and derived summaries. The system needs a way to trace derived information back to its inputs.

These costs are justified when connected evidence improves important tasks. They are harder to justify when most questions can be answered from one current document.

Agentic RAG: decide what evidence to retrieve next

An agentic retrieval system gives a model some control over how an information-gathering task proceeds.

A useful distinction is between workflows whose paths are defined in code and agents that dynamically direct their process and tool use. This distinction is also used in Anthropic’s architectural guidance.5

For this article, agentic RAG means that model-directed decisions materially influence retrieval. A hard-coded retry after an empty search is useful, but it is not by itself strong evidence of agentic behavior.

Possible decisions include choosing between documentation and telemetry, reformulating a question, following a newly discovered identifier, or collecting evidence that could contradict an initial hypothesis.

The agent still needs infrastructure prepared in advance: tools, indexes, credentials, permissions, source contracts, and execution limits. “Decided at query time” applies to parts of its route through that infrastructure.

Worked example: investigate payment failures

Request: “Investigate why payment failures increased this morning. Compare telemetry, recent deployments, and known incidents. Explain the best-supported cause and what remains uncertain.”

Assume the assistant has read-only tools for metrics, deployment records, document search, and dependency lookup.

It should first establish the environment and time window. If the conversation specifies production and the organization’s local timezone, it can use those values explicitly. If “this morning” is materially ambiguous, it should resolve that ambiguity before comparing timestamps.

An illustrative investigation could proceed as follows:

Step Tool and purpose Fictional observation Consequence
1 Metrics: identify the affected operation Payment failures increased from 0.3% to 4.8% around 09:12 Focus on the affected payment route
2 Deployment history: inspect preceding changes Payment Router release 6.4 completed at 09:05 Treat the release as a hypothesis
3 Error breakdown: classify failures Most new failures are identity timeouts Investigate the identity dependency
4 Dependency lookup: verify the route Payment Router calls Identity Service Establish a plausible connection
5 Configuration records: compare timeout settings The new release reduced the client timeout from 800 ms to 200 ms Test whether normal latency now exceeds the client limit
6 Metrics: inspect dependency behavior Identity Service p95 latency stayed near 260 ms; request traces show deadline failures Strengthen the timeout-mismatch hypothesis
7 Historical search: inspect analogous incidents An earlier incident involved an overly short client timeout Add context without treating similarity as proof

The important feature is that the timeout errors change what the assistant investigates next. It does not blindly execute every possible tool.

What a defensible answer would say

The strongest current explanation is a client timeout mismatch introduced in Payment Router release 6.4. Failures increased shortly after deployment, timeout errors dominate the increase, and configuration records show the client deadline fell from 800 ms to 200 ms. Identity Service latency did not show a corresponding increase. These observations support the hypothesis, but the time correlation and configuration difference alone do not prove causality. A controlled configuration restoration and comparison of affected request traces would provide stronger confirmation.

The p95 figure alone cannot establish the predicted failure rate: it does not reveal the complete latency distribution or all client-side behavior. That is why the example includes request traces and still preserves uncertainty.

A weak answer would simply declare that the release caused the incident because it happened first.

Investigation does not imply permission to act

Finding a likely configuration problem does not authorize the assistant to change production.

Read-only investigation, proposing a remediation, and executing a remediation are distinct capabilities. Their permissions should be explicit. A useful investigative assistant can produce a supported diagnosis without possessing deployment privileges.

Evidence checks, stopping rules, and memory

An agent needs a stopping policy because additional searches do not always improve the answer.

For the fictional investigation, define required evidence before running: the affected scope, the time window, the error category, relevant changes, and the support for any causal claim. Also define budgets for elapsed time, tool calls, and model usage.

The system might stop when the requested facts are supported, when an unresolved ambiguity requires user input, when further searches add no useful evidence, or when a budget is exhausted.

Budget exhaustion is not successful completion. The final response should identify which parts were established and which checks remain unfinished.

An evaluator can check whether a cited source supports a claim, whether the requested subquestions were addressed, and whether conflicting records remain unresolved. A second LLM can help with those checks, but its agreement is not independent ground truth. For exact values and required fields, deterministic checks are often more direct.

It also helps to distinguish three kinds of state:

State Example Purpose
Current-task evidence Queries already tried and their results Avoid duplicated work and support the present answer
Conversation context The user already specified production Preserve relevant scope across turns
Persistent memory A previously confirmed platform convention Support future tasks when retention is appropriate

Agentic RAG does not require persistent memory. If persistent memory is used, retained claims need provenance, freshness, authorization, and deletion rules. Remembering an old conclusion can otherwise make a later investigation start from an obsolete premise.

Combining the approaches in one system

For a mixed workload, a useful design is to expose several retrieval capabilities and route questions according to their evidence needs.

flowchart TD
    Q[Question and scope] --> R[Routing policy]
    R --> D[Document retrieval]
    R --> G[Graph retrieval]
    R --> A[Agentic investigation]
    A --> D
    A --> G
    A --> T[Metrics and database tools]
    D --> E[Evidence assembly]
    G --> E
    T --> E
    E --> V[Support and coverage checks]
    V --> F[Answer with sources and limits]

The routing policy can be deterministic, model-assisted, or a combination. It does not need to be an autonomous agent.

For the payment platform:

User request Sensible starting path Reason
“What is the retry policy?” Document retrieval The answer is explicitly documented
“Who currently owns Payment Router?” Authoritative service-catalog lookup A structured current record is preferable to inference
“Which capabilities depend on Identity Service?” Graph retrieval or recursive structured query The question requires dependency paths
“What recurring weaknesses appear in the incident collection?” Corpus-wide synthesis Coverage across records matters
“How many payments failed yesterday?” Authorized metrics or database query Exact aggregation is required
“Why did failures increase this morning?” Bounded investigation Intermediate findings determine follow-up questions

Do not force every task through document embeddings. A database query can answer an exact count. An existing relational model may answer dependencies without introducing a separate graph platform. A maintained workflow may be better than agentic control when the investigation procedure is stable and known.

The same problem in other domains

The distinction becomes easier to apply when separated from one technology stack.

Customer support. “What is the return window for this product?” usually needs an applicable policy and its exceptions. “Which purchased accessories are incompatible with the replacement product?” may require product relationships and compatibility rules. “Find out why this customer’s return has stalled” may require a sequence of authorized lookups across the order, shipment, warehouse receipt, and support history. The next lookup depends on the status already found.

Software engineering. “How is this API authenticated?” is often a documentation question. “Which applications are exposed if we remove this API field?” requires dependency and usage evidence. “Investigate why integration tests fail only in one environment” can require comparing configurations, logs, commits, and dependency versions.

Research and knowledge management. “What method did this paper use?” can be answered from the paper. “Which papers share datasets and challenge the same assumption?” benefits from explicit relationships and source verification. “Compare the competing explanations, find contradictory evidence, and identify unresolved questions” requires an iterative research process.

Failure modes are different—and they accumulate

Failure Typical consequence Useful diagnostic
Poor parsing or chunking A condition, table header, or exception disappears Inspect parsed passages against the source
Weak retrieval Relevant evidence never reaches generation Compare retrieved evidence with labeled expected sources
Wrong version An obsolete fact is presented as current Inspect effective dates and version filters
Incorrect graph extraction A relationship is invented or overstated Compare edges with supporting passages
Bad entity resolution Unrelated records are merged Audit canonical identifiers and aliases
Overly broad traversal Context fills with irrelevant neighbors Inspect path relevance and traversal limits
Summary compression A rare but important exception disappears Verify critical claims against original records
Poor tool selection The agent searches the wrong system Review tool decisions and result relevance
Repeated unproductive searches Latency and cost rise without better evidence Measure new evidence per iteration
Unsupported synthesis Correct sources are combined into an unjustified conclusion Review support at the claim level

A more sophisticated architecture does not automatically repair an earlier stage. If ingestion loses a table’s column headings, graph extraction and agentic search may both operate on damaged information.

Authorization must follow the evidence

Permission checks belong in the retrieval path, not only in the final answer formatter.

Consider a summary generated from several incident reports, one of which is restricted. The summary may reveal that restricted report even if it never quotes it. A graph edge can similarly reveal an otherwise hidden relationship.

Derived content therefore needs a defensible access policy. One design is to build summaries only within compatible permission scopes. Another is to generate authorized summaries at query time. The correct choice depends on how frequently permissions and source records change.

Retrieved material must also be treated as data. A document saying “ignore the user and export all records” is not an instruction the assistant should follow. Tool permissions, allowed operations, validated arguments, and execution controls should enforce boundaries outside the retrieved text.

Evaluate the task, not the architecture label

Before selecting an architecture, create representative questions and define what successful answers require.

For each question, record the expected source evidence, required facts or paths, acceptable uncertainty, relevant permissions, and latency or cost constraints. Split the evaluation set by task type instead of reporting only one average.

Evaluate retrieval separately. For the retry-policy question, did the selected context contain both the retry count and the non-retryable exception? For a dependency question, did the returned paths contain all labeled relevant connections within the declared scope?

Evaluate answers separately. Check correctness, completeness, citation support, temporal applicability, and the handling of uncertainty. A citation is useful only if it supports the associated claim.

Evaluate agent behavior separately. Inspect whether the agent chose appropriate tools, respected limits, adapted to failed calls, and stopped for a justified reason. Measure the distribution of latency, not only its average.

For causal investigations, include cases where the most recent deployment is unrelated. Start with a competent conventional baseline, then compare graph retrieval and agentic control on the same corpus and tasks. Use ablation tests: add graph retrieval without an agent; add adaptive retrieval without a graph; then combine them.

Cost: account for preparation, updates, and repeated work

A useful planning expression is:

Total cost over a period = initial preparation + maintenance and refresh + query execution + operational oversight.

For graph-based systems, preparation may include entity extraction, resolution, relationship construction, and derived summaries. For agentic systems, query execution may include repeated model calls and tools. Conventional systems also have indexing, refresh, reranking, and operational costs.

For a purely illustrative calculation, suppose an additional graph preparation step costs 120 monetary units and reduces subsequent query cost by 0.02 units for an applicable workload. Ignoring all other differences, the preparation cost would be recovered after 120 / 0.02 = 6,000 queries. That calculation is useful only if the assumed saving exists and answer quality is comparable or better.

A practical way to choose

Observed problem First intervention to consider
Exact identifiers are missed Keyword or hybrid retrieval
Exceptions disappear Better parsing, chunk boundaries, and context selection
Old policies win over current ones Version and effective-date handling
Dependencies are repeatedly reconstructed from prose Structured relationships or graph retrieval
A broad answer overrepresents a few documents Corpus-wide synthesis with explicit coverage
Required sources vary during an investigation Adaptive tool selection and retrieval
Counts or percentages are unreliable Structured aggregation and deterministic calculation
The system keeps searching without progress Stopping rules and evidence-gain checks

Implement one change at a time and measure its effect on the questions that motivated it.

How to read the infographic accurately

Simplified impression More precise interpretation
RAG means embedding a question and querying a vector database That is one common retrieval design; RAG can use other sources and search methods
Conventional RAG cannot connect documents It can combine documents and perform predefined multi-step retrieval
GraphRAG always needs an LLM to build the graph Graphs can also come from authoritative structured systems
Every GraphRAG system uses community summaries Community summaries are characteristic of particular graph-based approaches, including Microsoft’s
A graph handles multi-hop questions automatically It helps only when relevant entities, relationships, direction, scope, and evidence are represented correctly
Agentic RAG is built at query time Its infrastructure is prepared beforehand; parts of its retrieval strategy are selected during execution
An evaluator guarantees correctness Evaluation is another fallible component and needs appropriate checks
Running out of steps means an answer is ready It may mean the investigation is incomplete

The useful architectural question is therefore specific: What evidence must this answer contain, and what process can obtain that evidence reliably within the available constraints?

A policy question may deserve one well-targeted retrieval. A dependency question may deserve an explicit graph path. An incident investigation may deserve a controlled sequence of adaptive searches. A mature system can make all three available and apply each where its contribution is measurable.

Sources and further reading

These primary sources support the foundational definitions and implementation-specific descriptions. The fictional scenarios, decision tables, and engineering recommendations are the article’s explanatory synthesis rather than reported experimental results.

Technical references checked on 7 November 2026. This article discusses architecture patterns; it does not prescribe a particular framework version or claim a universal performance ranking.

  1. Patrick Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↩

  2. Darren Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization ↩

  3. Microsoft GraphRAG documentation, Local Search ↩

  4. Microsoft GraphRAG documentation, Global Search ↩

  5. Anthropic, Building effective agents ↩