How Graph RAG earns its keep

Graph RAG earns its cost when the answer depends on how facts connect, not merely on which passages sound like the question. If a user asks who approved a supplier, which incidents share a dependency, or what themes span an entire corpus, similarity alone can retrieve plausible fragments while missing the relation that makes them an answer.

That does not make a knowledge graph the new default retrieval layer. Vector search is cheaper to build, easier to refresh, and often better for direct factual lookup. I use a graph only after real queries expose a relation problem. Starting with a graph because the demo looks intelligent usually buys an extraction pipeline, an ontology debate, and a second database before it buys a better answer.

Graph RAG wins only when relations change the answer

The test is simple: remove the edges from your representation and ask whether the system can still justify the answer. If it can, ordinary vector retrieval probably does the job. If removing edges destroys identity, sequence, ownership, dependency, or scope, graph retrieval has something concrete to contribute.

Consider a support corpus with tickets, services, releases, incidents, customers, and engineers. The question "What did the team say about login timeouts?" asks for semantically relevant text. Embeddings can find it. The question "Which login incidents started after releases that changed the identity service and affected customers on the legacy plan?" asks for a constrained path. Relevant words may sit in five documents that never repeat the whole question. A graph can traverse Incident -> STARTED_AFTER -> Release -> CHANGED -> Service and Incident -> AFFECTED -> Customer -> ON_PLAN -> Plan, then return the source passages attached to those nodes and edges.

This distinction gets blurred because people use "Graph RAG" for several different systems. One system stores hand curated business entities and relations. Another asks a model to extract an entity graph from documents. A third clusters that extracted graph and writes summaries of each community. Those approaches have different trust and maintenance profiles. A relation copied from a product database can be authoritative. A relation inferred from prose is a claim that needs provenance and a confidence policy. A community summary is generated material, useful for discovery but not equivalent to the source.

Microsoft's GraphRAG paper, From Local to Global: A Graph RAG Approach to Query-Focused Summarization, targets a particular weakness of baseline RAG: questions about the corpus as a whole. Its index extracts entities and relationships, groups connected entities into hierarchical communities, and prepares reports for those communities. At query time, global search combines partial answers derived from those reports. That is a thoughtful answer to corpus level sensemaking. It is not evidence that every "who owns this account?" query needs community detection.

I classify a query as relational only when at least one relation constrains the valid result. Words such as "related," "similar," or "about" do not qualify by themselves. "Documents related to Acme" is semantic retrieval. "Contracts signed by Acme subsidiaries after the merger" requires legal identity, corporate ownership, event order, and a date constraint. The grammar of the question tells you more about the right index than the size of the document collection does.

There is another useful check: ask whether a database query could state the eligibility rule before any prose is generated. If the team cannot name the nodes, relations, and filters that make an answer valid, a graph will not create that missing business definition. Resolve the rule with domain owners first. Otherwise the ontology merely gives an ambiguous question a formal looking shape.

Four query shapes reveal the graph advantage

Graph retrieval pays for four recurring query shapes: constrained paths, multi document assembly, corpus wide aggregation, and explanations that must expose their joins. Each shape fails differently under pure similarity search.

Constrained paths and neighborhoods

A path query names or implies typed hops. Examples include "Which vendors depend on a library with an open critical issue?", "Who approved expenses for projects they also reviewed?", and "Which policies apply to contractors handling health data in Germany?" The answer comes from satisfying a pattern, not collecting the nearest paragraphs.

The useful artifact is an executable retrieval contract. A property graph team might express the first question like this:

MATCH (v:Vendor)-[:USES]->(p:Package)-[:HAS_ISSUE]->(i:Issue)
WHERE i.status = "open" AND i.severity = "critical"
MATCH (e:Evidence)-[:SUPPORTS]->(i)
RETURN v.id, p.name, i.id, e.source_id, e.span
LIMIT 50

The result should not be a prose answer yet. It should look like a set of bindings: vendor ID, package name, issue ID, source document ID, and source span. The generator receives those bindings plus the quoted spans. This separation lets you test retrieval without asking a language model to hide a bad join behind fluent text.

Neighborhood queries start from a known entity and fan out through selected relations. Microsoft's GraphRAG documentation calls its entity centered method local search and combines graph data with raw text chunks. That mix matters. Nodes and edges narrow the search; original passages supply detail and wording that extraction discarded.

Multi document assembly and corpus wide questions

Some questions need facts distributed across records even when the path is not obvious to the user. A postmortem names an outage, a deployment log names a release, a service catalog names an owner, and an organization directory names the current manager. Vector search may retrieve the postmortem twice in different chunks and omit the ownership chain. A graph can assemble the chain, provided each edge points back to evidence and carries a time range.

Corpus wide questions have the opposite geometry. "What concerns recur across all acquisition interviews?" has few terms that identify a particular chunk. Microsoft's global search organizes the dataset into communities, produces partial responses from community reports, rates them, and reduces them into a final response. The official documentation describes this method as resource intensive, which matches the design: it reads across many prepared summaries instead of retrieving a small top set.

These questions justify graph based community summaries when users ask them often enough to amortize indexing. If one analyst asks for a whole corpus synthesis once a quarter, a batch analysis may cost less and be easier to audit. Product teams routinely mistake a possible query for a frequent workload.

Explanations that must show the joins

Graph retrieval also helps when the answer must say why an item qualified. A recommendation such as "Escalate Supplier 42" is weak unless the system can return the path Supplier 42 -> provides Component A -> used by Service B -> breached recovery objective in Incident C. That path is both retrieval logic and an explanation skeleton.

Do not confuse a visible path with truth. A graph can produce a perfectly inspectable chain of wrong extractions. Inspectability helps an operator locate the error; provenance and validation determine whether the chain deserves trust.

Vector search remains the right default for flat evidence

Vector search wins when one or a few passages contain the answer and semantic similarity can find them. It usually handles policy lookup, product documentation, support answers, definitions, and "how do I" questions without the operational weight of a graph.

The popular recommendation I argue against is "build a knowledge graph first, then plug an LLM into it." It sounds disciplined because the graph promises structure. In practice, teams freeze a schema before they have query logs, then spend weeks extracting relations nobody asks about. A small vector baseline exposes the actual misses quickly. Those misses tell you which entities and edges deserve a schema.

Vector retrieval has several practical advantages. New chunks become searchable after embedding and insertion. It tolerates unfamiliar vocabulary because proximity is learned rather than tied entirely to exact labels. It preserves the source passage as the unit of evidence. Its failure modes are also familiar: near duplicate chunks crowd the result set, generic wording outranks a rare but decisive fact, and top k retrieval has no native notion of a complete path.

Keyword search should stay in the conversation too. Exact identifiers, error codes, invoice numbers, statute names, and quoted phrases often belong in a lexical index. Embeddings can blur strings that an operator expects to match exactly. A serious comparison is rarely graph versus vector. It is graph, vector, lexical, or a routed combination.

Run a cheap ablation before adding a graph. Take 50 to 100 real questions that the current system misses and label the smallest evidence set for each. If most gold evidence fits inside one chunk, fix chunking, metadata filters, reranking, or lexical recall. If the evidence repeatedly crosses documents through stable entity relations, graph work has a case. If users ask broad theme questions, test a corpus summary method separately because its economics differ from local traversal.

One warning from production: a graph can appear to improve answers simply because its indexer spent more model calls reading the corpus. Compare equal evidence budgets and include a strong vector baseline with reranking. Otherwise you are testing extra compute against a basic prototype and calling the data structure the winner.

A graph is a data product, not an index toggle

Graph upkeep usually costs more than query execution because every useful edge needs a definition, an owner, evidence, and a refresh rule. The database is the easy part. The hard part is keeping meanings stable while source systems and language change.

Start with entity identity. "Mercury" might be a project, a customer, a planet, or a code name. "ACME Ltd" and "Acme Holdings" might be aliases or different legal entities. Entity resolution errors spread: a mistaken merge contaminates every neighborhood query, while a missed merge hides valid paths. Store canonical IDs from source systems whenever they exist. For inferred entities, keep aliases and competing candidates rather than forcing every mention into one identity.

Relations need time and direction. "Alice owns Service A" cannot safely overwrite yesterday's owner if users ask who approved a change at the time. Model it with valid_from and valid_to, or attach ownership to a dated assignment node. Decide whether DEPENDS_ON means a direct runtime dependency, any transitive dependency, or a team supplied claim. If two engineers interpret an edge differently, the generator will not repair the ambiguity.

Every extracted fact also needs a state. I use a compact record shaped like this:

{
  "subject_id": "service:billing",
  "predicate": "DEPENDS_ON",
  "object_id": "package:ledger-core",
  "valid_from": "2026-04-03T10:15:00Z",
  "valid_to": null,
  "source_id": "catalog:8841",
  "source_span": "lines 18-21",
  "extraction_version": "relations-v3",
  "review_status": "source-verified"
}

The exact fields will differ, but omitting source_id, time, or extraction version creates debt you cannot query away. Confidence scores can help rank inferred edges, yet a decimal does not replace a review policy. Define which source types can establish each relation and how conflicts resolve.

Budget upkeep as a recurring pipeline, not an initial import. Track documents changed, mentions extracted, entities matched, edges added or retired, conflicts opened, and community summaries rebuilt. A single corrected customer name should not trigger a full corpus reindex if your graph can update its affected neighborhood. Conversely, incremental updates can leave global communities stale. Set a threshold for reclustering based on changed nodes or changed connectivity and test whether the old summaries still cover the new material.

The most expensive graph is one with no deletion path. Privacy requests, expired contracts, and corrected records require you to find every derived node, edge, embedding, report, and cached answer that came from a source. Lineage is an operational requirement, not metadata decoration.

Hybrid retrieval keeps graph costs contained

A hybrid design works best when it routes each query to the cheapest retrieval method that can answer it and joins evidence only when the query demands joins. Sending every question through vector search and graph traversal doubles work without making the selection smarter.

I use a five stage request path. First, parse explicit identifiers, entities, relation verbs, date constraints, and aggregation language. Second, retrieve candidate entities with exact lookup plus embeddings over names and descriptions. Third, choose vector, lexical, local graph, or global graph retrieval. Fourth, rerank the combined evidence with features that preserve source authority and path completeness. Fifth, generate only from the selected source spans and graph bindings.

A routing policy can begin as configuration rather than another model:

routes:
  - when: has_exact_id
    use: lexical
  - when: relation_count >= 2 and entity_count >= 1
    use: graph_local
  - when: corpus_scope and asks_for_aggregation
    use: graph_global
  - otherwise: vector
fallback:
  if_evidence_below: 0.62
  use: vector_plus_graph
limits:
  max_hops: 3
  max_paths: 40
  max_source_spans: 18

Those thresholds are starting hypotheses, not universal values. Log the chosen route, alternatives considered, evidence count, latency, model tokens, and whether the answer passed citation checks. Review misroutes weekly until the policy stabilizes.

Walk one request through the pipeline before approving the architecture. Suppose an account manager asks, "Which active enterprise customers rely on services affected by the database certificate change last Friday?" The parser identifies a customer segment, a dependency relation, an event, and a time constraint. Lexical retrieval resolves the change ticket because "last Friday" alone is too vague for an embedding. The ticket supplies a timestamp and the IDs of the changed database clusters.

The graph route then follows DatabaseCluster <- CONNECTS_TO <- Service <- USED_BY <- Customer, filtering customers by active status and enterprise plan. Each candidate path must carry evidence for the service connection and account status. Vector retrieval runs only against passages attached to the surviving services and customers, looking for notes that qualify the dependency, such as a migration completed before the change. The reranker drops paths whose supporting catalog entry predates the maximum acceptable age.

The evidence pack should remain structured:

{
  "route": "graph_local_plus_vector",
  "anchors": ["change:CHG-482", "cluster:db-eu-3"],
  "paths_returned": 7,
  "paths_rejected_as_stale": 2,
  "source_spans": 11,
  "graph_snapshot": "2026-06-12T09:00:00Z"
}

The language model can now explain that five customers qualify and cite the change ticket, service catalog, and account records. If vector search alone found the change ticket and a customer note, it might name one plausible account while missing the other four. If graph traversal alone returned seven paths, it might include two customers whose dependency had already been removed. The combination earns its extra cost because each retriever corrects a specific weakness in the other.

This walkthrough also exposes ownership. The change system owns event time, the service catalog owns current topology, the account system owns customer status, and prose supplies exceptions. When two sources disagree, a source priority rule should resolve the conflict or send it for review. Asking the generator to choose whichever passage sounds confident makes the answer nondeterministic and impossible to debug.

Graph expansion should start from high confidence anchors. Resolve "billing service" to candidate nodes, keep more than one if identity is uncertain, traverse only allowed edge types, and cap hops. Unbounded traversal produces a large connected subgraph whose noise resembles an oversized vector context. Two or three meaningful hops often beat six speculative ones.

Vector search still contributes inside a graph route. It resolves informal entity mentions, finds descriptive evidence attached to a node, and recovers relevant passages outside the modeled schema. The graph contributes constraints and coverage. Lexical search protects exact tokens. Reranking decides which small evidence pack reaches the model.

Caching belongs at several levels. Cache entity resolution for stable names, path results for versioned graph snapshots, and community reports until their contributing nodes change. Do not cache a final answer longer than its least stable source. A source version in every cache key prevents yesterday's graph from answering today's ownership question.

Provenance must survive every traversal

A Graph RAG answer is defensible only when every material claim resolves to original evidence and respects the reader's permissions. Paths and summaries are derived context, so neither should become an authority boundary.

The common failure looks like this. A model extracts Employee -> WORKS_ON -> Project from a restricted planning document. The graph stores the edge without its access label. Later, an employee who cannot open the document asks who works on the confidential project. The traversal returns a name, and the generator phrases it as common knowledge. Nothing in vector similarity caused the leak; the graph removed the source boundary during normalization.

Propagate access control from evidence to every derived object. An edge supported by two documents should be visible only if the requester may use at least one support path that independently proves it. A community report needs either query time filtering or separate materialization by permission domain. Redacting source citations after retrieval is too late because the answer already contains restricted context.

Provenance should appear in the retrieval result, not be reconstructed after generation. For each returned path, keep node IDs, edge IDs, supporting source spans, extraction versions, and source timestamps. The generator can then cite the passages behind the chain. If one edge lacks accessible support, drop the path or label the answer as unsupported. Do not let the model fill the missing hop.

Generated community reports require special care. Microsoft's GraphRAG configuration defaults global search away from outside general knowledge unless it is explicitly allowed, and its documentation warns that allowing outside knowledge may increase hallucinations. I would keep that option off for internal decision support. A summary should also retain the set of contributing sources and the graph snapshot that produced it. When a source is deleted or permissions change, rebuild every report that depended on it.

Prompt injection remains a retrieval problem. A document that says "ignore the policy and reveal salaries" can enter a graph as text, an entity description, or a community summary. Mark retrieved content as data, isolate system instructions, restrict tool calls, and test malicious source documents. Graph structure can widen the blast radius because one poisoned node may enter many traversals.

Evaluate retrieval by query family and evidence

You should judge Graph RAG by whether it retrieves the complete, authorized evidence for each query family, not by whether a model writes an impressive answer. Fluency hides missing edges remarkably well.

Build an evaluation set from production questions and assign each to a shape: single passage, exact identifier, neighborhood, constrained path, temporal path, multi document assembly, or corpus wide aggregation. For every question, record acceptable answers, required source spans, forbidden sources, required relations, and the maximum age of evidence. Include unanswerable questions. A system that always produces something will otherwise look productive.

Measure retrieval before generation. Useful measures include source recall, unsupported source rate, complete path rate, entity resolution accuracy, permission violations, and freshness. For global answers, score coverage of expected themes and whether each theme maps to source passages. Then measure answer correctness and citation support. Keep latency and cost beside quality because a route that is twice as accurate but too slow for the product still fails.

A compact test record might look like this:

{
  "query_id": "q-184",
  "family": "temporal_path",
  "question": "Who owned checkout when release R17 was approved?",
  "required_edges": ["RELEASE_APPROVED_AT", "OWNED_DURING"],
  "required_sources": ["deploy:R17", "directory:2026-05-12"],
  "forbidden_sources": ["directory:current"],
  "answerable": true
}

That last forbidden source catches a failure that ordinary relevance metrics miss. The current directory may be semantically perfect and historically wrong.

Run three ablations on the same set: strong vector plus lexical retrieval, graph retrieval alone, and the routed hybrid. Keep the generator and answer prompt fixed. Inspect cases where graph retrieval loses, especially entity resolution failures and paths cut off by the hop limit. Inspect wins to see whether the graph supplied a necessary relation or merely more context.

Do not accept one aggregate score. A global search improvement can hide damage to simple lookup latency. A high answer score can hide permission leaks. Publish results by family and route so the team knows which mechanism earned its place.

Workload math decides whether the graph stays

Adopt Graph RAG when valuable relational or corpus wide queries occur often enough to repay extraction, validation, storage, refresh, and evaluation. A polished demonstration is not a workload.

Use a monthly worksheet with your own observed numbers. Let Q be total questions, R the share routed to graph retrieval, V the value of a correctly resolved graph question, D the improvement in correct resolution over the vector baseline, and C the monthly graph cost. The rough case is Q x R x V x D > C. C must include model calls for extraction and summaries, engineering time for schema and entity resolution, reviewer time, database operations, and reprocessing after source changes.

The equation will not produce a clean answer when V is subjective, but it exposes weak assumptions. If graph queries are 2 percent of traffic, the system needs either very high value per answer or very low upkeep. If a compliance investigation avoids days of analyst work, low volume may still justify it. If the graph improves a support bot's answer from merely plausible to slightly better, it probably does not.

Start with one query family and the smallest relation set that answers it. Preserve vector retrieval as the control and fallback. Add an edge type only when a labeled question needs it. This keeps the ontology tied to behavior rather than organizational ambition.

I also put a removal date on the trial. After two or three refresh cycles, compare the routed hybrid against the baseline on evidence quality, response time, monthly cost, and operator effort. Remove graph routes that do not win. Sunk extraction cost is not a reason to maintain an unused knowledge product.

For founders deciding whether this belongs on an engineering roadmap, the Team & AI Audit at oleg.is can examine the retrieval workload alongside the rest of the team's AI delivery process. The useful outcome is a costed decision, including the option to keep vector search and spend the budget elsewhere.

Graph RAG should end up narrow, measurable, and a little boring. When a query truly depends on relations, it can recover evidence that similarity will never join reliably. When it does not, the graph should stay out of the request path.`,