GraphRAG: When Vector Search Isn't Enough for LLM Retrieval

Standard RAG treats a document collection as a bag of independent chunks. That works fine for factual lookups - "what is the refund policy" pulls the one chunk that answers it. It falls apart on questions that require connecting facts across multiple documents: "which vendors does our highest-risk supplier also work with" has no single chunk that contains the answer, because the answer lives in the relationships between entities, not in any one passage. GraphRAG exists to close that gap.

Why Vector Similarity Alone Misses Relational Questions

Vector search finds chunks that are semantically close to the query. It has no concept of entities or how they relate to each other across the corpus. Ask a vector-only RAG system a multi-hop question - one that requires chaining two or more facts together - and it either retrieves fragments that individually look relevant but don't connect, or it retrieves nothing useful at all because no single chunk contains the composed answer.

What GraphRAG Actually Adds

GraphRAG, as formalized in Microsoft's research and now implemented across several open-source frameworks, adds an extraction and indexing step before retrieval: an LLM reads each chunk and pulls out entities (people, organizations, products, concepts) and the relationships between them, building a knowledge graph on top of the raw text. At query time, instead of only doing vector similarity search, the system can also traverse the graph - following entity relationships to assemble context that no single chunk contains.

Documents
    │
    ▼
Chunking (same as standard RAG)
    │
    ▼
Entity + Relationship Extraction (LLM pass per chunk)
    │
    ▼
Knowledge Graph (nodes = entities, edges = relationships)
    │
    ▼
Community Detection (cluster related entities into topics)
    │
    ▼
Community Summaries (LLM-generated summary per cluster)
    │
    ▼
Query Time: Vector Search + Graph Traversal + Community Summaries

Step 1: Entity and Relationship Extraction

Each chunk goes through an LLM extraction pass that pulls structured entities and relationships:

from pydantic import BaseModel
import json

class Entity(BaseModel):
    name: str
    type: str  # PERSON, ORG, PRODUCT, CONCEPT, etc.
    description: str

class Relationship(BaseModel):
    source: str
    target: str
    relation: str
    description: str

extraction_prompt = """Extract entities and relationships from this text.
Return JSON: {{"entities": [...], "relationships": [...]}}

Text:
{chunk_text}"""

async def extract_graph_elements(chunk_text: str, llm) -> dict:
    response = await llm.ainvoke(extraction_prompt.format(chunk_text=chunk_text))
    data = json.loads(response.content)
    return {
        "entities": [Entity(**e) for e in data["entities"]],
        "relationships": [Relationship(**r) for r in data["relationships"]],
    }

This is the expensive part of GraphRAG - every chunk needs an LLM call for extraction, and entity names need resolution across chunks ("Acme Corp" and "Acme Corporation" must merge into one node). Budget for this being 3-5x the cost of a standard RAG ingestion pass.

Step 2: Building the Graph

Extracted entities and relationships get loaded into a graph database. Neo4j is the common choice for production:

from neo4j import GraphDatabase

driver = GraphDatabase.driver("bolt://localhost:7687", auth=("neo4j", "password"))

def load_graph(entities, relationships):
    with driver.session() as session:
        for e in entities:
            session.run(
                "MERGE (n:Entity {name: $name}) "
                "SET n.type = $type, n.description = $description",
                name=e.name, type=e.type, description=e.description,
            )
        for r in relationships:
            session.run(
                "MATCH (a:Entity {name: $source}), (b:Entity {name: $target}) "
                "MERGE (a)-[rel:RELATES {type: $relation}]->(b) "
                "SET rel.description = $description",
                source=r.source, target=r.target,
                relation=r.relation, description=r.description,
            )

Step 3: Community Detection and Summarization

For questions that require reasoning across the entire corpus ("what are the main themes"), GraphRAG runs a community detection algorithm - typically Leiden clustering - over the graph to group densely connected entities into topical clusters. Each cluster then gets an LLM-generated summary. These community summaries are what makes GraphRAG able to answer global, corpus-wide questions that no amount of chunk-level retrieval can answer, because no single chunk contains a summary of the whole corpus.

# Simplified - graspologic or networkx + community detection libs handle the algorithm
import networkx as nx
from networkx.algorithms.community import louvain_communities

G = nx.Graph()
# ... populate G from your Neo4j graph ...

communities = louvain_communities(G, seed=42)

for i, community in enumerate(communities):
    entity_descriptions = [G.nodes[n]["description"] for n in community]
    summary = await llm.ainvoke(
        f"Summarize the common theme across these related entities:\n"
        + "\n".join(entity_descriptions)
    )
    # store summary indexed by community id for query-time retrieval

Query Time: Local vs Global Search

GraphRAG systems typically expose two retrieval modes, and picking the right one per query matters:

• -Local search - start from entities mentioned in the query, traverse their graph neighborhood, pull in connected entities and relationships. Best for specific, entity-centered questions.
• -Global search - query against the community summaries instead of individual chunks. Best for broad, corpus-wide questions about themes or patterns.

A practical production system routes the query to the right mode with a cheap classifier call first: does this question name specific entities (local) or ask for a broad synthesis (global)? Routing incorrectly doesn't break the system, but it does waste the graph structure's advantage.

Is GraphRAG Worth the Complexity?

Honestly - for most RAG use cases, no. GraphRAG roughly triples your ingestion cost and adds real operational complexity (a graph database, entity resolution, community detection pipelines) for a capability that only matters when your actual query distribution includes multi-hop or corpus-wide questions. I only reach for it when a client's use case explicitly requires relational reasoning - fraud detection across connected entities, compliance questions that span multiple related documents, or research synthesis over a large corpus. For straightforward factual Q&A, standard hybrid RAG with reranking gets you 90% of the value at a third of the engineering cost.

Frequently Asked Questions

Do I need Neo4j specifically, or can I use another graph store?

Neo4j is the most common choice because of Cypher's query ergonomics and mature tooling, but any property graph database works - Amazon Neptune, ArangoDB, or even a well-indexed PostgreSQL schema with recursive CTEs for smaller graphs. The extraction and summarization logic is database-agnostic.

How does GraphRAG handle documents that update frequently?

This is GraphRAG's weakest point. Updating a chunk in standard RAG means re-embedding one chunk. Updating a chunk in GraphRAG can require re-extracting entities, re-resolving them against existing nodes, and potentially re-running community detection if the update changes graph structure. For corpora that change hourly or daily, the re-indexing overhead can dominate. GraphRAG suits relatively stable corpora with infrequent, batched updates better than real-time-updated ones.

Can I combine GraphRAG with standard vector RAG instead of replacing it?

Yes, and that's how most production systems actually run it - vector search and BM25 handle direct factual retrieval as the fast path, while graph traversal and community summaries handle the multi-hop and global questions that vector search alone can't answer. Route between them based on query type rather than treating GraphRAG as a full replacement.

What's the biggest mistake teams make adopting GraphRAG?

Adopting it before confirming their actual users ask multi-hop or global questions. Most teams profile their query logs after the fact and discover 80-90% of real queries are single-hop factual lookups that standard RAG already handles well - meaning the extra GraphRAG infrastructure cost was paid for a minority of the query distribution. Check your query logs before you build the graph.