graphrag-creator

Create your own GraphRAG: point this tool at a folder of documents and it builds an LLM-derived knowledge-graph index — entities, relationships, hierarchical communities, and community reports — then answers questions over it, including global sensemaking questions ("What are the main themes in this corpus?") that conventional vector RAG cannot handle.

The implementation follows Microsoft Research's paper: Edge et al., From Local to Global: A GraphRAG Approach to Query-Focused Summarization (arXiv:2404.16130).

How it works

flowchart LR
    subgraph Indexing time
        A[Source documents] -->|chunking| B[Text chunks]
        B -->|LLM extraction + gleaning| C[Entities, relationships, claims]
        C -->|merge + summarize| D[Knowledge graph]
        D -->|hierarchical Leiden/Louvain| E[Graph communities]
        E -->|LLM summarization, bottom-up| F[Community reports]
    end
    subgraph Query time
        F -->|shuffle + batch| G[Map: community answers scored 0-100]
        G -->|filter + rank| H[Reduce: global answer]
    end

Loading

Every stage maps to a section of the paper:

Stage	Paper	Module
Documents → text chunks (600 tokens, 100 overlap)	§3.1.1	chunking.py
Chunks → entity/relationship/claim instances, with self-reflection "gleaning" rounds to recover missed entities	§3.1.2, §A.2	extraction.py
Instances → knowledge graph (exact-name merge, LLM-summarized descriptions, duplicate count → edge weight)	§3.1.3	graph.py
Graph → hierarchical communities (Leiden when leidenalg is installed, otherwise networkx Louvain, applied recursively)	§3.1.4	communities.py
Communities → report-style summaries, bottom-up, with degree-prioritized context and sub-community substitution when over budget	§3.1.5	reports.py
Query → map-reduce over community reports with 0–100 helpfulness scoring and filtering	§3.1.6	query/global_search.py
Entity-focused questions over graph neighborhoods + source text	—	query/local_search.py

Installation

pip install -e ".[anthropic]"        # with the default Claude provider
# optional extras:
pip install -e ".[openai]"           # OpenAI provider
pip install -e ".[leiden]"           # exact Leiden community detection (igraph)
pip install -e ".[dev]"              # pytest

Quickstart

# 1. Scaffold a project
graphrag-creator init myproject
cd myproject

# 2. Drop .txt / .md documents into input/
cp ~/my-docs/*.txt input/

# 3. Build the index (uses ANTHROPIC_API_KEY from the environment)
export ANTHROPIC_API_KEY=sk-ant-...
graphrag-creator index

# 4. Ask questions
graphrag-creator query "What are the main themes in this dataset?"                 # global
graphrag-creator query --method local "Who is Jane Doe and what does she do?"     # local
graphrag-creator query --level 0 "Summarize the corpus at the highest level"      # root communities (C0)

# 5. Inspect
graphrag-creator stats
graphrag-creator visualize      # writes output/graph.html (self-contained, open in a browser)

Try it without an API key

The mock provider runs the entire pipeline offline with a deterministic heuristic "LLM" — useful for demos, tests, and understanding the artifacts:

graphrag-creator init demo && cp examples/sample_corpus/input/* demo/input/
graphrag-creator index --root demo --provider mock
graphrag-creator query --root demo --provider mock "What are the main themes?"

Configuration

graphrag-creator init writes a commented settings.yaml. Key knobs:

llm:
  provider: anthropic          # anthropic | openai | mock
  model: claude-opus-4-8
  max_concurrency: 4           # parallel LLM calls during indexing

chunking:
  chunk_size: 600              # tokens per chunk (paper default)
  overlap: 100

extraction:
  entity_types: [PERSON, ORGANIZATION, LOCATION, EVENT, CONCEPT, TECHNOLOGY]
  max_gleanings: 1             # self-reflection rounds per chunk
  extract_claims: false        # also extract factual claims (extra LLM calls)

communities:
  algorithm: auto              # leiden if installed, else louvain
  max_levels: 3                # hierarchy depth (C0 = roots)

query:
  global:
    community_level: 1         # which level answers global queries (paper: C1/C2 best)
    map_batch_tokens: 8000     # paper found 8k context windows best
    min_helpfulness: 1         # drop map answers scoring below this

Tailor entity_types to your domain (e.g. [GENE, PROTEIN, DISEASE, DRUG] for biomedical corpora) — the paper notes domain-tailored extraction meaningfully improves index quality.

Cost note: indexing calls the LLM once per chunk (plus gleaning rounds and description merges), and each community produces one report. For large corpora, consider a cheaper model for indexing (llm.model: claude-haiku-4-5) and keep a stronger model for querying (graphrag-creator query --model claude-opus-4-8 ...).

Index artifacts

graphrag-creator index writes plain JSON to output/ so you can build on top of the index directly:

File	Contents
chunks.json	text chunks with token counts and source document
entities.json	merged entity nodes: name, type, summarized description, degree, community membership
relationships.json	merged edges: description, weight, source chunks
claims.json	factual claims about entities (if extract_claims: true)
communities.json	the hierarchical community tree
community_reports.json	LLM-written report per community (title, summary, rating, findings)
index_meta.json	run statistics and provenance

Using it as a library

from graphrag_creator.config import load_config
from graphrag_creator.index import GraphRAGIndex, run_index
from graphrag_creator.llm import create_provider
from graphrag_creator.query import global_search, local_search

config = load_config("myproject")
provider = create_provider(config)

index = run_index(config, provider=provider)          # or GraphRAGIndex.load(config.output_path)
result = global_search(provider, config, index, "What are the key tensions in the corpus?")
print(result.answer)

Development

pip install -e ".[dev]"
pytest            # full pipeline is covered offline via the mock provider

Design notes & deviations from the paper

• Community detection: the paper uses Leiden (via graspologic). We use leidenalg when installed and fall back to networkx's Louvain — both modularity-based; Louvain keeps the default install pure-Python.
• Entity matching is exact string matching on normalized names, as in the paper's own evaluation; the paper notes GraphRAG is resilient to duplicates because they cluster into the same communities.
• Token counting uses tiktoken when available and a word-count approximation otherwise; budgets are sizing heuristics, not hard limits.
• Local search is an addition (the paper focuses on global sensemaking): it anchors the question to top-matching entities and builds context from their neighborhood, claims, community reports, and source chunks.