Skip to content
Published on

From Documents to a Knowledge Graph: An Honest Pipeline

🇰🇷KO🇺🇸EN🇯🇵JA🇨🇳ZH
Split View필사 모드
Share
Authors

Introduction — The Demo Is One Call; Production Is a Pipeline

"Feed in documents, get a knowledge graph." In a demo, this looks like a single LLM call. Drop a few documents into LangChain's LLMGraphTransformer or LlamaIndex's PropertyGraphIndex and you get a handsome picture of nodes and edges. The whole thing takes five minutes.

The trouble starts right after. Turning tens of thousands of a customer's pages into a graph you can actually query — what a Forward Deployed Engineer actually does — is not one call but a six-stage pipeline. And most of the cost and pain lives not in the stage everyone watches (extraction) but in the stage nobody demos (entity resolution).

This post walks that pipeline honestly — including where the money goes and where it breaks.

documents
  │  (1) schema / ontology first  — decide node & edge types
  ▼
chunks ──(2) extraction──▶ raw triples   ← one LLM call per chunk, boundable
  │       LLM-guided, schema-constrained
  ▼
(3) entity resolution   ← same entity, many surface forms · THE HARD PART
  │   blocking → scoring → clustering (canonicalization)
  ▼
(4) load into graph store   (+ provenance on every edge)
  │
  ├─(5) incremental updates & schema drift  ← never one-and-done
  └─(6) quality / eval   ← precision (correct?) · recall (missing?)

where the effort actually concentrates (rough rule of thumb):
  extraction         ~20%   (visible, boundable)
  entity resolution  ~50%   (long tail, manual)
  eval + provenance  ~30%   (or you ship a graph you can't trust)

Schema First — Without an Ontology, You Just Pile Up Noise

The temptation is to start extracting. But if you extract without deciding what to extract, the result is not a graph — it is a heap of noise. The same concept comes out as company, firm, and organization; relationship names differ from sentence to sentence. You can draw a graph like that, but you cannot query it.

So the first stage is not code, it is a decision: which node types and edge types exist. That is your domain ontology, and why you settle it first is the subject of Turning Domain Knowledge into an Ontology. What matters here is that the decision governs the whole pipeline. The next stage's extractor takes this schema as a constraint and refuses to emit anything outside it. The schema becomes the grammar of the graph.

Extraction — Traditional NER or an LLM?

The second stage is actually pulling entities and relationships out of text. There are two lineages.

The traditional approach chains named-entity recognition (NER) and relation extraction into a pipeline. It is mature, cheap, and strong at volume. The costs are just as clear: each domain needs freshly labeled training data, errors in the entity step propagate into the relation step (the pipeline's chronic disease), and it struggles with context that spans sentences.

LLM-based extraction is the opposite trade. It starts zero-shot or few-shot with no labels and maps results onto a schema while keeping context. In exchange it introduces two new problems — hallucination (inventing relationships that are not there) and surface-form variance (calling the same thing something different each time). The first makes stage 4 harder; the second makes stage 3 harder.

Three real tools make the trade-off concrete.

  • LangChain's LLMGraphTransformer. Its convert_to_graph_documents turns documents into nodes and relationships. You constrain types with allowed_nodes and allowed_relationships (both default to an empty list, meaning allow everything), and allowed_relationships even accepts (source_type, relation, target_type) tuples. It still lives in the experimental module, and with some models it returns empty results.
  • LlamaIndex extractors. The same job comes in three flavors. SimpleLLMPathExtractor extracts freely with no schema and "may produce a larger number of diverse relationships" but "might lack consistency in entity and relation naming." SchemaLLMPathExtractor uses a predefined schema and "produces a more structured graph" but "might miss a lot of relationships that don't fit the predefined schema." DynamicLLMPathExtractor sits between them — guided by an initial schema but able to expand it — aiming for "a balance between diversity and consistency." The trade-off is fossilized right into the class names.
  • Microsoft GraphRAG. It extracts entities and relationships from each text unit and attaches a description to each, then combines the many descriptions of one entity into "coherent, non-redundant descriptions."

The cost here is predictable. Extraction is fundamentally one LLM call per chunk. It grows linearly with the corpus, batches cleanly, and caches well. It is the most manageable stage in the pipeline. The hard part is elsewhere.

Entity Resolution — The Genuinely Hard Part

When extraction finishes you have hundreds of thousands of triples, and a large fraction of them are different names for the same thing. Kim Minsu, Minsu Kim, and Kim, M. might be one person — or might not. Sorting that out is entity resolution, and it is where the quality of a knowledge graph is won or lost.

The textbook shape is three stages.

  1. Blocking. Narrow the candidates. Comparing every pair of entities blows up quadratically (100,000 entities means ten billion pairs), so you use techniques like embedding-based approximate nearest-neighbor search to gather only plausibly-similar entities into the same block. As the field's maxim goes, blocking matters more than matching — a pair you miss here will never meet later.
  2. Scoring. Assign a similarity to the pairs inside a block — string similarity, relational proximity, embeddings, behavioral signals.
  3. Clustering / canonicalization. Group same-entity pairs by score and collapse each group into a single canonical node.

The failure modes are symmetric.

  • Over-merge. Two different people named Kim Minsu collapse into one node. Now the graph lies — one person's history is spliced with another's. It is silent, which is what makes it dangerous.
  • Under-merge. IBM Corp, International Business Machines, and IBM Corporation stay as three nodes. A query hits one of the three and misses the other two-thirds.

Embeddings help a lot here. Dense vectors link entities by meaning even when the strings do not overlap — they can gather those three IBM forms "without a single explicit rule." But embeddings alone are not enough. Neither rule-based methods nor pure LLMs hold up at web scale on their own. Production entity resolution is a combination: blocking plus similarity matching plus probabilistic or ML classification plus global consistency reasoning.

Here is the honest part. Many tools do not do this stage for you. GraphRAG's default pipeline, for instance, merges mentions that share the exact same name and summarizes their descriptions — but it does not connect IBM Corp to IBM Corporation. Real entity resolution is on you. This is where an FDE spends the most time on a customer project.

Loading Into a Graph Store — And Provenance on Every Edge

The resolved triples now go into a graph store. A property graph (Neo4j and friends) attaches properties to nodes and edges; RDF loads triples into a triplestore you query with SPARQL. The choice follows your team's tooling and query patterns.

The one thing you must not skip at load time is provenance. Every edge should carry which document and which chunk it came from. This is exactly why GraphRAG preserves document metadata and links each document to the text units created from it. A graph with provenance can be audited, can be cited, and tells you where to fix a mistake. A graph without it is a pile of plausible-sounding claims. And that provenance becomes the raw material later, when graph RAG grounds its answers in sources.

Incremental Updates and Schema Drift — Never One-and-Done

A customer's documents do not stand still. New contracts arrive, the org chart changes, products get added. The graph has to move with them.

A full rebuild is easy but expensive, so you want incremental updates. GraphRAG 1.0's update command "computes the deltas between an existing index and newly added content and intelligently merges the updates to minimize re-indexing" — it tries to fold in only what changed. There is an honest footnote, though: if the new content shifts the community structure itself, much of the index must be recomputed, and in the worst case it degrades to a full re-index. The saving grace is that LLM caching makes re-runs substantially cheaper.

The subtler problem is schema drift. The ontology you froze in stage 1 slowly falls out of step with reality. New entity types and relationships appear, and a schema-constrained extractor quietly drops them on the floor. So the schema is not something you erect and forget — it is a living asset that needs governance. Construction is not an event; it is an operation.

Quality and Evidence — Do Not Ship What You Cannot Evaluate

"The graph looks good" is not an evaluation. A constructed knowledge graph can and should be measured at the level of triples.

The basic axes are four — accuracy, completeness, consistency, and redundancy. In practice you measure the first two first. Precision gives you correctness — the fraction of extracted triples that are right — and recall gives you completeness — the fraction of the triples that should exist that you actually captured. Hand-building a gold sample and checking edge by edge is the most honest starting point, and link-prediction-based measures can assess quality automatically without a gold standard.

And at the junction of evaluation and trust sits provenance again. Pairing each value with its source span (attribution) both filters hallucination — a triple that cannot cite a source is one to doubt — and lets the end user verify the answer. The eval pipeline and provenance are not decoration you bolt on later; they are the only thing that makes the graph trustworthy.

The Honest Cost — Where It Breaks

To sum up, the cost distributes like this.

  • Extraction is linear in the number of chunks. It is visible, budgetable, and caches well. The easiest stage.
  • Entity resolution is a long tail. Eighty percent automates; the remaining twenty demands domain knowledge and manual work — and most of the quality is decided right here.
  • Community summarization (GraphRAG-style) gets as expensive as extraction at scale, sometimes more, because it generates an LLM report for every community.
  • Incremental updates are not free, and in the worst case degrade to a full rebuild.

The demo shows you the first line only. The rest never appears in the demo, so it never appears in the estimate, so the project slips.

Closing

Through an FDE's lens, turning documents into a graph is roughly 20% extraction and 80% resolution, evaluation, provenance, and operation. Execution — the extraction — is getting cheap; the LLM does it for you. What holds value is judgment: the schema that decides what counts as a node, the resolution rules that decide when two things are the same, the evaluation that can prove you wrong. Those are not the axes today's models hand you.

If one sentence survives, it is this. Schema first, evidence always. The graph is not the goal but a means, and what that means is for — retrieval and answers — is covered in Graph RAG, Explained, while the tools that actually run this pipeline are surveyed in The Knowledge-Graph Tooling Landscape.

References

Discuss on Twitter • View on GitHub