I built a knowledge graph to sit alongside vector search, and the most useful thing it produced was a clear answer to what each layer is actually for.
This was a build-to-learn experiment: a real stack with a graph database, migrations, a test suite and an eval harness, built to answer an architectural question rather than to ship a feature. No users, so read the conclusions accordingly.
The Question
Vector retrieval answers “what does this document say.” It does that well, and it has a hard ceiling: it returns the closest passage and cannot connect facts that live in different documents.
Plenty of real questions need the connection rather than the passage. Which jurisdictions along a path impose a given requirement, which of those requirements interact, what changes when one input changes. Top-k cosine similarity has no mechanism for any of that. It can hand you the paragraph that mentions two things. It cannot tell you the two things are related, or find the third thing related to both.
So: put a graph next to it and see what each one is good for.
The Three Layers
The graph models the nouns of the domain and the typed relationships between them, built with a temporal knowledge-graph library over a graph database.
Two decisions in the extraction step did most of the work, and both are the kind that separate a graph that works from one that quietly doesn’t.
Numbers stay out of the graph. A fee or a limit belongs in a table where it can be queried exactly, not in a graph where it becomes a fuzzy entity you have to hope you matched correctly. The extraction prompts are patched specifically to refuse to make nodes out of them.
Names get canonicalized aggressively. The same organization written three ways across three documents has to collapse into one node. Skip this and the graph quietly fragments into near-duplicates, every traversal misses half its neighbors, and nothing errors. It just gets worse at its job in a way that’s invisible from the outside. This is the unglamorous half of making a knowledge graph work and it’s why a lot of them don’t.
The vector layer is pgvector over document passages. The structured layer is ordinary relational tables holding the values you compute against. An orchestration layer exposes all three to a model as separate tools and lets it pick.
What I Learned
A graph is a lossy abstraction, and losing text is the entire point of it.
It stores that two things are related. It does not store the paragraph that says so. Ask it for a specific number and you get a well-connected entity where you needed a sentence. For factual lookup, retrieving the raw passage wins, and it isn’t close.
That reads like a criticism and it isn’t. It’s a statement about what each layer is for:
- The graph earns its place on traversal, and on connecting documents that share no vocabulary
- The vector layer earns its place on returning the exact text
- The tables earn theirs on arithmetic
A system that uses one of those for all three jobs does two of them badly. The useful question was never “which retriever is better.” It was “what is each one unable to do.”
Why This Holds Without a Benchmark
This is a property of the data structure, not a result that needs a head-to-head to believe. A graph stores that two things are related. It does not store the paragraph that says so. Ask it for a figure and it returns an entity, because an entity is the only thing it has. That’s true of every knowledge graph ever built, which is exactly why the interesting question was never “which one wins” but “what is each one for.”
Query expansion follows the same logic. Entities pulled from the graph and appended to a query before embedding reach passages related through the graph that share no vocabulary with the question, because that’s what the graph knows and the embedding doesn’t. It’s a mechanism, and it’s one worth building deliberately rather than hoping the embedding finds its way there.
The Practical Version
If you’re deciding whether to put a knowledge graph in your retrieval stack:
Do it if your questions are about relationships between things, across documents, and your users are asking “which,” “what connects,” or “what else.” Don’t do it expecting better fact lookup, because you will be disappointed in a way that takes months to notice.
And budget for canonicalization, because that is the actual work. The graph is easy to build and hard to keep coherent, and an incoherent graph fails silently.
Building retrieval over a corpus that’s giving you trouble? Let’s talk.