The Challenge
In plain terms: people who need one specific rule out of a 300-page regulatory document can ask for it in a sentence and get the answer with the page it came from, and the system is built so that it cannot cite a page it did not actually read.
Regulatory documents in this domain run 50 to 300 pages, mix narrative rules with dense tables, and contradict each other across jurisdictions. Someone who needs a specific requirement has two options: already know it, or go reading.
The obvious product is a chatbot over the documents. The obvious product is also how someone gets fined, because an answer that sounds authoritative and cites nothing is worse than no answer. Anything shipped here has to be traceable to the page it came from.
Chunking Is Most of the Problem
Fixed-interval chunking destroys documents like these. Split every thousand characters and you get the bottom half of a limits table with nothing indicating which jurisdiction or season it governs. Retrieved later, that fragment looks authoritative and means nothing.
So chunking runs through a model. OCR extracts each page as text and table units, the pipeline builds a structural outline of the document, and a language model chooses boundaries between those units with the prompt’s explicit goal of keeping related content together and tables with the prose that qualifies them.
The granularity is deliberate: the OCR layer emits one unit per page and per table, and the model chooses where to group those units rather than rewriting inside them, which keeps the boundaries auditable. A fixed-interval fallback splits every N units when the call fails, and oversized chunks get hard-split with token overlap to stay under the embedding limit.
Citations That Trace to Real Retrieval
Retrieval is Postgres with pgvector, an HNSW index and cosine distance. Generation is an agentic tool loop: the model decides when to search, issues a query, and receives chunks. It is never handed a pile of context up front and asked to be careful with it.
The source list is assembled server-side from the chunks a tool call actually returned. Jurisdiction, document name, version, page numbers, chunk id, excerpt and retrieval score all come from real retrieval results. The model cannot author an entry in that list, and it cannot add a document that was never retrieved.
The prompt requires a citation before any regulatory claim, the client turns those references into deep links into the source document, and a reviewer can go straight to the page and check the work. Traceability is the product, not a feature bolted onto it.
Where the Engineering Goes Next
Structural traceability is the floor, not the ceiling. The layer above it is semantic: verifying that each claim is entailed by the passage it cites, rather than merely traceable to it. That is a solved research problem and an unsolved product one, and it is the direction this class of system is heading.
Knowing which guarantees you have and which you are building toward is most of what separates a system you can put in front of a regulator from one you cannot.
Retrieval Thresholds Are a Product Decision
Seasonal restrictions are the hardest class of question in this corpus. The relevant chunks score around 0.45 on cosine similarity, under the 0.5 floor a retrieval system typically ships with, while the general-case chunks clear that bar easily. A floor tuned for precision would filter the seasonal chunks out before the model ever saw them, and the answer would come back confident, well cited, and about the wrong season. Nothing would be hallucinating in the usual sense.
The system handles that class explicitly. The similarity floor sits at 0.25, and the prompt carries a rule for seasonal questions: if no seasonal data was retrieved, refuse and say so. If the document mentions a seasonal restriction but points elsewhere for the values, say that rather than substituting the general figures. Use seasonal values only when they were actually retrieved.
A too-strict retrieval threshold is a silent failure. It never errors. It answers a slightly different question than the one asked, in a confident voice, with a citation attached.
Technical Highlights
- Model-chosen chunk boundaries over a structural document outline, with a fixed-interval fallback
- Source lists assembled server-side from real retrieval results, so the document set behind an answer cannot be invented
- Agentic tool loop where the model chooses when to retrieve rather than receiving context up front
- Postgres with pgvector and an HNSW cosine index, with no separate vector store to keep in sync
- Shared reference data and tenant-scoped documents searched in one unioned query, with the tenancy asymmetry commented in place as deliberate so nobody “fixes” it into a security bug
- Scanned documents with no text layer detected at ingest and marked failed, rather than silently indexing zero chunks
- Domain synonym expansion on the tenant-document path, where three different words for the same escort vehicle mean nothing alike to an embedding