The Challenge
In plain terms: a team used to read permit manuals and DOT web pages for every jurisdiction and re-key the rules by hand, then do it again when the rules changed. This system reads the sources, extracts the facts into structured data, has an independent check review every extraction, and puts anything uncertain in front of a person to approve. People make the calls. The system remembers them and keeps the data current.
Getting a language model to read a permit manual is the easy part. Knowing whether to trust what it read is the whole problem.
Oversize and overweight freight is governed by rules that differ in every jurisdiction: legal dimension limits, escort and pilot-car requirements, permit fees, curfews, insurance minimums, penalty schedules. There are 64 relevant jurisdictions across the US and Canada. The rules live in PDF permit manuals and DOT web pages, written by 64 different agencies with 64 different conventions, and they change without announcement.
A platform that plans and prices heavy-haul freight needs those rules as structured data, current, with provenance back to the source that justifies each value.
What I Built
A monitoring pipeline that discovers official regulatory sources per jurisdiction, downloads them, extracts structured facts against JSON schemas, cross-checks every extraction with an independent judge, and re-checks sources on a schedule so downstream systems query facts that are still true.
The Interesting Part Is the Gates
Any competent engineer can wire an LLM to a PDF. What separates a demo from a system you’d let price freight is the machinery that decides what gets published, and most of the engineering here went into making sure that machinery cannot fool itself. Four properties hold.
Trust and coverage are separate axes. A jurisdiction’s status reflects the judge’s verdict. A separate, additive coverage grade reflects how much of the schema is actually populated. The two are kept deliberately independent, because an extractor’s self-reported confidence should never be able to launder itself into a trust signal, and a sparsely populated jurisdiction should never look trustworthy by default. Measured across the corpus, self-reported confidence and real field completeness are essentially uncorrelated, which is exactly why they get their own axes.
Silence is not approval. “Validated” means a recorded pass, not the absence of unresolved judge feedback. Those sound equivalent and are not: a jurisdiction reviewed cleanly and one never reviewed at all would otherwise leave the same trace on disk. A deterministic audit compares category files against judge artifacts so the two cases always separate, and absence of evidence is never encoded as evidence of absence.
Completeness is measured by shape, not by path. Language models routinely write semantically correct values at non-canonical schema paths, so a completeness counter that only credits exact canonical paths would score correct extractions as missing. This one classifies the shape of every leaf path instead of checking top-level keys. Across the corpus that recognizes 1,775 populated values a shallow check would have counted as gaps, so the metric that catches thin jurisdictions is measuring the data rather than the drift.
Publishing is per category, and add-only. A judge verdict is per category rather than one scalar per jurisdiction, so a category blocked by a source behind a CDN wall or a statute that is simply missing never holds the clean categories hostage. The publish step is a per-category selector with strict add-only sync: a partial publish ships the good categories without stripping previously published data or lowering the bar on the bad ones. Good data ships, bad data waits, and neither decision contaminates the other.
Asking It Questions
Structured data behind a gate is only useful if someone can reach it. The dashboard carries a chat agent that answers regulatory questions in plain language, over the same corpus, with citations back to the source.
Retrieval works on distilled fact cards rather than raw document text. Roughly 3,000 facts are embedded, covering the 60 jurisdictions far enough through the pipeline to have publishable data, and a question is answered by embedding the query, ranking by cosine similarity, and handing the top matches to the model as tool results. The model composes the answer and cites the source URLs that came back with each fact, plus a verbatim excerpt where the extractor recorded one.
There is no vector database, and that was a measured decision rather than an omission. Benchmarked as a vectorized operation, cosine similarity across the whole corpus runs in well under a millisecond, so an approximate index would add operational surface to solve a ranking problem that does not exist at this size. The note recommending against one carries the benchmark and a threshold for revisiting it, and is equally clear that the real cost sits elsewhere: the index ships as one large JSON file, and shrinking that download is the optimization actually worth doing. Infrastructure you can justify declining is worth as much as infrastructure you can justify buying.
The agent has fourteen tools, and they are not all reads. It can inspect corpus status, pull coverage matrices, work the review queue where a person resolves whatever the judge flagged, trace pipeline runs, and trigger a workflow. Mutations are attributed to the authenticated user rather than to the agent, so the audit trail survives contact with an assistant that can change things, and the human review step stays a human one.
An Index That Cannot Shrink
The sharpest edge in this system is the seam between a non-deterministic pipeline and the derived search index it maintains. A cache that hydrates from the wrong place, or a model-driven step that clears a working directory between invocations, can hand the rebuild an almost empty corpus. A smaller index does not fail. It just answers worse, and nothing downstream notices.
So the rebuild is monotonic by construction. It starts from the previously published index, overlays whatever is present on disk, and unions the two by fact id. An in-flight wipe can only fail to add facts. It cannot remove them. Two regression tests hold that property in place.
The rule generalizes past this system: when a non-deterministic process maintains a derived index, the rebuild has to be incapable of shrinking it, because nothing downstream will notice if it does.
How the Fixes Got Made
Each of these went through the same loop: a written proposal, an adversarial verifier pass that critiques and rewrites it, then a scoped implementation with unit tests run against the real corpus before any of the riskier orchestration got wired up.
The adversarial pass earned its keep. It rejected proposals outright, including one plausible-looking fallback that would have filled missing weight limits with a federal default. That would have institutionalized fabrication inside a compliance dataset while making the coverage numbers look better, which is the exact failure mode the gates exist to prevent.
Technical Highlights
- 64 jurisdictions covered: 50 US states, DC, and 13 Canadian provinces and territories
- Independent LLM-as-judge pass reviewing every extraction against source text before publish, run as a separate invocation rather than a self-check
- Object storage as the single source of truth, with an append-only extraction history and no database to drift from it
- Monthly scheduled agentic workflow orchestration, plus five manual run modes for backfill, retry, and expansion
- Serverless read API with per-jurisdiction fact retrieval, behind JWT auth
- Vector-embedding fact index powering semantic search and a tool-using chat agent in the dashboard
- Deterministic audits layered over probabilistic extraction, so gate decisions are reproducible even though generation isn’t