The Challenge
Heavy-haul carriers move loads that are too wide, too tall, or too heavy to travel without a permit. Every state DOT issues those permits as a PDF, every state formats them differently, and the authorized route is printed as a table of road names and turn instructions that a human is expected to read and follow.
The job was to turn that PDF into structured data and a real route on a map: the dimensions, the dates, the escort requirements, the restrictions, and the actual road geometry the driver is legally allowed to occupy.
Two constraints made it hard. A permit is a compliance document, so the same PDF has to produce the same extraction on every single read, forever. And a wrong road is worse than no road, because a route that looks authoritative and isn’t will put an oversize load under a bridge that can’t clear it.
What I Built
A Python service that ingests permit PDFs and emits structured permit data plus resolved route geometry.
Measuring It Honestly
The part I’d point at first isn’t the pipeline. It’s the measurement policy, because on this problem the easiest thing in the world is to convince yourself you’re doing better than you are.
The eval harness has two independently versioned tiers. The per-PR tier is a zero-ratchet regression gate over cached documents, split between a visible set and a locked-away held-out set. It runs offline in under a second, and any accuracy drop fails the pull request outright. There is no “we’ll fix it next sprint” threshold to negotiate with.
That gate is also the corpus you develop against, and it carries synthetic documents alongside real ones. Both facts make its pooled score a bad estimate of real-world accuracy, and a tuned-against number always flatters you. So the harness carries a written rule that the gate number may never be quoted as accuracy. It is a regression tripwire and nothing else. Real-world accuracy comes from separate held-out sets of real permits, measured on a cadence and always cited with its sample size.
Building the thing that stops your own team from overstating your own numbers turns out to matter more than any individual model choice.
Structure First, Then Language
The naive version of this is “OCR the PDF, hand the text to a model, ask for JSON.” It fails on the documents that matter, because the route lives in the layout, and flattening a table to text destroys the thing you needed.
The current pipeline reads document structure before it reads language. Layout-aware OCR preserves the table cells. Deterministic detectors then classify the document into one of a small set of route families, the recurring typographic conventions used to lay out a route table. The useful finding is that these families don’t map to states. A handful of families covers the country, where a parser per state never would. Only after the family is known does the language model run, and it runs against a structure it can trust.
Expert Consensus, Not a Single Call
Route extraction runs as an ensemble of four expert groups, each owning one facet of the problem: general sanity, state boundary terminals, the ordered corridor spine, and exit ramps. Each group fans the same payload across several distinct models and scores agreement as the modal fraction over that group’s headline field. If agreement falls below half, the group spawns exactly one tiebreaker, always a model deliberately not on the roster, because a re-vote from a model that already answered cannot break a split. The tiebreaker only ever adds a vote, never removes one.
The part worth stealing is how the groups combine. They don’t vote against each other, because a cross-group vote is a category error: the boundary group has nothing useful to say about exit numbers. Instead each group owns its slice of the output outright and an orchestrator merges by ownership. Disagreement between groups isn’t a tie to be broken, it’s a sign one of them was asked the wrong question.
A separate interpretation pass then folds ramps and frontage roads into their parent highways and grounds every output corridor against the verbatim source text. If a road appears in the model output with no textual anchor in the permit, it gets dropped and audited. That rule is the entire anti-hallucination story, and it’s enforced mechanically rather than prompted for.
The resolved corridors are then matched against a live road graph, which is where the details bite. I-35E and I-35W are different roads that a naive normalizer will happily collapse into one, and they go to different cities.
Determinism as a Contract
Raw OCR blocks and every model response are persisted on first read and replayed on every read after. Re-extracting a stored permit never calls a model again. It replays. That’s what makes the output reproducible under audit, and it’s also what makes the evaluation harness possible, because a cached extraction can be scored offline, bit for bit, with no network.
Results
One scored comparison, run against 49 real permits across 25 states and 7 route families and graded on 845 truth conditions built engine-blind from the source documents rather than from either engine’s output:
- Restriction recall went from 0.080 to 0.523, roughly 6.5x, against the previous structure-blind approach
- Restriction precision gave up 1.8 points, 0.822 against 0.840, which was the trade accepted for the recall gain
That is a dated measurement on a defined cohort rather than a marketing number, which is exactly the distinction the accuracy policy exists to enforce. Every figure this system publishes comes with its sample attached.
Technical Highlights
- Live road-graph resolution over a self-hosted OSM/PostGIS instance with derived neighbor and boundary-crossing tables
- A route-family taxonomy classifying the distinct ways state DOTs typographically encode a route table, so a new state is a classification problem rather than a new parser
- Ownership-based merge across expert groups, which removes a whole class of consensus bug that voting introduces
- Two-tier eval harness: an offline zero-ratchet per-PR gate plus a live operator-run tier scored against expert-approved ground truth
- A written accuracy policy separating the CI tripwire from the real-world headline, so neither number can be quoted as the other