Document reading
Vision-first, tiered: a cheap page prioritiser, GPT vision on the pages that matter, and Azure Document Intelligence demoted to a fallback that has to earn its trigger.
Both products start from scanned paper: letters, forms, statements, contracts, affidavits, in PDF and TIFF, at every quality from born-digital to a photocopied fax. The document reader is shared, tiered, and vision-first: an inversion of how the first pipeline started.
Three tiers
Tier 0: page prioritisation. A cheap model classifies each page thumbnail into a coarse type: correspondence, agreement, statement, notice, identity document, blank or other. Only relevant pages proceed. Pages skipped is a tracked metric, because it’s most of the cost saving.
Tier 1: vision extraction. GPT vision reads the prioritised pages against a per-document-type schema and returns ExtractionField records: typed values with confidences and page citations. The static instruction block is prompt-cached. Images are pre-processed once at ingestion (deskewed, auto-rotated, capped on the long edge, greyscaled where safe) and stored with a TTL, referenced by URI rather than passed inline through the agent graph.
Tier 2: OCR fallback. Azure Document Intelligence runs only on a trigger: mean extraction confidence below a threshold, handwriting over roughly a third of the page, or two consecutive schema-validation failures. The fallback rate is dashboarded with an alert if it exceeds a few percent in a week. If it climbs, either the scans changed or the prompts regressed, and both need a human.
The original pipeline ran OCR on every page and sent text to the model. Reversing it (vision by default, OCR when vision admits it’s struggling) cut cost per document and improved extraction on layouts where text order loses the meaning: tables, selection marks, multi-column statements.
Where the model is not allowed
Form detection and resolution are pure functions. A handful of detectors look for phrases in the OCR text; a deterministic resolver picks the form type, or multi if more than one is present, or none. This decision is authoritative: it overrides whatever the model believes it is looking at, and it selects which analyser runs. A guardrail assertion in the safety suite checks that the override holds.
One form type is read entirely without a model. Selection marks, polygon geometry and label-anchor proximity from the OCR layer resolve its checkboxes and its two free fields deterministically. The other hand-marked forms are read by vision under an “observe only” prompt that extracts what is ticked and nothing about what it means.
When several forms are present, their analysers run concurrently and the results are merged with a record of which forms were found. This was the first place the async wrappers paid for themselves.
Determinism
Every model call on the batch path runs at zero temperature with a fixed seed, top_p one, no penalties, JSON mode on, and provider-unsupported parameters dropped rather than errored. This does not make a vision model deterministic. It makes it repeatable enough that a golden-set regression is a signal rather than noise, and it makes the parameters a versioned, logged fact rather than a default nobody checked.
What was hard
Vision through the model-abstraction layer was unproven at batch scale and needed real work on image encoding, structured output and retry interaction. And the shared statement-transaction extractor, which reads a statement once into a normalised transaction list that every downstream rule consumes: depends on whether merchant category, channel and country are printed or have to be inferred, which turned out to be the single biggest open question in the flags design. See the investigation engines.