Custodian of Records

Sending an account to a law firm is the most consequential thing a collections business does with it, and it is gated on paperwork. The right original documents have to exist in the document system. Each has to actually be what its index says it is. The key figures on them (balance, dates, names, account references) have to agree with the system of record. A rules engine decides which accounts go; the Custodian of Records team audits a sample every month to check the rules engine isn’t letting ineligible accounts through.

The audit is manual, from scratch, per account. This system does it for every account, and turns the reviewer’s job into confirming exceptions.

One evidenced verdict per account, not a monthly sample

For each account in scope, an account verdict (pass, fail, or review) assembled from:

• A document inventory: the required set, resolved for this account; for each entry, found or missing, the document’s ID and indexed type, whether the file is readable, and which exception (if any) was applied.
• A type check per retrieved document: does its content match its indexed type? A transaction history filed as a final statement is the canonical failure.
• Extracted fields with full evidence: raw and normalised values, page, bounding region, OCR and extraction confidence, and a review flag.
• Reconciliation of those fields against the system of record, field by field, under approved comparison rules.
• Balance support: whether an acceptable document actually supports the balance being claimed. The two most common reasons a placement is rejected are an affidavit amount that doesn’t match, and no statement that supports the charge-off balance.

Every element is a citation away from the page it came from. The reviewer never opens a document to find out why the system said what it said.

Two review modes that must never be mixed

The pipeline is the platform’s four steps, instantiated for this domain:

 account list ──► work item (mode: audit | pre-legal), idempotent claim
                        │
              DocIntel pipeline (shared):
              inventory → page fetch → OCR → doc-type classifier → field extractor
                        │
              Parallel:  presence + exceptions ∥ reconcile ∥ balance support
                        │
              VerdictAssembler → warehouse → reviewer queue (exceptions only)

Two orchestrators, not one: an audit orchestrator that only confirms documents exist and are the right type, and a pre-legal orchestrator that runs the full reconciliation. They are separate configurations that must never be mixed, because the source material for this project explicitly warns that mixing them produces false rejections. That constraint became a first-class pipeline parameter, and it’s the most important design decision in the project. Two review modes, never mixed.

The required-document set turned out to be sparse, not a cube

The required-document set is resolved on three axes, then exceptions applied. Product, seller or pool, and jurisdiction. The first cut modelled this as a dense cube. The production rules export, when it arrived, showed the real structure was sparse: one axis only matters for one document type, another only for one seller family, and everything else resolves state-wide. The config became a state map with a small overlay, effective-dated, mirrored from the legal rules engine and validated against it, not a live dependency on it. The required-document model.

Account-level and pool-level validation are separate paths. Some documents belong to a purchased pool, not an account; they are validated once per pool and looked up by pool ID, and an account inherits the result. Except for sellers who require coverage of every entity in a portfolio, which is an explicit config flag that sends the account down the per-account path anyway.

No tolerance is ever inferred. A balance either matches or it doesn’t. Any “close enough” is a versioned config row approved by the business and compliance. This came straight from the non-goals and I have kept it absolute.

Exceptions are config rows, never model guesses. “This seller’s accounts have no statements” is a row in an exception matrix, not something the model is allowed to notice.

Historical rejections became the regression suite. The team’s real, explained rejection examples, the ones that motivated the project, seed the golden set, each carrying the expected outcome and the business explanation. Every one must reproduce.

Nearly complete; two use cases remain

Nearly complete. Presence, type classification, extraction, reconciliation and balance support are built on the shared platform and running over real accounts. Two use cases remain: template validation for a class of terms-and-conditions documents, which has concrete accept/reject criteria but is waiting on the approved-template corpus from its business owner; and merged-package completeness, the page-level check that a compiled evidence package is complete, internally consistent and about one account.

The rest of the programme (a set of scheduled, rule-based reporting and prioritisation jobs with no model in them) sits with a different engineer and isn’t mine to write up.

Presence is not proof, and the labels arrive late

Presence is not proof. A record labelled “final statement” existing in the index is not evidence a final statement exists. The content has to match the label, the statement has to fall in a date window anchored on the account’s own dates, and the figures on it have to reconcile. Each layer catches failures the previous one can’t see. The essay.

Labels that arrive late. Statements are not renamed to their roles until a downstream process runs; validate ahead of it and the relevant statement has to be inferred from the closest date. Account numbers can change around charge-off, so the account can legitimately have more than one.

Grounding the model in a moving target. Three separate business artefacts (a validation grid, a rules-engine export, a criteria deck) arrived over three months, each partially resolving open questions and partially reopening them. The spec has a versioned changelog for exactly this reason; the config it drives has one too.

Pages

• Two review modes, never mixed: audit vs pre-legal, and why they can’t share a config.
• The required-document model: product, seller, jurisdiction, exceptions, and the sparse config that came out of the real data.
• Balance support and misclassification: the two rejection drivers, and how the model is kept out of the decision.