Presence is not proof

Before a collections account can be placed with a law firm, the paperwork has to be right: the original documents have to exist, they have to be what they’re labelled as, and the figures on them have to agree with the system of record. A rules engine decides which accounts qualify. A team audits a sample of its output every month, by hand, to catch what it lets through.

The system I built to do that audit started, as these things do, with a simple plan: check the document index for the required types and report what’s missing. That’s a set difference. It took an afternoon.

It was also the least useful of the four things the system now does, because a document being present in an index is the weakest possible evidence that the document exists.

”Exists” has four layers, and each catches what the one before cannot

Here is what I mean, in order of how hard each is to check and how much each one catches.

1. A record with the right label is in the index. This is what the afternoon’s set difference verifies. It catches the account with no statement at all, which is real and worth catching. It says nothing about what’s in the file. The canonical failure, a transaction history that someone indexed as a final statement years ago in a hurry, passes this layer perfectly.

2. The content matches the label. Open the file. Is this actually a final statement? Title, layout, seller identity, table structure, signature blocks. A vision model predicts the type from a fixed list with a confidence and a page; deterministic code compares that to the index and decides whether the gap is a mismatch. This layer catches misindexing, which is the failure the team’s own rejection history said was common, and which no amount of querying the index will ever surface.

3. The document is valid for this account, now. A statement is not a charge-off statement because it says “statement” on it. It’s one because its date falls in a window anchored on the account’s own charge-off date. An affidavit is valid in this jurisdiction and not that one. A document that belongs to a purchased pool rather than an account has to be looked up by pool, not searched for per account. This layer is pure config (product, seller, jurisdiction, dates, exceptions) and it catches the document that is genuinely a final statement and genuinely irrelevant.

4. What it says agrees with the system of record. Extract the balance, the dates, the names, the account references; compare each against the authoritative source under an approved rule. Exact, or a tolerance a human signed off on, or a documented exception. This is the only layer that catches the failure that actually gets placements rejected most: a real, correctly labelled, correctly dated statement whose balance doesn’t support the claimed amount.

Four questions. Four different tools: a set comparison, a classifier plus a deterministic mismatch rule, a dated rule config, a reconciliation engine. Each layer assumes the previous one and catches something it can’t. The platform is these four layers, generalised, with the model allowed in exactly one of them.

Running a layer you do not need teaches reviewers to ignore you

Here is the part that took me longest to internalise. Not every question needs all four layers, and running layers you don’t need is worse than useless.

The monthly audit asks: is the rules engine letting through accounts with missing or misindexed documents? That’s layers one and two. If you run layer four on an audit and fail an account because a balance differs, you’ve failed it for a reason that has nothing to do with the question, and the reviewer learns to ignore your fails. The pre-legal check asks: is this account ready, now? That’s all four, and layer four is the point.

The two questions became two orchestrators that never share a run, with a merge-blocking test that an audit run can never emit a balance result. It’s the most important decision in the project and it comes straight from taking “exists” apart.

”The tests pass” has the same four layers

Once you see the layers you see them in every system that reports on itself.

The tests pass. Layer one: CI is green. Layer two: the tests that passed are the ones that exercise the change. Layer three: they ran against the current code, not a cached artefact. Layer four: what they assert is what the acceptance criteria actually said. The agents that build my own product will happily report layer one as if it were layer four, which is why the verifier re-derives “done” from the work item and the breaker has to produce runnable evidence, not an opinion. Nothing marks its own homework is this essay in a different domain.

The PR was reviewed. A record exists. Did the reviewer open the diff? Was it the right diff? Did they check it against what was asked?

The customer was verified. A row says so. By what? Against which record? Under which rule?

In each case the cheap check is the presence of a record, and the cheap check is the one systems drift towards, because it’s the one that can be done with a query. The expensive checks need the thing itself opened, dated, and compared. Every layer you skip is a class of failure you’ve decided to find out about later.

Treat layer one as an inventory, never a verdict

Build layer one in an afternoon, by all means. Then treat its output as an inventory, not a verdict, and be honest in the UI about which layer produced each conclusion. The moment a screen shows “present ✓” for a document nobody has opened, someone downstream will read it as “proven”, and the gap between those two words is where the rejected placements live.