Fail closed

The disputes system has to answer a question on every letter: is the person writing this the person on the account? Get it right and the colleague skips a chunk of investigation. Get it wrong in one direction and a legitimate customer waits while someone double-checks. Get it wrong in the other direction and account details have been confirmed to a stranger, which has a name in the regulations and a reporting clock.

Every check in these systems has that shape. Documentation “complete” when it isn’t means a debt is asserted collectable without proof. A vulnerability flag missed means a person in crisis got a standard letter. A fraud escalation not raised means a case that should have gone to a specialist went to the queue. In each case one error is an inconvenience and the other is an event.

The engineering consequence is simple to state and surprisingly hard to hold: every safety-critical check defaults to the non-passing state, and you have to earn your way to a pass.

Fail-closed is a set of rules, not a mood

It’s a set of rules, not a mood.

Missing input is review, not pass. If the warehouse returns null for a field the comparison needs, the comparison is review. If the document inventory can’t be fetched because the document store is having a bad afternoon, the case is needs review, with a diagnostic, never a pass on the checks that didn’t run, never a stalled queue.

Low confidence is review, not pass. An extraction under the confidence gate, a document-type prediction that isn’t sure, a match score in the grey band. The status is review and the colleague sees a highlighted region with a question, not a chip that says done.

A citation that doesn’t resolve is not a conclusion. The platform renders it as an unverified suggestion: visually different, excluded from the roll-up.

The roll-up is fixed and pessimistic. Any fail in a finding is a fail. Any review, or any missing evidence, is a review. Only the case where every comparison passed and nothing was missing is a pass. There is no “mostly passed”.

Anything the model matched needs a human tick. Customers sometimes enclose their own copy of an agreement. The system can recognise it and match it in as evidence for the documentation check. But a document the model matched, as opposed to one the index holds, requires an explicit colleague confirmation before it counts toward “complete”. The model can propose; it cannot complete.

Thresholds are tuned for the bad error first. The identity matcher has a false-accept target of zero and a false-reject ceiling of fifteen percent. Not “balanced”. The thresholds are set to hit the floor on false accepts, and then the false-reject rate is brought down as far as it will go without moving the floor.

The cost is a review queue, and the answer is not to loosen the defaults

Fail-closed has an obvious failure mode of its own: everything becomes review, the colleague’s queue fills with cases the system declined to decide, and you’ve built an expensive way of saying “I don’t know.”

This is real, and the answer is not to loosen the defaults. It’s three other things.

First, measure the review rate per component as a first-class metric, alongside the override rate, and treat a rising review rate as a defect to fix upstream (a better extractor, a resolved data question, a threshold the business finally supplied) rather than as a threshold to relax.

Second, make review specific. “Needs review” with no reason is useless. “Needs review: the address on the statement doesn’t match the current address on record, and no historical addresses are available to compare” is thirty seconds of a colleague’s time. The finding carries its missing evidence and its reasons; the case pack shows them; the review is scoped to the thing the system couldn’t decide.

Third, let the override data do the tuning. Every colleague confirmation and correction goes to a label store. If a component is producing reviews that colleagues confirm as passes ninety-eight times out of a hundred, that’s evidence for moving a threshold: evidence, with a number, reviewed weekly, applied as a versioned config change. Not a feeling on a Tuesday.

The system is allowed to say “I don’t know.” It’s a real output, with a real cost, tracked and driven down. What it is not allowed to do is guess in the direction that makes the number look good.

It is a property a codebase loses by accident, so it is tested

The part I’d stress to anyone building one of these: fail-closed is not a design choice you make once. It’s a property a codebase loses by accident: a new comparator that returns pass on null because that was the convenient default; a fallback path that skips a check and forgets to mark it skipped; a roll-up rewritten to be “less noisy”.

So it’s in the safety suite as executable assertions, merge-blocking: the documentation, verification and escalation checks under missing input must produce the non-passing state; the roll-up must be pessimistic; a model-matched document must not count without the tick. The suite runs beside the golden-set regression, and it’s the part I’d keep if I could keep only one thing.

Alter acts first because everything it does can be undone

Alter, the product I build outside work, has the opposite disposition. It acts first (files your thought, promotes something to today) and gives you one tap to undo. That’s the right call when every action is reversible and the cost of asking is losing the user’s attention.

The work systems have no reversible actions. Nothing they conclude can be untold to a stranger or unplaced from a law firm. So nothing is autonomous, everything carries evidence, and the default is no. Same principle, opposite setting: autonomy is bounded by reversibility. Where nothing can be undone, the system’s job is to make a human’s yes fast, cheap and well-informed, and to never say it for them.