Evaluation and AI safety
Golden sets with deterministic ground truth, hard floors that don't tune, an AI-safety suite that blocks merges, and the override flywheel that keeps both honest in production.
Two separate frameworks, deliberately. Evaluation asks whether the system is accurate. AI validation and safety asks whether it can be made to misbehave. Conflating them is how a pipeline with a great extraction score ships a prompt injection.
Evaluation
Golden sets. Version zero is about a hundred labelled cases per product, seeded from the real, explained failures that motivated the project, each carrying the expected outcome and the business explanation. Version one grows to about five hundred, stratified, with an adversarial slice: poor scans, multi-account files, merged packages, misindexed documents, exception rows. Labels are owned by subject-matter experts, secured, referenced by ID, and refreshed quarterly from production plus every override.
Component gates, tuned after a baseline, with three that do not tune:
| Component | Gate |
|---|---|
| Extraction | ≥ 97% typed · ≥ 90% handwritten, post-fallback |
| Classification | ≥ 95%; dispute-vs-complaint F1 ≥ 0.95 |
| Doc-type classification | ≥ 95% accuracy · ≥ 97% misclassification recall |
| Presence | ≥ 99% missing-document recall · ≤ 1% false-”present” |
| Verification | false-accept ≤ 0.1%, target 0 · false-reject ≤ 15% |
| Documentation completeness | false-”complete” ≤ 1% |
| Flags | vulnerability recall ≥ 98% · high-severity ≥ 97% · macro precision ≥ 85% |
| End-to-end verdict | false-pass ≈ 0 · overall ≥ 85% |
The bold rows are asymmetric. A false accept on verification is a data breach. A false “complete” asserts a debt is collectable without proof. A missed vulnerability flag is a person in crisis whose letter went to a standard queue. These have floors, not targets, and the thresholds upstream are tuned to hit the floor first and the false-reject rate second.
Judges. No LLM-as-judge for anything with deterministic ground truth: presence, comparison, completeness, classification against a governed list. An LLM judge, calibrated against expert ratings and never the same model as the generator, scores only the narrative case summary against a rubric.
Online. The production metric is the override rate per component: how often a reviewer corrects the system. A weekly review of override reasons drives the backlog. A weekly blind audit of confirmed cases catches the failure mode where the system is good enough that people stop checking: automation bias is treated as a measurable risk, not a training issue. Drift monitors watch the document-type mix, OCR confidence, and new templates.
AI validation and safety
Versioned with the pipeline, merge-blocking in CI, jointly owned with a separate validation function. Four suites:
- Injection. Correspondence is adversarial by nature: some of it is written by people who would like the system to conclude something false, and some of it is written by professionals who know what a language model is. Resistance on the deterministic invariants is gated at 100%: no letter content can change which form type was resolved, which list a classification is drawn from, or which action the decision table selected.
- Regression. The golden set, replayed on every change to a prompt, a model version, or a config file.
- Hallucination. Citation resolvability at 100%: every cited page, region and reference must exist and contain what it is cited for.
- Guardrail assertions. The invariants turned into executable tests: values are drawn only from the governed lists; the deterministic form type overrides the model’s; the decision table, not the model, chose the action; no cross-account leakage in session state; and fail-closed behaviour on documentation, verification and escalation under missing input.
A red-team loop feeds new attacks into the adversarial slice, so the injection suite grows with the threat rather than the release.
Why this is on the platform
Because the second product got it for free. The harness skeleton (golden-set loader, per-component scorecard, gate configuration, the guardrail assertion framework, cost telemetry per agent per case) is shared code. The records product added its own labelled accounts and its own gates and inherited the rest, including the safety suites. Evaluation as a platform concern is the difference between “we tested it” and “we test it”.