Evals & cost control

Every prompt in Alter is a versioned, tested component. Each one (recognition, brief, build, think, explore, briefing, compilation, the onboarding read) has a golden eval suite that runs as a CLI runner, and every run carries a dollar cap that’s asserted at the end. If a prompt change makes a suite more expensive than its budget, that’s a failure, same as a wrong answer.

At runtime, cost is bounded per user: a daily LLM spend cap with graceful degradation; recognition defers and the UI says so, rather than the product silently breaking. Every LLM call is attributed to one of twelve enumerated purposes, so when spend moves, it’s traceable to a feature rather than a mystery.

Per-category model routing is live in production: cheap, fast models for classification-shaped work, the expensive ones reserved for reasoning, and specific purposes deliberately pinned to a top-tier model where the quality difference showed up in the goldens. On the benchmark harness (which scores output quality with an LLM judge, not vibes) routing cut blended inference cost by about 65%, with a CI assertion that it stays above 50%.

Two things I’d tell anyone building this:

Measure the suites, don’t estimate them. For months my documented cost-per-run was an early guess that had drifted from reality by roughly 3×. The number only became true once the runner printed it and the assertion used it. An eval budget you haven’t measured is the same species of thing as a backup you’ve never restored.

Some assertions are unstable, and that’s information. A handful of the recognition suite’s checks pass and fail run to run on identical inputs. The temptation is to pin them until they’re green. The discipline is the opposite: an assertion that flickers is telling you the class of behaviour is under-specified, and the fix belongs in the prompt or the contract, not in the test.