What broke
The incidents and the lessons they left, including the one that changed how I decide a feature is "done."
Every honest project has a graveyard. Here’s Alter’s: not the typos, but the ones that taught something.
”Green” didn’t mean “working”
The defining lesson of the whole rebuild. Each service had its own passing tests, and the build was green, but none of that ever ran the composed system: the real services talking to each other over the network, real language models, streaming, the whole stack stood up together. So a feature could pass every check and still be broken the moment you actually used it. “Phase shipped (green)” quietly came to mean “works in isolation, unknown in reality.”
The fix was a rule, not a patch: prove it against the running stack, not the mocks (a full docker compose bring-up, exercising the live gateway and the real service-to-service contracts, because per-service unit tests structurally cannot).
The lesson has a second half I only learned much later: audit the gates themselves, on a schedule. A routine review of the CI configuration found that the end-to-end suite (twenty-nine paths, carefully written, all passing locally) was never actually being executed by the pipeline. A test that doesn’t run is worse than a test that doesn’t exist, because it buys you confidence you haven’t earned.
Tests that were green for the wrong reason
A close cousin, and the sharpest example of the above. Some database code was only ever tested against a fake database, so a query that named two columns the real table never had passed its tests and then crashed every single save (raw SQL verified only against a mocked ORM, not the live schema; every save-to-wiki 500’d on an INSERT into columns that didn’t exist). The lesson earned a standing rule: check raw SQL against the real schema, inside a transaction you roll back.
The same blind spot has a more exotic form. Postgres infers the type of each bind parameter from how it’s used, so a single parameter used both as an integer column value and inside a SUM() fails to even parse, with an “inconsistent types deduced” error. Against a mock, every test is green. Against the real database, the statement can never be prepared at all.
The empty value that wasn’t empty
My favourite, because it’s so small. A brand-new page had “empty” content by default, except the default wasn’t actually empty; it was one invisible byte. So every not-yet-written page crashed the moment the system tried to decrypt its “nothing” (a Postgres DEFAULT E'\\x' is a 1-byte value, not an empty one; decryption choked on it). A single character in a migration fixed it.
The model that said no
The one that doesn’t look like a bug at all. A prompt that had run correctly for weeks began coming back refused after a routine model change: not a wrong answer, not a malformed one, but a categorical decline of the entire task class. Nothing had changed on my side.
Two things came out of it. First, a diagnostic habit: on any unexpected LLM result, read the stop reason before reading the output, because a refusal and a bad answer look identical if you only inspect the text. Second, a structural fix: the prompt was reframed, and a detected refusal now falls back to a second, more capable model rather than failing the user.
The wider lesson is uncomfortable and permanent. Prompts are dependencies you don’t control. They don’t break loudly, they don’t break deterministically, and they can break because someone else shipped an improvement. Versioned prompts with golden suites are how you find out; a fallback path is how you survive finding out at 3am.
The check that was checking the wrong thing
Alter enforces a per-user daily spend cap, and it works. It also, for a while, ran on a set of background jobs for users whose subscriptions had lapsed, because the cap answers “has this person spent too much today?”, which is a completely different question from “is this person entitled to anything at all?”
Both guards existed. Both were correct. Neither was wrong in isolation, and the gap between them was invisible until I went looking at where the money was actually going. The rule I took from it: a cost check is not an entitlement check, and every background job needs to be asked which of the two it’s actually performing.
Failures that leaked out to the user
A few of these, all the same shape: something upstream broke and the ugliness reached the screen. A provider having a bad minute would relay its raw error text straight into the interface (a transient 529 overloaded_error shown verbatim), and long-running research streams would simply die after 30 seconds (the front-end proxy’s default timeout severing the connection mid-answer). Both fixed by putting a deliberate layer between the outside world and the user, retrying and sanitising provider errors, and configuring the stream to stay open.
The pattern across all of them: the bugs that hurt weren’t in the clever parts. They hid in the seams, between services, between a test and reality, between a default and its meaning, between two guards that were each individually correct. Which is why the work in launch readiness and observability is mostly about watching those seams.