The only red run was the first one

The pull request for the second milestone went green this morning: 2 minutes 35 seconds, sixteen steps, a real Postgres, a real signed session cookie, the old system’s pay logic diffed against the new one under two timezones. I opened the run history to see how far back that record went. Twenty-three runs since the 28th of August. One failure, on the first day, twenty seconds in. Every run since has passed.

That record covers a week when nobody could sign in, and a fortnight when the pay calculation disagreed with the system it was copying. CI was green through both. I don’t think it lied at any point. I think “green” was making a different claim each time, and I want to write down what each claim was, because I’ve caught myself reading the badge as a verdict when it’s a list.

What a run does

Continuous integration, for anyone reading this outside the trade, is a script that a hosting service runs against every change you push. Mine is a GitHub Actions workflow, one file, forty lines. It rents a Linux machine for a few minutes, checks out the code, and works down a list. Here’s today’s list with how long each step took:

install dependencies (frozen lockfile)    14s
lint                                       4s
typecheck                                 30s
apply database migrations                  2s
seed the demo data                         2s
unit tests, five packages                 24s
API tests over HTTP, real Postgres        26s
rebuild the API contract, fail on drift   22s
build the domain package                   2s
oracle sweep, Tokyo                        4s
oracle sweep, UTC                          3s

The database is the part I’d point a beginner at. The workflow asks GitHub for a “service container”: a Postgres 17 running beside the job, empty, thrown away afterwards. Every run applies all six migrations to it from nothing. That means CI is the one place that proves the migration chain works on a clean machine, which matters more than it sounds. On my laptop the database already exists, so a migration that only works because of what came before would pass there and fail on a fresh install. This week an agent hand-edited a migration journal to get my local database back in step, and I had no way to tell whether the result was clean. CI applied all six from scratch and told me.

The red run

The first run ever failed at the install step:

ERR_PNPM_IGNORED_BUILDS  Ignored build scripts: esbuild@0.18.20, esbuild@0.25.12, unrs-resolver@1.12.2

pnpm, the package manager, has a rule since version 10: a dependency that wants to run a script during install doesn’t get to unless you’ve approved it by name. esbuild needs one to download the right binary for the machine. The rule exists because install scripts are how a hijacked package runs code on your computer. On my laptop I’d clicked through the approval prompt without noticing. On a fresh machine with no one to click, pnpm refused.

There were two ways to make that red go away. The honest one is a list in the workspace file naming the packages you trust to run scripts, and that’s what went in. The tempting one appeared a few days later in the Docker build, where an agent added a fallback: if the install fails, approve every script and try again. That line made the image build succeed. A security review in the following week cut it, and the reasoning is worth keeping. The install can fail for a second reason, a lockfile that no longer matches the package list, which is the exact thing --frozen-lockfile exists to catch. The fallback turned that failure into a quiet success too, and it approved every install script from every package while it was at it. Two safety checks disabled to make one red line go green.

The list has needed two new entries this milestone: a compiler the new test runner uses, and a telemetry script that ships inside the API documentation viewer. Both times a fresh run on CI was what noticed. The gate is loud, and I’ve decided loud is the point.

Green through a broken sign-in

The first milestone ended with two days of a Google sign-in that wouldn’t complete. I wrote that up earlier in this series. The detail that belongs here: CI was green for every one of those commits, and it couldn’t have been anything else. The API tests mocked the auth library, because the library couldn’t load under the test runner we had. A mocked sign-in passes whatever the real one does.

So for the whole of the first milestone, the green badge said: the code compiles, the linter is happy, and a stand-in for the auth library behaves like a stand-in. True, every time. Not the claim I was reading it as.

That gap closed in the second milestone. The tests moved to a runner that can load the real library, the four mock files went, and the API tests in CI now write a session row into the service Postgres, sign a cookie the same way the library signs it, and send that cookie at the real guard. When the run says the manager route rejects a teacher, the guard that will reject them in production is the one that ran. The badge’s claim grew, and the badge looks identical.

Green through a wrong port

The second milestone I’ll write up another time was the pay calculation: the old system’s logic loaded into a sandbox, the new one written from scratch, both fed thousands of generated months and diffed to the yen. That port was wrong for most of its life. It had to be. It was built one rule at a time, and each rule was wrong until it wasn’t.

CI stayed green throughout, and this time the design was deliberate. The workflow doesn’t check that the two systems agree. It checks a list of fixture classes that already agree, committed to the repository, and fails if any of them stops agreeing. The list may only grow. Promoting a class onto it is a commit a human makes on purpose. So the claim the badge made during that fortnight was narrow and honest: nothing that matched yesterday has stopped matching today. The classes still being worked on weren’t on the list, and their disagreement was expected, so it didn’t fail anything.

I like this shape more the longer I look at it. A gate that demanded full agreement from day one would have been red for two weeks and everyone would have stopped reading it. A floor that only rises is green from the first commit and still means something on the last.

The two timezone runs come from the same milestone. A date helper once passed under UTC and failed under Tokyo, because it built a date from parts and let the machine’s clock decide which day it was. The old system’s own test script runs everything twice for the same reason. Mine does too, and there’s a footnote: the shell on my Windows laptop doesn’t pass a timezone setting through to Node, so the two-timezone sweep is something only CI runs without ceremony. The badge covers a case my desk can’t.

Twenty-two seconds to check a document against itself

One step rebuilds the API, has it write out the description of its own routes, regenerates the TypeScript types from that, and fails if the result differs from what’s committed. I wrote about this gate in the last post, because it was green while the committed document was wrong: two routes described as returning 200 that return 201. The emitted copy and the committed copy agreed, because the same mistaken line produced both.

I’m keeping it in the list here because it’s the cleanest example of the thing this whole post is about. That step’s claim is “the document in the repo is the document the code emits”. It has never claimed the document is right. I heard the second claim because I wanted to.

What green has never meant here

The workflow doesn’t build the Docker images. When the container hardening landed (non-root user, production-only files, the fallback above cut), CI passed, and the passing proved nothing about the images, because no step touches them. The evidence for that change was a local build and a smoke test, written down by hand. Image builds in CI are a later milestone, and until then the badge on that pull request was green about other things.

It has never exercised the Google callback, the one path that broke both times someone clicked. It can’t. Google isn’t in the service container.

And one I’d have got wrong without a note in my own status file: when the linter broke locally and got fixed, CI was green before and after, because the files that broke it were never committed. A green run can’t confirm a fix to a problem the run never had. The proof of that fix was watching lint go red and then green on my laptop, and the note says so, because otherwise “CI green” would have been offered as evidence for something it never checked.

The lessons I’m keeping

A run is a list of claims, one per step. Read the step, not the badge. The badge is the same colour whether the step checks a mocked library or a real one.

Keep the loud gates loud. The only red run in twenty-three was an install gate doing its job. The fix that made it quiet also disabled the lockfile check, and it took a review to notice.

A floor beats a bar during a long port. Gate on a list of things that already pass and let it only grow. Green from the first day, honest on the last.

Put the variations you can’t run at your desk in CI. Two timezones, a database built from nothing every time. Those are the cases my laptop is worst at.

Green can’t vouch for a fix to a problem it never saw. If the failing input never reached CI, the fix’s evidence has to come from somewhere else, and the somewhere else should be written down.

The pull request is mergeable. Next comes the milestone most portfolio projects fake: proving, at the database and not just in the application, that a teacher cannot read another teacher’s pay. I’d like CI to be the place that proof runs.