Three green checks that tested nothing

I finished the first milestone of my practice project today. The dashboard I’ve been building runs on a mini-PC in my house, and this evening I signed into it with Google from my phone, on mobile data, with the house wifi turned off. The page came back reading “You are Aaron, role manager”. That’s the whole milestone: one authenticated request travelling the entire pipe, from a device outside my network to a container on a box under my desk.

The deployment itself took an afternoon and was mostly unremarkable. What I want to write down is the part that unsettled me. Three separate checks came back green during that afternoon while testing nothing at all, and I believed two of them out loud before we caught it.

I wrote about a sign-in that took two days to fix a few days ago. That story ended on a line I thought was the lesson: the automated tests were green and a human clicking a button found the bug. This is the sequel, and it’s the worse version. That time the application lied to me. This time the tests did, and tests are the thing I’d normally trust to catch a lying application.

The check that tested a database that wasn’t there

The app needs a Postgres password. My agent generated one with base64, which produces a string of letters, digits, and two other characters: + and /.

The database connection string looks like this:

postgres://minim:PASSWORD@postgres:5432/minim

It’s a URL. A / inside the password half of a URL ends that section early, and the whole string stops meaning what you wanted it to mean. The Docker config builds this line by pasting the password into the middle of it, so a password with the wrong character in it produces a connection string that doesn’t parse.

We caught that one by reading, before it caused any damage. The part worth writing down is what happened next.

To confirm the fix, my agent ran the obvious command:

docker compose exec postgres psql -h 127.0.0.1 -U minim -d minim

Translated: go into the database container, connect to the database with this password, tell me if it works. It printed AUTH_OK. My agent reported the password fixed, and I read that and believed it.

It was wrong. The official Postgres container ships with a rule that trusts any connection arriving from inside itself. 127.0.0.1 from within that container means “me, talking to me”, and the rule says don’t bother checking a password for those. The command succeeds with the correct password. It also succeeds with the wrong password, and with no password. It had just reported success against a database that was still using the old password entirely.

There were two mistakes stacked here, and the second one is the one I’d have missed forever. Postgres only reads its password setting the first time it creates a database. Change the setting afterwards and restart, and nothing happens; the database keeps the password it was born with. My agent had tried to delete the old data directory to force a fresh start, that deletion had failed with a permissions error, and the “check” that was supposed to catch the failure was a command structurally incapable of catching it.

The real test took thirty seconds and looked like this: open a tunnel from my laptop to the database on the mini-PC, and connect from outside the container. That connection is subject to the password rule. It failed immediately, which is how we found out the truth.

The three checks that all avoid the database

The password bug bothers me for a reason that has nothing to do with passwords.

While the connection string was broken, I could ask the running application three questions, and it answered all three:

GET /api/healthz   ->  200 {"status":"ok"}
GET /api/me        ->  401
GET /              ->  the app's HTML page

Those are the checks I’d been using all afternoon to confirm the deployment was working. Every one of them passed. Every one of them would have kept passing with the database connection completely broken, because not one of them talks to the database.

healthz returns a hardcoded “ok”. /api/me returns 401 for anyone without a session, and it decides that before it needs to look anyone up. The homepage is a static file. My entire smoke test was three questions carefully arranged, by accident, to avoid the one component most likely to be misconfigured.

That’s the shape I keep finding. The check runs. The check passes. The check’s coverage has quietly shrunk to something that no longer includes the thing you’re worried about, and a passing result looks identical either way.

The migration that would have run on the wrong database

The third one had the largest blast radius, and we caught it by luck rather than by process.

The mini-PC has no Node.js on it, and installing one would have meant more software to keep patched on a box I want to stay boring. So the plan was to run the database migrations from my laptop, pointed at the mini-PC through an SSH tunnel. You set an environment variable naming the target database, you run the migration tool, it connects where you told it.

The tool printed this line on startup:

injected env (11) from ..\..\.env

It had loaded my local .env file and used its DATABASE_URL in place of the one I’d set. My local .env points at the Postgres running on my laptop. Had the migration proceeded, it would have migrated the wrong database, on the wrong machine, and reported success.

Nothing about that failure mode announces itself. A migration tool that says “applied 3 migrations” gives you the same message whichever database it hit. The only reason we noticed is that the tool then hung for unrelated reasons, and checking why sent us looking at that line.

The fix was to stop using a variable name the tool could clobber. I gave it a name that doesn’t appear in any .env file, and the ambiguity disappeared.

An error message that means two different things

One more, because it cost the most time and it’s the same disease in different clothing.

Before any of this I had to give the agent unattended SSH access to the mini-PC. The attempt failed with:

Permission denied (publickey)

The obvious reading is “this machine doesn’t recognise your key”. That reading sent us down a path of reinstalling the key, and I was told at one point that my earlier key installation had failed. It hadn’t. The key had been installed correctly the whole time.

My personal SSH key is protected by a passphrase. The agent ran ssh in a mode that forbids interactive prompts, because it can’t type a passphrase into a prompt nobody’s watching. In that mode, an encrypted key it cannot unlock produces Permission denied (publickey), the same message you get when the far end has never seen your key. A key that was never authorised and a key that can’t be decrypted right now are two different problems, and ssh reports them with one sentence.

You can’t diagnose from a signal that can’t distinguish the cases. We stopped trying and made a dedicated key with no passphrase, scoped to that one machine, which removed the ambiguity rather than reasoning through it.

What I’m taking from this

A check is a claim about the world, and the claim is usually smaller than you think. “The app is up” was really “three endpoints that avoid the database are up”. Before trusting a green result, I now ask what it would take for this check to pass while the thing I care about is broken. If I can answer that quickly, the check isn’t doing the job I hired it for.

Test a credential from where a real client sits. Connecting to a database from inside its own container skips the security rules that a real connection meets. The same shape applies to anything with a permissions layer.

Configuration tools have opinions about precedence, and they don’t always tell you. A tool reading an env file over your explicit setting is a normal design choice. Combined with a local database and a remote one that share a variable name, it’s a way to modify the wrong system and be told it went well.

Prefer removing an ambiguity to reasoning through it. Two of these cost time because a signal meant two things. Neither needed a cleverer diagnosis; both needed a setup where the signal could only mean one thing.

Watch what your agent claims, not just what it does. I direct AI agents through this build and review what they produce. Twice today I was handed a confident conclusion that was wrong, and both times the agent caught it and corrected itself before it caused damage. That’s the loop working. It also means the review has to include the verification, not only the code. An agent that runs a meaningless check and reports success is harder to catch than one that writes a bug, because the output looks like exactly what you wanted.

The milestone is finished and merged. The pay calculations are next, with the old system watching over my shoulder, one yen at a time. I’ll be writing my checks differently.