Near miss: your deploy script ships your working tree, not a commit

This is a near miss. Nothing reached production. On 2026-07-09 I fixed a one-file bug in the admin releases form on app.vodou.ai. It was a frontend-only change to a React form. I almost shipped it with the repo’s deploy script, and that script would have put two unfinished billing files into production along with it.

What the script actually did

The script is scripts/deploy.sh in the app repo. I had treated it as “the way you deploy” and had stopped reading it. It runs rsync over all of backend/ from my laptop’s working directory. It doesn’t deploy a commit. It deploys whatever is sitting on disk.

That day git status showed create-checkout-session.php and webhook.php as modified. They were billing work in progress, uncommitted and untested. Running the script would have made them live in order to ship a change to an unrelated form.

So I didn’t run it. I built the frontend and synced only the build output directory that nginx serves. That was the right call for the wrong reason, though. I was being careful that one day. The script hadn’t changed, and it would do the same thing the next time I was in a hurry.

I had been wrong about what a deploy script is. I thought of it as transport. It’s policy: which code gets shipped, from where, and whatever side effects the author bolted on.

The deploy doesn’t know what a commit is

The rule the script broke is this: every file a deploy writes to the server is reproducible from one commit SHA. Syncing from the working tree breaks it on every run where anything is uncommitted, and you can’t see that in the deploy’s own output. In a dry run, webhook.php shows up as one ordinary transfer line among dozens. It looks exactly like the files you meant to send, because a deploy is supposed to send files that differ from production.

The ProcessWire write-up states the fix: build the deployable file list from an immutable commit, not from the current working directory. deploy_from_git builds that in. You deploy a SHA, and “dirty” is a mode you ask for explicitly, never the default.

die_if_dirty() has the right instinct but checks the whole tree

clubmate’s deploy script has die_if_dirty(): refuse to deploy from a dirty tree. That’s the right instinct at the wrong scope. It checks the whole tree. On 2026-07-09 it would have blocked my frontend deploy because of unrelated backend WIP. A guard that blocks the change you actually need to ship is the kind people learn to comment out, and then you have no guard at all.

A guard that only looks at the paths the deploy ships survives. It blocks exactly the case that’s dangerous and stays quiet about everything else:

git diff --quiet HEAD -- backend/ || { echo "backend/ has uncommitted changes; refusing"; exit 1; }

That one line would have stopped a backend deploy that day and let a frontend deploy through. git diff doesn’t see untracked files, so if a new file in that path would also ship, add a second line:

[ -z "$(git ls-files --others --exclude-standard -- backend/)" ] || { echo "backend/ has untracked files; refusing"; exit 1; }

That’s the principle I’d transfer to anything else: scope a guard to what the action touches. Tree-wide guards get disabled. Path-scoped ones stay in the script.

Check your own deploy in five minutes

This check compares what your deploy would send against a build of HEAD made from a clean export. It uses the same excludes your deploy uses, so it covers both source-synced deploys and build-output deploys. A build directory is gitignored and no commit contains it, so comparing it directly against HEAD would print every file. Here you compare a build against a build.

Set three values from your deploy script:

# Run from the repo root.
OUT=backend                          # the directory your deploy syncs (e.g. frontend/build)
BUILD='true'                         # your build command, run from the repo root (e.g. 'cd frontend && npm ci && npm run build')
EXCLUDES=deploy.exclude              # the file your deploy passes to rsync --exclude-from

# 1. Build the working tree the way your deploy does.
eval "$BUILD"

# 2. Build HEAD in a clean temp directory.
ref=$(mktemp -d)
git archive HEAD | tar -x -C "$ref"
(cd "$ref" && eval "$BUILD")

# 3. Diff build against build, with the deploy's own excludes.
rsync -rcni --delete --exclude-from="$EXCLUDES" "$OUT/" "$ref/$OUT/"

If your deploy has no exclude file, use whatever --exclude flags it passes, and nothing more. Don’t add your own. The point is to see what the deploy sees.

Pass: no output at all. Every file the deploy would send is byte-identical to what HEAD builds.

Fail: any line. Here are the three shapes:

>fc.T...... api/webhook.php
>f+++++++++ api/scratch.php
*deleting   api/old-endpoint.php

• >fc means a file that differs from HEAD: a modified file your deploy ships. This is my webhook.php.
• >f+++++++++ means a file that HEAD doesn’t have at all: an untracked file your deploy ships.
• *deleting means a file that HEAD has but your working tree doesn’t, because you deleted it locally and didn’t commit. If your deploy uses --delete, it removes that file from the server. If it doesn’t, the server keeps a copy that no longer matches anything. Either way the server diverges from the commit. It counts as a fail.

If this prints thousands of lines of node_modules/, vendor/, .venv/ or a cache directory, the check isn’t noisy. It’s telling you your deploy ships those directories, because it’s using your deploy’s excludes. The same goes for .env: if it shows up, your deploy sends it.

If your build isn’t reproducible (it embeds timestamps or absolute paths), you’ll see >fc lines on files you never touched. That’s a separate finding, and the fix is the same: deploy the output from $ref, not from your working tree.

Make the deploy name its commit

Passing once proves nothing about next week. The check that holds is the one the deploy enforces on itself. Every deploy prints the SHA it shipped, and the server says which SHA it’s serving:

sha=$(git rev-parse HEAD)
echo "$sha" > "$OUT/version"
echo "deploying $sha"
# ... your sync ...
[ "$(curl -fsS https://your-site/version)" = "$sha" ] || { echo "server is not serving $sha"; exit 1; }

Put the scoped guard above that, and a dirty deploy can’t happen quietly. Either the guard refuses, or /version names a commit you can check out and read.

My rule now: if a deploy can’t name the commit it shipped, it doesn’t ship.

---

A note on migrations. The same script also runs database migrations against the live billing database, usage_tracking.db, on every run. My first draft of this post argued that this was half the bug. I never checked whether any migration was pending that day, and most migration runners do nothing when none is. So I can’t show that it would have done anything. The only real danger I can point to is migration files coming from the dirty tree, and the commit invariant above already covers that. I still think Zeron’s handbook is right to make migrations a separate, manual step after the code deploy. But that’s a separate and weaker claim, and this near miss isn’t evidence for it.