My release gate passed whenever it couldn't read the database it checked

My release gate was failing for a reason that had nothing to do with the release. My quick fix was to shadow the database it read with an empty, read-only file. That would have turned a false failure into a false pass, so I didn’t ship it. Then I read the code properly and found something worse. The gate didn’t need my stub to pass vacuously. It already did that every time the file was missing.

The bug class: a check that reads state it never declared keeps “passing” when that state is absent, empty or wrong, and the pass means nothing.

A gate that takes one database as an argument and quietly reads a second

My gate is a bash script that takes the clean database shipped to fresh installs as its argument, and one of its checks also reads the developer’s live database through a hardcoded path. Here is that check:

if [ -f "$DEV_DB" ] && [ "$TREE_ROOT" = "$ROOT" ]; then
  act=$(sqlite3 "$DEV_DB" "SELECT 1 FROM mcp_servers WHERE name = '$n' AND active = 1 ..." 2>/dev/null)
  [ "$act" = "1" ] && UNSHIPPED="$UNSHIPPED $n"
fi

It has two ways to pass without looking.

The file is missing. [ -f "$DEV_DB" ] is false, so the whole block is skipped and nothing is said about it. Any run on a machine without that file passed this check without checking anything. That was true long before I thought of a stub. I wrote the guard to avoid a crash, and what it actually did was switch the check off.

The file is empty or has the wrong schema. A 0-byte file passes -f. sqlite3 prints Error: in prepare, no such table: mcp_servers and exits 1. The assignment act=$(...) receives that 1, and the next line never looks at it. act is empty, nothing gets flagged, and the script prints ✅.

It’s tempting to blame 2>/dev/null, and I did at first. That’s the wrong culprit. Remove the redirect and you get an error message in the log, and the gate still passes. The defect is that the exit status gets thrown away. If you copy just one fix from this post, it’s that one.

The fixed block: “could not look” gets its own exit code

Invariant: a check that reads a resource must fail when that resource is missing, empty or has the wrong schema. “Found no problems” and “could not look” must never share an exit code.

Here is what that block has to be. At the time of writing, my tree still has the old one:

DEV_DB="${DEV_DB:-$ROOT/app.db}"          # declared, and overridable

if [ "$TREE_ROOT" = "$ROOT" ]; then
  [ -s "$DEV_DB" ] || { echo "cannot look: $DEV_DB missing or empty"; exit 2; }

  has=$(sqlite3 -readonly "$DEV_DB" \
    "SELECT 1 FROM sqlite_master WHERE type='table' AND name='mcp_servers'") \
    || { echo "cannot look: sqlite3 failed on $DEV_DB"; exit 2; }
  [ "$has" = "1" ] || { echo "cannot look: no mcp_servers table in $DEV_DB"; exit 2; }

  for n in $CANDIDATES; do
    act=$(sqlite3 -readonly "$DEV_DB" \
      "SELECT 1 FROM mcp_servers WHERE name = '$n' AND active = 1 LIMIT 1") \
      || { echo "cannot look: query failed for $n"; exit 2; }
    [ "$act" = "1" ] && UNSHIPPED="$UNSHIPPED $n"
  done
fi

Exit 0 means it looked and found nothing. Exit 1 means it looked and found something. Exit 2 means it couldn’t look. CI can treat 1 and 2 differently, but neither of them is green. The table check matters because a 0-byte file is a valid empty SQLite database: SELECT ... FROM sqlite_master on one returns nothing and exits 0, so the query only fails later, at the table. -readonly also stops sqlite3 from creating an empty file at a mistyped path. Pointed at a missing file, it exits 1 with unable to open database instead.

The same shape exists outside SQLite. A CI step runs version=$(jq -r .version package.json 2>/dev/null) and then compares $version against a tag. If the file is missing or isn’t valid JSON, jq exits non-zero, $version is empty, the comparison takes whichever branch an empty string lands in, and nobody reads the status. You get the same fix: check the status and use a distinct exit code.

Test your own gate: trace it, grep it, then break its inputs on a copy

Find what it reads, two ways. A trace shows what one run touched:

strace -f -e trace=%file -o /tmp/gate.trace ./release-gate.sh     # Linux
sudo fs_usage -w -f filesys bash sh sqlite3                       # macOS; run the gate in another shell

Use %file, not just openat. A [ -f ] test is a stat, not an open, so an open-only trace never shows a file that was checked and found missing. fs_usage needs root, and System Integrity Protection can hide activity from some processes. Filter by process name, not file extension. If you filter to .db you’ll only find the files you already suspected.

A trace only covers the branches that run executed. My bad read sat behind [ "$TREE_ROOT" = "$ROOT" ]. A trace taken in a checkout where that condition was false would never show the database. So also read the source:

# hardcoded paths the arguments don't mention
grep -rnE '(\$HOME|\$ROOT|\$\{?[A-Z_]*DIR\}?|~)/[A-Za-z0-9_./-]+' scripts/ ci/

# query results captured with no exit-status check on the same line
grep -rnE '=\$\((sqlite3|psql|jq|curl)' scripts/ ci/ | grep -v '||'

A hit doesn’t prove a bug. It’s a line you have to read. Note that local x=$(cmd) hides the status even under set -e.

Break each input on a copy, never on the live file. Truncating a database a daemon has open in WAL mode can corrupt it or strand the WAL. Take a consistent copy and point the gate at it:

sqlite3 app.db ".backup '/tmp/gate-test.db'"
DEV_DB=/tmp/gate-test.db ./release-gate.sh; echo "exit=$?"         # control: real result, 0 or 1

rm -f /tmp/gate-test.db
DEV_DB=/tmp/gate-test.db ./release-gate.sh; echo "exit=$?"         # missing

: > /tmp/gate-test.db
DEV_DB=/tmp/gate-test.db ./release-gate.sh; echo "exit=$?"         # empty

rm -f /tmp/gate-test.db; sqlite3 /tmp/gate-test.db 'CREATE TABLE x(y)'
DEV_DB=/tmp/gate-test.db ./release-gate.sh; echo "exit=$?"         # wrong schema

A healthy gate prints a cannot look: line and exit=2 for each of the last three: missing or empty for the first two and no mcp_servers table for the third. A broken gate prints its success line and exit=0 for any of them. My original gate would have failed before the first command ran. The path was hardcoded, so DEV_DB= did nothing. If you can’t point your gate at a copy, count that as a failed result. A check you can only test by damaging the live data is a check nobody tests.

The rule: every resource a check reads should show up in its arguments or environment, and every check should exit with its own code when it couldn’t look.