A corruption alarm that latches goes silent exactly when it's right

If your agent keeps its conversations, memory or run history in SQLite, you probably have a health check. It runs PRAGMA quick_check every so often, sets a flag when something is wrong and shows a banner. It looks finished. Mine looked finished too, and in one month it got three things wrong. It confirmed damage using the connection that had just failed. Once it latched, it stopped saying anything. And when real damage finally arrived, it had kept no evidence of how it started.

I built this into Vodou, which stores every chat turn in gateway.db, a WAL-mode SQLite file that three processes share. The alarm is there because of 2026-08-15. That day the gateway could not save one message for about 46 hours, and nothing said so.

One verdict, three witnesses: the live handle, a fresh read-only connection, the sidecar inodes

The monitor lives in MCP-servers/Vodou-Console/src/db-health.ts. Every ten minutes it runs quick_check on the gateway’s long-lived connection. If that fails, it re-reads twice right away, with no sleep, because it runs on the event loop. If it still fails, it opens a new read-only connection, asks the same question and closes it. That second opinion can only downgrade a verdict. If it can’t be asked (no provider, or the open throws), the verdict stays where it was. Not being able to ask is not an all-clear.

On the same tick, and whether or not the check passed, it runs stat() on -wal and -shm and compares their inodes with what it saw first. At the first bad read, at every tenth one after that, and at any confirmed failure, it writes a forensic snapshot. The snapshot holds the inode and size of all three files, lsof for all three with the inode each file descriptor points at (capped at 2 seconds), and the live handle’s data_version, freelist count, page count and journal mode.

For the operator, this means three states instead of one: the file is damaged, our connection is confused about a healthy file, or our connection is writing into files that no longer have names. Each one needs a different fix, and the worst move is applying the fix for one to another.

quickCheckOnce re-read through the connection that had just failed

On 2026-08-30, /health said “CORRUPTION DETECTED … Writes will start failing and messages will be LOST”. Writes had landed 22 seconds earlier. Every check I ran out of process passed: quick_check ok, full integrity_check ok, FTS5 integrity-check clean, a MATCH returning 18,635 rows, 74,824 messages readable.

The confirm loop already knew one failed read doesn’t prove a damaged file. That’s why it re-read before latching. But it re-read through getDbHandle(), the same long-lived connection. A fault local to that handle, like a stale page in its cache or a WAL snapshot it couldn’t resolve, fails every attempt. So the loop could only ever confirm what one connection believed. Two log lines named different blob ids, and I first read that as proof of real damage. It fits a confused handle just as well.

I didn’t mute the field. The alarm was doing its job, but its evidence came from one connection. I shipped six tests. One proves the fresh connection downgrades. Five pin what must not change: two agreeing connections still latch, a fresh provider that throws doesn’t rescue the verdict, and a healthy read is never demoted. When I disabled the fix, exactly that one test failed.

Thirty bad reads before the damage, with no timestamps and no context

On 2026-09-02 the FTS shadow tables were actually damaged, the third time in a month. The investigation named every writer. It ruled out iCloud, sleep, a second SQLite copy, fork with an open handle, file watchers and cross-version FTS5 writes (a 45-second two-process repro came back clean). Then it stalled on the two facts that would have decided it. WHEN did it start? gateway.log had no timestamps, and the daemon log had rotated. WHO held the file at that moment? Thirty handle-local transients came before the damage, and each was logged with nothing next to it.

So every [db-health] line got an ISO timestamp, and the forensic snapshot above got written. It failed on its first real day. The module is ESM, the snapshot called require(), and it threw require is not defined at both corruption events on 09-02, the only two moments it existed for. The unit tests never took the path that shells out to lsof.

if (state.ok) kept 25 failing ticks quiet for 4.5 hours

The fourth incident, on 2026-09-04, was the first one I could name. Docker Desktop had the repo shared into its Linux VM. A SQLite inside the VM opened gateway.db. It couldn’t see the host’s POSIX locks or its memory-mapped WAL index, so it decided it was the only connection. It checkpointed, and when it closed it deleted -wal and -shm. Three host processes kept writing into unlinked files. The gateway’s freelist said 803 pages while the file on disk said 742: two WAL worlds on top of one main file. When two of those processes exited cleanly, they checkpointed an orphaned WAL into a main file that had moved on. That caused the real damage: 100 problem lines in gateway_messages_fts_data, recovered in full from a backup taken before the damage.

The alarm did fire. It fired once. The logging sat inside if (state.ok), so only the move from ok to not-ok produced a line. The monitor then spent 25 ticks over 4.5 hours in that branch without a word. The one full integrity_check of the incident failed into the same silence. The verdict even changed while it was latched, from 2nd reference to page 56933 to Rowid 687194767425 out of order, and neither message was snapshotted. From the log, a quiet latch looks exactly like a dead monitor, and that’s how it was read.

Now it speaks on the first failure, again whenever the verdict text changes, and otherwise every 30 minutes. Each source (quick check, full check) has its own throttle, and a latch clock answers “how long has this been true?” The inode watch dated the swap at 07:02. The first failing check came at 07:10, eight minutes later.

A health verdict is no wider than the connection that produced it

The failure class, stated so you can check it in your own code: a corruption verdict must be confirmed by a connection that did not produce the failure, a latched verdict must keep writing when its content or age changes, and evidence about file identity must be captured on the tick that first fails. Any one of these is either true or false in your code.

There’s a WAL-specific corollary too. -wal and -shm are removed only when the last connection closes. If your process holds a connection open and the inode of either one changes, someone who couldn’t see your locks removed them. That can’t happen innocently. It also means a graceful shutdown of that process is dangerous: a clean close checkpoints the orphaned WAL. The fix is to kill it hard and restart, not to run your repair script. The repair script would throw away rows that only exist in the live WAL.

Five minutes against your own WAL-mode database

Point this at the database your long-running server keeps open.

DB=/path/to/app.db
PID=$(pgrep -f 'your-server' | head -1)

# Inodes on disk right now (Linux first, macOS fallback)
stat -c '%i %n' "$DB-wal" "$DB-shm" 2>/dev/null || stat -f '%i %N' "$DB-wal" "$DB-shm"

# Inodes the live process actually holds (NODE column)
lsof -p "$PID" 2>/dev/null | awk '/-(wal|shm)( \(deleted\))?$/ {print $8, $9, $10}'

# A second opinion from a connection that has never failed
sqlite3 "file:$DB?mode=ro" 'PRAGMA quick_check;'

Passing: the NODE numbers from lsof match stat, and quick_check prints ok. Failing: a different number, (deleted), or stat reporting no such file while lsof still lists one. Any of those means the process is stranded, and a graceful stop will make it worse. Use mode=ro on purpose. A bare path to the wrong file makes sqlite3 create an empty database and report it healthy.

Next, compare that fresh verdict with what your health endpoint says right now. If the endpoint says corrupt and the read-only connection says ok, your alarm is reporting one handle’s view. Last, write one test: stub the check to fail with 2nd reference to page 56933, tick, switch the message to Rowid 687194767425 out of order, tick again, then advance a fake clock 31 minutes and tick a third time. Count the log lines. If you get one, your alarm goes silent after latching, just like mine did.

The Hermes reports re-run the check but never question which connection ran it

The Hermes agent project has hit the same class in public, and their fixes are good. PR #41795 found that an integrity check was skipped for the whole process lifetime after the first connect. A long-lived writer kept checkpointing a torn file and made roughly 20 quarantine copies in 10 minutes. PR #100263 and issue #100227 show a fixed-string FTS probe that missed corruption in one segment and printed “no repair needed” while every real append failed. PR #99652 describes treating a generic SQLITE_CORRUPT as FTS-only, which let a gateway serve for about 10 hours while every write was lost.

Those fixes ask whether the check runs often enough and is sensitive enough. None of them ask whether the connection running the check is the thing that’s broken, or whether the alarm is still talking an hour later. Issue #100896 is the closest. It reports four incidents in five weeks with a gateway and a dashboard writing the same WAL database, plus a warning that fired 7 minutes before onset. I saw 8 minutes between the inode swap and the first failed check. That early window is where the evidence lives, and a check that only looks at page contents can’t see it.

Nothing here repairs anything, and the inode watch has a blind spot

This is detection only. The banner tells a person, and the person restarts or restores. I chose that deliberately: recovery written for a corruption whose cause you don’t know can hide the one signal that would identify it. The inode watch also arms the first time it sees a sidecar file. A swap that happens before this process has seen -wal exist goes unnoticed. And a snapshot of a healthy file still costs a full integrity_check, so it’s rate-limited and can miss the exact tick where things turn.

If your agent’s memory and history live in a local SQLite file, vodou.ai ships this monitor. A second connection has to agree before it calls the file damaged, and it records who held the file when things first went wrong.