# A corruption alarm that latches goes silent exactly when it's right > A SQLite health monitor needs a second connection, a latch that keeps talking, and evidence captured at the first bad read. Mine had none of the three. - Author: Chad Priest - Published: 2026-09-15 - Canonical URL: https://blog.vodou.ai/sqlite-corruption-alarm-latch-forensics/ - Tags: memory, observability, architecture, ai-agents --- If your agent keeps its conversations, memory or run history in SQLite, you probably have a health check. It runs `PRAGMA quick_check` every so often, sets a flag when something is wrong and shows a banner. It looks finished. Mine looked finished too, and in one month it got three things wrong. It confirmed damage using the connection that had just failed. Once it latched, it stopped saying anything. And when real damage finally arrived, it had kept no evidence of how it started. I built this into [Vodou](https://vodou.ai/register?utm_source=blog&utm_medium=feature&utm_campaign=db-health-a-forensic-snapshot-at-the-moment-gateway-db-first), which stores every chat turn in `gateway.db`, a WAL-mode SQLite file that three processes share. The alarm is there because of 2026-08-15. That day the gateway could not save one message for about 46 hours, and nothing said so. ## One verdict, three witnesses: the live handle, a fresh read-only connection, the sidecar inodes The monitor lives in `MCP-servers/Vodou-Console/src/db-health.ts`. Every ten minutes it runs `quick_check` on the gateway's long-lived connection. If that fails, it re-reads twice right away, with no sleep, because it runs on the event loop. If it still fails, it opens a new read-only connection, asks the same question and closes it. That second opinion can only downgrade a verdict. If it can't be asked (no provider, or the open throws), the verdict stays where it was. Not being able to ask is not an all-clear. On the same tick, and whether or not the check passed, it runs `stat()` on `-wal` and `-shm` and compares their inodes with what it saw first. At the first bad read, at every tenth one after that, and at any confirmed failure, it writes a forensic snapshot. The snapshot holds the inode and size of all three files, `lsof` for all three with the inode each file descriptor points at (capped at 2 seconds), and the live handle's `data_version`, freelist count, page count and journal mode. **Diagram: Who gets to say the file is damaged** Health tick: inode compare and quick_check on the live handle, re-read twice, then a fresh read-only connection decides between handle-local and confirmed corruption; stranded and confirmed both take a forensic snapshot ```text [10-minute tick] --> [stat -wal / -shm] [stat -wal / -shm] --inode moved--> [STRANDED (problem)] [10-minute tick] --> [quick_check on live handle] [quick_check on live handle] --fails--> [re-read x2] [re-read x2] --still fails--> [fresh read-only connection] [fresh read-only connection] --reads clean--> [HANDLE-LOCAL (fixed)] [fresh read-only connection] --also fails--> [CORRUPTION latched (problem)] [CORRUPTION latched (problem)] --> [forensic snapshot] [STRANDED (problem)] --> [forensic snapshot] notes: stat -wal / -shm: inode vs first sighting STRANDED: restart, do not repair fresh read-only connection: can only downgrade HANDLE-LOCAL: file fine, our view is not forensic snapshot: stat, lsof, pragmas Two connections have to agree before anything latches. ``` For the operator, this means three states instead of one: the file is damaged, our connection is confused about a healthy file, or our connection is writing into files that no longer have names. Each one needs a different fix, and the worst move is applying the fix for one to another. ## `quickCheckOnce` re-read through the connection that had just failed On 2026-08-30, `/health` said "CORRUPTION DETECTED ... Writes will start failing and messages will be LOST". Writes had landed 22 seconds earlier. Every check I ran out of process passed: `quick_check` ok, full `integrity_check` ok, FTS5 `integrity-check` clean, a MATCH returning 18,635 rows, 74,824 messages readable. The confirm loop already knew one failed read doesn't prove a damaged file. That's why it re-read before latching. But it re-read through `getDbHandle()`, the same long-lived connection. A fault local to that handle, like a stale page in its cache or a WAL snapshot it couldn't resolve, fails every attempt. So the loop could only ever confirm what one connection believed. Two log lines named different blob ids, and I first read that as proof of real damage. It fits a confused handle just as well. I didn't mute the field. The alarm was doing its job, but its evidence came from one connection. I shipped six tests. One proves the fresh connection downgrades. Five pin what must not change: two agreeing connections still latch, a fresh provider that throws doesn't rescue the verdict, and a healthy read is never demoted. When I disabled the fix, exactly that one test failed. ## Thirty bad reads before the damage, with no timestamps and no context On 2026-09-02 the FTS shadow tables were actually damaged, the third time in a month. The investigation named every writer. It ruled out iCloud, sleep, a second SQLite copy, fork with an open handle, file watchers and cross-version FTS5 writes (a 45-second two-process repro came back clean). Then it stalled on the two facts that would have decided it. WHEN did it start? `gateway.log` had no timestamps, and the daemon log had rotated. WHO held the file at that moment? Thirty handle-local transients came before the damage, and each was logged with nothing next to it. So every `[db-health]` line got an ISO timestamp, and the forensic snapshot above got written. It failed on its first real day. The module is ESM, the snapshot called `require()`, and it threw `require is not defined` at both corruption events on 09-02, the only two moments it existed for. The unit tests never took the path that shells out to `lsof`. ## `if (state.ok)` kept 25 failing ticks quiet for 4.5 hours The fourth incident, on 2026-09-04, was the first one I could name. Docker Desktop had the repo shared into its Linux VM. A SQLite inside the VM opened `gateway.db`. It couldn't see the host's POSIX locks or its memory-mapped WAL index, so it decided it was the only connection. It checkpointed, and when it closed it deleted `-wal` and `-shm`. Three host processes kept writing into unlinked files. The gateway's freelist said 803 pages while the file on disk said 742: two WAL worlds on top of one main file. When two of those processes exited cleanly, they checkpointed an orphaned WAL into a main file that had moved on. That caused the real damage: 100 problem lines in `gateway_messages_fts_data`, recovered in full from a backup taken before the damage. The alarm did fire. It fired once. The logging sat inside `if (state.ok)`, so only the move from ok to not-ok produced a line. The monitor then spent 25 ticks over 4.5 hours in that branch without a word. The one full `integrity_check` of the incident failed into the same silence. The verdict even changed while it was latched, from `2nd reference to page 56933` to `Rowid 687194767425 out of order`, and neither message was snapshotted. From the log, a quiet latch looks exactly like a dead monitor, and that's how it was read. Now it speaks on the first failure, again whenever the verdict text changes, and otherwise every 30 minutes. Each source (quick check, full check) has its own throttle, and a latch clock answers "how long has this been true?" The inode watch dated the swap at 07:02. The first failing check came at 07:10, eight minutes later. **Diagram (timeline)** Four gateway.db incidents from August 15 to September 4 and what each one exposed in the health monitor ```text 08-15 -> ~46h of lost writes, no alarm existed (problem) 08-30 -> alarm confirmed itself on the failing handle (problem) 09-02 -> 30 transients before damage, no timestamps (problem) 09-02 -> snapshot threw require() at both events (problem) 09-04 -> VM deleted -wal/-shm; latch silent 4.5h (problem) 09-04 -> repeating latch, inode watch, stranded state (fixed) ``` ## A health verdict is no wider than the connection that produced it The failure class, stated so you can check it in your own code: **a corruption verdict must be confirmed by a connection that did not produce the failure, a latched verdict must keep writing when its content or age changes, and evidence about file identity must be captured on the tick that first fails.** Any one of these is either true or false in your code. There's a WAL-specific corollary too. `-wal` and `-shm` are removed only when the last connection closes. If your process holds a connection open and the inode of either one changes, someone who couldn't see your locks removed them. That can't happen innocently. It also means a graceful shutdown of that process is dangerous: a clean close checkpoints the orphaned WAL. The fix is to kill it hard and restart, not to run your repair script. The repair script would throw away rows that only exist in the live WAL. ## Five minutes against your own WAL-mode database Point this at the database your long-running server keeps open. ```bash DB=/path/to/app.db PID=$(pgrep -f 'your-server' | head -1) # Inodes on disk right now (Linux first, macOS fallback) stat -c '%i %n' "$DB-wal" "$DB-shm" 2>/dev/null || stat -f '%i %N' "$DB-wal" "$DB-shm" # Inodes the live process actually holds (NODE column) lsof -p "$PID" 2>/dev/null | awk '/-(wal|shm)( \(deleted\))?$/ {print $8, $9, $10}' # A second opinion from a connection that has never failed sqlite3 "file:$DB?mode=ro" 'PRAGMA quick_check;' ``` Passing: the NODE numbers from `lsof` match `stat`, and `quick_check` prints `ok`. Failing: a different number, `(deleted)`, or `stat` reporting no such file while `lsof` still lists one. Any of those means the process is stranded, and a graceful stop will make it worse. Use `mode=ro` on purpose. A bare path to the wrong file makes `sqlite3` create an empty database and report it healthy. Next, compare that fresh verdict with what your health endpoint says right now. If the endpoint says corrupt and the read-only connection says `ok`, your alarm is reporting one handle's view. Last, write one test: stub the check to fail with `2nd reference to page 56933`, tick, switch the message to `Rowid 687194767425 out of order`, tick again, then advance a fake clock 31 minutes and tick a third time. Count the log lines. If you get one, your alarm goes silent after latching, just like mine did. ## The Hermes reports re-run the check but never question which connection ran it The Hermes agent project has hit the same class in public, and their fixes are good. [PR #41795](https://github.com/NousResearch/hermes-agent/pull/41795) found that an integrity check was skipped for the whole process lifetime after the first connect. A long-lived writer kept checkpointing a torn file and made roughly 20 quarantine copies in 10 minutes. [PR #100263](https://github.com/NousResearch/hermes-agent/pull/100263) and [issue #100227](https://github.com/NousResearch/hermes-agent/issues/100227) show a fixed-string FTS probe that missed corruption in one segment and printed "no repair needed" while every real append failed. [PR #99652](https://github.com/NousResearch/hermes-agent/pull/99652) describes treating a generic `SQLITE_CORRUPT` as FTS-only, which let a gateway serve for about 10 hours while every write was lost. Those fixes ask whether the check runs often enough and is sensitive enough. None of them ask whether the connection running the check is the thing that's broken, or whether the alarm is still talking an hour later. [Issue #100896](https://github.com/NousResearch/hermes-agent/issues/100896) is the closest. It reports four incidents in five weeks with a gateway and a dashboard writing the same WAL database, plus a warning that fired 7 minutes before onset. I saw 8 minutes between the inode swap and the first failed check. That early window is where the evidence lives, and a check that only looks at page contents can't see it. ## Nothing here repairs anything, and the inode watch has a blind spot This is detection only. The banner tells a person, and the person restarts or restores. I chose that deliberately: recovery written for a corruption whose cause you don't know can hide the one signal that would identify it. The inode watch also arms the first time it sees a sidecar file. A swap that happens before this process has seen `-wal` exist goes unnoticed. And a snapshot of a healthy file still costs a full `integrity_check`, so it's rate-limited and can miss the exact tick where things turn. If your agent's memory and history live in a local SQLite file, [vodou.ai](https://vodou.ai/register?utm_source=blog&utm_medium=feature&utm_campaign=db-health-a-forensic-snapshot-at-the-moment-gateway-db-first) ships this monitor. A second connection has to agree before it calls the file damaged, and it records who held the file when things first went wrong. --- Source: [A corruption alarm that latches goes silent exactly when it's right](https://blog.vodou.ai/sqlite-corruption-alarm-latch-forensics/) by Chad Priest, from Building Vodou in Public.