# Your isolated test lab isn't isolated: 3 rows, 8 daemons > My failure-injection harness wrote to the live database, health-checked a stranger's process, and leaked 8 daemons. Three bugs in the test, none in the code. - Author: Chad Priest - Published: 2026-08-26 - Canonical URL: https://blog.vodou.ai/your-isolated-test-lab-isnt-isolated/ - Tags: testing, debugging, sqlite, devops --- I wrote a script whose entire job is to break my system on purpose in a sandbox and print what every surface says. `scripts/broken-lab.sh`, 516 lines, committed as `ca66f6c8`. It found three real defects in its first hour, which felt great until I looked closer and found that three of the bugs it found were its own. It wrote three rows into the live `gateway.db`. It health-checked a different process than the one it started. And it leaked eight daemons into a machine-wide process limit that the production stack shares. The system under test was fine. The instrument was lying. ## The harness wrote to production because the app resolved its own path The gateway reads `VODOU_PROJECT_PATH` to decide where its SQLite file lives. What I did not know is that `db.ts` only *trusts* that variable if the directory already contains `vodou-core.db`, and it resolves the root once at module load. My lab set the env var, skipped the baseline step that seeds that file, and the gateway silently fell back to the repo root. Then it staged a graph fan and recorded three runs in the live database: the one file this script promises in its header never to touch. Nothing errored. The harness passed. I deleted the rows by hand. The fix is not "set the variable more carefully." The fix is that the harness stops believing it: ```sh if port_is_taken; then # Isolation is asserted, never assumed. If the gateway created its DB # anywhere but the lab, every later reading describes the live system. if [ ! -f "$LAB/MCP-servers/Vodou-Console/gateway.db" ]; then say "ISOLATION FAILED — the lab gateway did not create $LAB/MCP-servers/Vodou-Console/gateway.db." say " It is writing to the LIVE databases. Stopping before anything else runs." lab_gateway_kill return 1 fi return 0 fi ``` ## A 200 on your port is not proof that it's your process Second bug, and this one cost a full debugging cycle. Startup waited for `/api/health` to return 200 and called that success. But a leftover gateway from an earlier lab run still held `:8791`, so my new process logged `Refusing to start a second instance`, exited, and the health probe cheerfully passed: against the stranger. Every subsequent request hit another lab's database. The new lab's `gateway.db` was never created. Then the kill test SIGKILLed a pid that had already died and reported a clean recovery. Now the loop checks `kill -0 "$pid"` on the pid it spawned before it believes any 200, and aborts outright if the port is taken before boot. The third bug: the lab gateway spawns its daemon and worker *detached*, from the repo binary, so `kill -9 -PGID` misses them and `pkill -f "$LAB/vodou-core"` can never match. The lab's path isn't in their argv. Eight orphans accumulated in one afternoon, and since `VODOU_MAX_PROCESSES` counts machine-wide, the next run was refused with `6 vodou-core processes are already running (limit 5)`. The harness had started starving the machine it promises not to touch. The only place the lab's identity survives is the environment, so that's what the reaper matches now: `ps -E -o pid=,command= | grep -F "VODOU_PROJECT_PATH=$LAB"`. ## Any harness that discovers its dependencies like production will grade production The class: **any harness that discovers its dependencies the same way production does will eventually grade production.** Not "uses the same database". *discovers* the same way. Config file, env var with a fallback, service discovery, a well-known port, a shared process pool. It shows up as pytest fixtures against the shared dev Postgres, LangChain eval suites pointed at the production vector store by an unset `INDEX_NAME`, Compose suites reusing host ports, and any MCP or daemon test that spawns subprocesses with no cleanup barrier. [Testcontainers: Your Isolated Tests Are Still Lying To You](https://raju-shanigarapu.vercel.app/blog/2026-06-18-testcontainers-your-isolated-tests-are-still-lying-to-you) is right that a shared dev database is a solved problem and half-isolation is the real trap. What it misses is my case: a throwaway container guarantees a clean database exists, not that the app under test *used* it. My isolation was correct in the harness and discarded by the callee. Closer is [Your test suite is lying to you about which process it's in](https://dev.to/andreas_huttenrauch/your-test-suite-is-lying-to-you-about-which-process-its-in-39l0), where a correct guard ran in the wrong process. Same family. My version is worse in one way: I didn't have a guard that ran in the wrong place, I had a probe that couldn't tell processes apart at all. **A harness must assert on every run that the process answering it is the pid it spawned, and that the writes it made landed under the path it created. Both are resolved by the system under test, not by the harness.** That's checkable: grep your test setup for a readiness probe with no pid comparison, and for a teardown that trusts argv. Five-minute check against your own stack, no Vodou anything: ```sh # 1. Does your suite touch production? Fingerprint before and after. psql "$PROD_URL" -tAc \ "select relname, n_live_tup from pg_stat_user_tables order by 1" > /tmp/before pytest -q # or npm test, go test ./... psql "$PROD_URL" -tAc \ "select relname, n_live_tup from pg_stat_user_tables order by 1" > /tmp/after diff /tmp/before /tmp/after && echo ISOLATED || echo "YOUR SUITE WRITES TO PROD" # 2. Is the thing answering your health check the thing you started? ./run-test-server & echo $! > /tmp/mine.pid curl -sf localhost:8080/health >/dev/null && echo "health says 200" lsof -ti tcp:8080 -sTCP:LISTEN > /tmp/listening grep -qx "$(cat /tmp/mine.pid)" /tmp/listening \ && echo "same process" || echo "TESTING A STRANGER: $(cat /tmp/listening)" # 3. Do you leak? Count before and after. pgrep -fc myservice # run this before the suite, and again after ``` Passing looks like: empty diff, "same process", identical counts. Failing looks like a table that grew by three rows nobody can explain, a pid in `/tmp/listening` that isn't yours, and a count that climbs one per run until something unrelated starts getting refused. (`n_live_tup` is an estimate; for small tables use exact `count(*)` per table.) The rule I'd give a stranger: a test that cannot prove which process it talked to and which file it wrote has not produced a result, it has produced a sentence. Make isolation an assertion with an abort, not a configuration you set once and trust. --- Source: [Your isolated test lab isn't isolated: 3 rows, 8 daemons](https://blog.vodou.ai/your-isolated-test-lab-isnt-isolated/) by Chad Priest, from Building Vodou in Public.