140 agent runs had zero memory and my grader stayed green

If your agents pull from a memory store, you probably grade that retrieval somewhere: relevance of the injected context, whether cited ids exist, hit rate per lane. Every one of those graders reads retrieval evidence. So ask what it reports for a lane where retrieval never ran. If the lane leaves no evidence rows behind, the grader has nothing to examine. It doesn’t go red. It goes quiet, and on a dashboard quiet looks the same as green.

That happened to me. For eleven days, 140 of 140 scheduled runs in one automation lane ran with zero memory. The board stayed green the whole time.

140 scheduled fires, zero memory ids, and a receipt grader with nothing to read

Vodou grades itself with a set of “flows”. Each flow is a row with a verdict (ok, warn, red or unknown) built from live evidence. One row, Flow 4, grades receipt honesty. When a turn says it used memories, do those memories exist, and did the answer use them? That’s a good question to ask. But to answer it, the row first drops every receipt that carries no memory ids, because a receipt with no ids has nothing to check for honesty.

A lane at 100% failure produces only receipts with no ids. Flow 4 dropped all of them, looked at what was left, found nothing wrong, and said so. The scheduled skill runs, the lane that executes skills on a timer with no human in the loop, had been running blind since before anyone looked.

The new row, Flow 14, grades coverage instead of honesty. It starts from every receipt the lane produced and sorts each one into one of three states.

The distinction in the middle matters. A retrieval that ran and found nothing relevant is correct behaviour, and grading it as a failure would train everyone to ignore the row. The defect is the third state. For the scheduled skill lane, the row goes red when 90% or more of turns never ran memory, over at least 10 turns. With no receipts that carry lane data, it answers unknown, never ok. The first live run printed RED, skill-console: 52/57 turns never ran memory and exited 2.

The per-turn memory count I nearly graded on was wrong for the healthy lanes

My first design graded on the per-turn memory count the receipt already carries. It was one field and one threshold, so it looked like the obvious choice. Before I shipped it I measured that field against the per-lane retrieval record, and on channel turns and heartbeat turns it under-counted what had really been injected. Grading on it would have turned a healthy lane red and left a dead one green. The row reads the per-lane record instead, and that record is the only thing it trusts.

That was the first of four problems that week. Fixing the rest was most of the work.

Flow 3’s 8.2-hour median was 700 chunks re-dated by one file rewrite

Flow 3 measures capture latency: how long from a conversation until its facts are searchable. For two weeks it sat on a permanent warn at median 8.2h, p90 21.6h. It computed that as the earliest chunk created_at minus the conversation’s last update.

created_at is not when a fact became searchable. When a file’s content changes, the indexer deletes all of that file’s chunks and inserts them again. A daily log grows all day, so every append re-dates every fact already in it. On one day’s log I found 700 chunks from 28 different conversations, all stamped inside a single 23-second window. A 09:00 conversation scored about 14 hours.

The row now reads the extraction ledger: the last message a cycle covered, compared with the time that cycle finished. A separate piece of work had already measured latency that way and got a median of 4.5 minutes. The rewritten row got 4.3 (n=672). Two independent implementations agreeing is the only reason I trust either one. The new reading is ok, median 4m, p90 14m. The thresholds come from the 300-second extraction poll interval, not from a number that happened to look right.

A comment listed four exclusions and the SQL implemented two

Flow 11 flags untagged memory chunks as damage. The comment above its query names four populations that are untagged on purpose: document chunks, imports awaiting classification, run logs, and [RUN] notes. The SQL excluded two of them. It flagged 11 chunks, and 2 of those were imports, which the comment itself calls a transient state and not damage. With all four exclusions in place, the count went from 11 to 9.

My notes say this is the fourth time the same class has come back. A documented exclusion that the query doesn’t implement is worse than no documentation, because the comment tells the next reader to trust a filter that isn’t there. The new test pins all four exclusions together. It also checks the other direction: a real untagged first-party fact still trips the row.

| head -3 hid the one failing bin test for five pushes

While moving Flow 3’s tests onto the ledger, I updated three and missed a fourth. CI went red on the binary’s test suite and stayed red for five pushes: 2152 passed, 1 failed. Every time, I reported “1051 green”. The test runner prints one result line per suite, I read it through head -3, and that cut the output off after the library suite’s line. I already had a written lesson about exactly this, and I made the mistake again with a pipe. Both suites are green now, 1051 and 2153.

The last fix was the same disease in configuration. A diagnostic script showed as “not running” and made its row the only red on the board. The stack registry already declared it optional. The grader never checked the stack registry. It read a second optional flag in processes.toml that had been set on two other processes and not on this one. Now the stack registry decides, and the processes.toml key is only a fallback for installs that have no stack registry.

A hit rate whose denominator is its own evidence rows cannot see 0%

This is the class, stated as a property you can check. Every grader’s denominator must come from the table that records what was supposed to happen (the scheduler, the job table, the turn ledger), never from the table that records what happened. If the population is “rows where evidence exists”, a total failure removes itself from the population. The grader is structurally unable to report it.

The other three bugs are versions of the same thing. Flow 3 used a write time as an event time. Flow 11’s query didn’t match the population its comment described. The process grader read a second copy of a fact. Each time, the thing being counted wasn’t the thing the row claimed to measure.

Find the WHERE clause that deletes your worst lane from its own grade

This takes five minutes against any agent stack that has a job or run table and some kind of retrieval log. Names here are generic, so rename them to match your schema. SQLite syntax is shown. In Postgres, use COUNT(*) FILTER (WHERE ...).

First, run the population your retrieval-quality metric actually uses:

SELECT lane, COUNT(*) AS graded
FROM retrieval_events
WHERE memory_ids IS NOT NULL AND memory_ids <> '[]'
GROUP BY lane;

Then start from the scheduler instead:

SELECT j.lane,
       COUNT(*)                                      AS runs,
       SUM(r.run_id IS NULL)                         AS never_ran,
       SUM(r.run_id IS NOT NULL AND r.hit_count = 0) AS ran_empty,
       SUM(COALESCE(r.hit_count, 0) > 0)             AS injected
FROM job_runs j
LEFT JOIN retrieval_events r ON r.run_id = j.id
WHERE j.started_at > datetime('now', '-7 days')
GROUP BY j.lane
ORDER BY 1.0 * SUM(r.run_id IS NULL) / COUNT(*) DESC;

Passing: every lane in the second result also appears in the first, and never_ran is near zero, or is explained by lanes that are designed to skip retrieval. Failing: a lane appears in the second result and is missing from the first, or its never_ran / runs is close to 1. That lane is invisible to your quality metric, and your dashboard is green for it because it has no rows, not because it’s healthy. Pay most attention to lanes started by cron and webhooks, which have no human watching.

While you’re in there, check whether your “created” timestamp is really a write time:

SELECT strftime('%Y-%m-%d %H:%M', created_at) AS minute,
       COUNT(*) AS chunks, COUNT(DISTINCT source_id) AS sources
FROM memory_chunks
GROUP BY minute
HAVING COUNT(DISTINCT source_id) > 5
ORDER BY sources DESC LIMIT 5;

If dozens of unrelated sources share one minute, that column is when the indexer last rewrote the file. Any latency or freshness number computed from it is fiction.

A run that retrieved nothing leaves nothing for a verification loop to taste

The current writing on agent verification is good, and it assumes an artifact exists. The self.md verification loop sits “between ‘done’ and a write actually landing” and runs seven checks on what the agent produced. Neural Pruning compares per-skill checks to a chef tasting before plating, and relies on a monthly audit to catch “the structural drift that silent failures hide.” Anna Jey’s guide defines verification as checking “whether an agent used the right evidence.”

All three examine something that exists. None of them asks what the check’s denominator is. A lane that never searched memory still produces fluent output, so the chef has a plate to taste. What’s missing from the plate is invisible unless something counts expected plates against served ones. Eagle Mem splits memory by timing (“recall at start, search on prompt”), which is the right shape. It’s also the shape where a hook that never fires leaves no record of not firing. A monthly audit would have caught my lane eventually. Eleven days of automation had already run without it.

Only the scheduled skill lane can go red; every other lane is capped at warn

The 90%-over-10-turns floor applies to one lane, because that’s the only lane where I have data to justify a floor. Channel, heartbeat and the other lanes warn when their never-ran share looks high, but they can’t turn the board red until I’ve measured what normal looks like for each. A warn is easier to ignore than a red. So the blind spot is smaller now, not closed. And the row turning red is detection, not a fix: the lane got a red row that afternoon, not its memory.