The runs panel said 'No runs recorded yet'. The ledger had every run.

Every agent stack grows a scheduler. A cron-shaped table, a loop that fires prompts on time, and a row written per run so you can answer “did the nightly triage actually run, and what happened.” Then a second surface is built, months later, by someone else, to show those runs. It reads a table. Nothing in the codebase checks that it reads the table the scheduler writes.

That is the shape of what I found today in the Vodou gateway, an Express process in MCP-servers/Vodou-Console/src/index.ts that fronts the chat, the scheduler, and the model. I shipped five small commits in one day. Two were the same bug wearing different clothes: a producer and a consumer that had never been introduced. One was a blocking subprocess call that had been slowing every request for as long as the console has been able to switch conversations. This post is the field report, with the checks you can run on your own scheduler in five minutes.

scheduled_task_runs had every row and graph_runs had none of them

The scheduler has recorded a row per fire in a table called scheduled_task_runs since migration 086 landed. Each row carries what a person actually wants to know: when it was scheduled for, when it started and finished, the status, a reason on failure, how many characters of output came back, who it was delivered to, whether delivery succeeded, and how late it fired in seconds.

The skill console has a Runs panel. It read graph_runs. That table is written by the recipe engine and the test suite, and by nothing else. So every scheduled skill in the console said “No runs recorded yet,” while the Skills page, one tab over, showed the same skill’s last outcome from scheduled_task_runs. Two surfaces, one product, contradicting each other about whether a thing had ever happened.

The fix is one read-only route in MCP-servers/Vodou-Console/src/api/scheduler.ts, declared before the /:id routes so Express does not swallow it:

schedulerRouter.get('/runs', (req, res) => {
  const task = String(req.query.task || '').trim();
  if (!task) { res.status(400).json({ error: 'task is required' }); return; }
  const limit = Math.max(1, Math.min(200, parseInt(String(req.query.limit || '40'), 10) || 40));
  const runs = getDb().prepare(`
    SELECT id, task_id, task_name, scheduled_for, started_at, finished_at, status, reason,
           output_chars, delivered_to, delivery_ok, lateness_s
      FROM scheduled_task_runs
     WHERE task_name = ?
     ORDER BY id DESC LIMIT ?`).all(task, limit);
  res.json({ task, runs });
});

The QA console refused every question because the template never said {{user_message}}

The second half of the same defect. A skill console is a chat that knows its skill: the person’s message is rendered into the skill’s prompt template through a {{user_message}} placeholder, and the result goes to the model. The nightly QA triage skill has a template with no such placeholder. It was written to be fired by the scheduler, not talked to.

So when I typed a question into that console, the render produced the template plus its injections and nothing else. The model never saw the question. It saw a prompt that looked like a scheduled fire, and answered with the scheduled-fire refusal it had been told to give. Every message. No prompt rule about “if the person asks a question” could ever apply, because the person’s words were not in the prompt.

I read that refusal three times as the model being stubborn before I read the render function. Same class as the runs panel: a producer (the person typing) and a consumer (the model) with no path between them, and a surface in the middle that looked alive. The fix in MCP-servers/Vodou-Console/src/api/skill-console-handler.ts is that a template with no slot gets the message appended, labelled, with a note that a real scheduled fire arrives as [scheduled fire @ …]. Templates that place the slot keep control of where it goes.

110 ms alone, 9.6 s during a console boot, and the cause was claude auth status

The third thing was the one that changed how I read the other two. I was measuring the Memory timeline endpoint while chasing an unrelated slow page. It answered in about 110 ms with nothing else happening. Under a pure REST burst it went to 0.3 s. During a real console boot it took 9.6 s. A 250 ms probe fired during a reload showed stalls of 2.3 s, 1.2 s and 1.1 s with nothing else slow on the server.

The gateway is one Node event loop. When the console boots, it switches to a conversation, and switching to a conversation that has not been warmed pre-warms the CLI session. That warmup checked auth first, which is correct: an earlier incident had it blindly spawning a logged-out CLI and resolving empty replies. But the check was a spawnSync of the claude binary with a 5 s timeout, in MCP-servers/Vodou-Console/src/llm.ts, on every switch. spawnSync blocks the loop. Every HTTP handler waited 1 to 2 s per call while a child process reported that, yes, I was still logged in.

Auth state does not change between conversation switches. The cached version memoises the raw result for 60 s and the warmup path reads the cache. There was already an 8 s probe cache for a different caller; it stayed. The fix takes effect on the next gateway restart, and I did not restart it today, so as of this writing the live process is still stalling. I want that on the record rather than implied away.

Two smaller ones from the same day belong here for completeness. Stream sequence numbers are per-process, hydrated from a buffer with a 10-minute TTL, so a client that kept its per-conversation high-water mark across a restart dropped the first ~150 characters of the next reply as duplicates. The WebSocket handshake now carries the process start time as an epoch and the client clears its cursors when it changes. And /next/, the redesign staging copy, 404’d on the bare path because express.static runs with index:false. One route.

The property: every run-shaped table has a UI reader, and every UI reader has a production writer

Here is the class, stated so you can check it rather than nod at it. For each table whose name or columns describe an execution (runs, fires, jobs, attempts, with a started and finished column), the set of code paths that INSERT into it in production must be non-empty, and the set of user-facing surfaces that SELECT from it must be non-empty. A table with writers and no surface is a ledger nobody reads. A surface with a table that has no production writers is an empty state that cannot be distinguished from a correct empty state. Both were true in this codebase at once, one on each side.

The second property is simpler. No synchronous subprocess call sits on a request path in a single-threaded server. Not “be careful with spawnSync.” A grep either finds one in a handler or it does not.

Five minutes on your own scheduler: count writers per table, then time a cheap endpoint during a reload

Find the run-shaped tables and who touches them. In SQLite:

SELECT name FROM sqlite_master
 WHERE type='table' AND (name LIKE '%run%' OR name LIKE '%job%' OR name LIKE '%fire%');

For each name, count rows and the newest one, then count code paths on each side:

for t in scheduled_runs graph_runs job_attempts; do
  echo "== $t"
  sqlite3 app.db "SELECT COUNT(*), MAX(started_at) FROM $t;"
  echo "writers: $(grep -rlE "INSERT INTO $t\b" server/ --include='*.ts' --include='*.py' | grep -v test | wc -l)"
  echo "ui readers: $(grep -rlE "\b$t\b" web/ server/api/ | wc -l)"
done

Passing output has a non-zero writer count and a non-zero reader count on every table with rows. Failing output looks like what I had: one table with 4,000 rows, writers 1, ui readers 0; a sibling table with 0 rows in production, writers 0 outside tests, ui readers 1.

Then the event loop. Pick your cheapest GET and probe it every 250 ms while you do the expensive thing your UI does on load:

while true; do
  curl -s -o /dev/null -w '%{time_total}\n' http://localhost:8080/health
  sleep 0.25
done

Passing output is a flat column of numbers near your baseline. Failing output is a baseline with spikes of a second or more that line up with a reload in the other window. When you see the spikes, run this before you read any code:

grep -rnE 'spawnSync|execSync|execFileSync' server/ --include='*.ts' --include='*.js' | grep -v node_modules

Every hit inside anything reachable from a request or a WebSocket message is a candidate. In Python, the equivalent is a subprocess.run inside an async def.

The published advice covers startup pre-warm, not the per-switch version, and covers tool surfaces, not run ledgers

The auth pre-warm class is not mine. OpenClaw shipped a release where provider auth pre-warm blocked the Node event loop for about 60 s on startup and broke channel handshakes, and a second report measured 179 to 310 s of pre-warm with event-loop maximums of 64 s. A write-up of the same release attributes the RPC degradation to the pre-warm plus a synchronous plugin-manifest fallback that scanned 91 manifests per call. Those are the loud version: once per boot, minutes long, everyone notices. Mine was 1 to 2 s per conversation switch, which is exactly quiet enough to live for months as “the console feels a bit heavy on load.” Fix the loud one and you have not checked for the quiet one.

On the ledger side, the guidance for exposing software to agents, Agent Surface and its repository, is thorough about how an agent reads docs, calls tools, and parses errors. OpenAI’s practical guide to building agents covers orchestration patterns and guardrails. Neither says that a scheduled run must produce a receipt a person can find, on the surface where that person looks. The receipt was written. The advice stops at the tool return, and so did my console.

Still live: an install without migration 086 gets an empty list that looks like a healthy empty list

The new route catches a missing table and returns an empty array. That was deliberate, so an older install does not 500. It also means the exact symptom I fixed, a correct-looking “No runs recorded yet,” is still what a person sees on a database that has not migrated. The honest response is a distinct status, not an empty list, and I have not written it. Separately, lateness_s is now served per run and nothing alerts on it. The ledger is readable. Nobody is yet reading it on my behalf.