Your cron job says completed. It can't tell quiet from broken

If you run an LLM step on a schedule, some mornings it has nothing to do. No new data, no new tickets, nothing shipped. Your wrapper exits 0, your job table says completed, and that’s correct.

Some mornings the model call dies twenty seconds in. Your wrapper exits 0 and your job table says completed, and that’s a lie. From the dashboard the two mornings look exactly the same. The quiet morning is the common one, so the broken one gets read as quiet too.

That’s what happened to the pipeline that writes this blog. This post covers the fix, the first version of the fix that was also wrong, and a check you can run on your own scheduled jobs before lunch.

Both writers exited 1 in 20 seconds and three layers said green

The blog runs three cron slots a day. Each slot does the same thing. It mines what shipped and what broke, has a model write a post, puts the draft through a rubric and a redaction gate, deploys a static site, and checks that the site returns 200. There are two writers, one for launch posts and one for incident stories, and the incident lane is the fallback when the feature lane is empty.

On 2026-08-26 the 08:00 slot called both writers. Both exited with rc=1 after about 20 seconds. After that:

the orchestrating script, blog-run.sh, exited 0
the background job runner marked the job completed
the scheduler recorded the run as having done the job

None of those three layers had a bug. “Produced nothing” really is a valid outcome for a slot, because the feature lane is supposed to be empty on days when nothing shipped. The script handled “no post” as one case, but two different events could cause it. Both got exit 0, so every layer above passed the green along faithfully.

The writers’ stderr wasn’t going anywhere either, so the reason for the crash was gone before anyone went looking. A slot that produced nothing looked the same as a slot with nothing to say.

08:00, 2026-08-26: a crash that became green on the way up

Four exit codes, reported after the deploy

The fix was a vocabulary, not a framework. The orchestrator now tracks two flags while the slot runs and turns them into one exit code at the end:

# 0  a post shipped, or there was genuinely nothing to publish
# 2  a finished draft was blocked by the redaction gate (the gate working)
# 3  a writer was invoked and failed or timed out (the slot lost work)
# 1  deploy or verify failed (unchanged)
# 4  another run held the lock; this fire stepped aside

The order matters as much as the codes. A failed writer doesn’t abort the slot. The pipeline keeps going: it deploys whatever is already on disk, skips the upload if the site hash hasn’t changed, checks the live site, and only at the very end prints SLOT FAILED and exits 3. If exiting early were the fix, then a dead writer would also leave the canonical domain stale, and that failure is worse than the one we started with.

Writer stderr now goes to the run log. Exit 2 is kept separate on purpose. A draft blocked by the redaction gate means a safety check caught something, which is different from a pipeline fault. It still has to be loud, though, because a feature post that silently disappears looks just like a day when nothing shipped.

The first fix hid 703 words the gate had stopped

My first version of the verdict logic was wrong, and it failed on the same day.

On 2026-08-26 the feature writer crashed. The slot then fell back to the incident writer, which produced a draft that passed the rubric. The redaction gate blocked it because it named an internal engine source file. So in one slot there was a crashed writer and a finished draft of 703 words stopped at the gate.

The first version checked the writer-failed flag first and returned right away. The exit code said 3 and the last log line said a writer crashed. Neither mentioned that a finished post was sitting in quarantine waiting for someone to fix one sentence. I had built a status that could tell quiet from broken, and it still couldn’t report two things at once.

An exit code can only hold one number, but a log has no such limit. The final version prints both conditions, SLOT BLOCKED and SLOT FAILED as separate lines, before it picks the code. Lost work outranks a blocked draft for the exit code, because lost work is the one that needs a person today. The log keeps the other one.

Here it is as a property you can check against any codebase:

Any outcome a human would respond to differently must reach the layer that human reads as a distinct status. In particular, a job’s “produced nothing” must never share a status with “tried to produce something and failed.”

A second property follows from the first: a wrapper’s exit status must depend on every child it invoked, not only on the last step it ran. In our case the last step was a site check that passed, so the script exited 0.

Both properties are true or false of your scheduler today. You don’t have to take my word for it.

Break one child on purpose and read your wrapper’s verdict

Pick one scheduled job whose output can legitimately be empty: a digest, an enrichment pass, an agent that acts only when there’s work. Run it twice, once quiet and once broken, and compare what comes back.

# 1. Quiet run: give the job nothing to do (empty input dir, empty queue, etc.)
./run-daily-job.sh; echo "quiet rc=$?"

# 2. Broken run: make the model/LLM step fail instantly.
#    Easiest generic trick: shadow the binary or script it calls with `false`.
mkdir -p /tmp/probe && printf '#!/bin/sh\necho "probe: forced failure" >&2\nexit 1\n' > /tmp/probe/python3
chmod +x /tmp/probe/python3
PATH=/tmp/probe:$PATH ./run-daily-job.sh; echo "broken rc=$?"

It passes if the two exit codes differ and the broken run’s log contains probe: forced failure. It fails if both print rc=0, and that’s the result I’d expect from most wrappers. It also fails if the codes differ but the stderr line can’t be found anywhere, because then you know it broke and have nothing to fix it with.

Then run the same check against history. If your runner keeps a table of runs, look for completed runs that left nothing behind:

SELECT r.id, r.started_at, r.status
FROM job_runs r
LEFT JOIN job_outputs o ON o.run_id = r.id
WHERE r.job_name = 'daily-digest'
  AND r.status = 'completed'
  AND o.run_id IS NULL
ORDER BY r.started_at DESC
LIMIT 20;

For every row, ask: from what’s stored, can I tell whether this was a quiet day or a crash? If the answer is “I’d have to go read logs, and the logs are gone,” you have the same bug we had on 2026-08-26.

Observability advice assumes the status is already honest

Most writing on production agents gets the symptom right. Towards Data Science describes a failure with “no crash. No error message. The system just kept running.” ZonFlip describes a bad result that “gets silently accepted by the next stage in the chain.” The usual prescription is more telemetry. GreenNode calls observability “the control plane.”

Telemetry wouldn’t have caught ours. Every layer recorded exactly what it was given. The job runner logged completed accurately, because the status set had no value that meant “ran, and lost work.” You can trace every span and still get a green board if the status vocabulary puts the empty case and the failed case together. The fix was four exit codes and a rule for their order, which fits Anthropic’s advice to find the simplest thing that works before adding complexity.

Still open: a slot that never starts has no exit code

The 0/1/2/3 vocabulary only helps if the pipeline actually runs. If the launcher fails (the scheduler can’t spawn the process, the environment is wrong, the machine was asleep), there’s no exit code to read. A missing run doesn’t look red. It looks like nothing. Catching that means checking for the absence of the expected run from the outside, not reading codes from the inside. That’s a separate check, and it has to be built on its own.

The scheduler that runs this blog is the one you’d get

This pipeline runs on Vodou. I built this into Vodou, which mines this blog from its own memory, writes it, grades it with a rubric, scans it with a redaction gate, deploys it and checks it, all on its own scheduler and background job runner. The 08:00 failure happened in the product, not in a demo, and the fix shipped in the open-source scripts/blog/ directory where you can read it.

What that means for you day to day:

Your memory, on your machine. The notes this post was built from (“never describe a failed slot as a success,” “the exit codes don’t cover launcher failures”) weren’t written down by hand. Vodou pulled them out of working sessions automatically and stored them in a local database I own. Whenever I open Claude Code, Cursor, or ChatGPT in the browser, those rules are already there. I don’t re-explain the incident.

It runs work while you’re away. Scheduled jobs, background scripts and MCP tool calls, all yours to add and change. The scripts, skills and schedules are all editable. Nothing is a black box you have to work around.

It notices before you do. Proactive loops flag an automation that stopped, capture that went quiet, or memory that stopped being extracted. Those are the silent failures this post is about.

Governance built in. Nothing outward-facing runs without your approval, and there’s an audit trail when you need to reconstruct a morning like 2026-08-26.

Vodou is for engineers who already run agents and scheduled LLM work, and who would rather own their memory and their pipeline than rent them. If that’s you, start at vodou.ai.

If your jobs can’t tell a quiet morning from a dead one, Vodou gives you a scheduler, a memory and loops built by someone who hit exactly that. Get started at vodou.ai.