# Your LLM fallback is probably recording the model that failed

> When the LLM extractor fails and a heuristic takes over, the rows kept the failed provider's name. How one grader replaced three drifting scripts and a blind grep.

- Author: Chad Priest
- Published: 2026-09-13
- Canonical URL: https://blog.vodou.ai/llm-fallback-provenance-extraction-grader/
- Tags: llm, memory, observability, ai-agents

---

If you run LLM extraction in production, you have a fallback. The model call times out, the provider returns a 529, the local server is asleep, and some cheaper path writes something so the pipeline doesn't stall. Usually that path is a heuristic, a smaller model or a regex. That's the right call. The question this post is about is narrower: when the fallback writes a row, what does the row say wrote it?

In my system the answer was the provider that had just failed. For weeks, every heuristic bullet that landed in memory was labeled `claude-cli`. Three shell scripts were watching for exactly this kind of damage, and on the same day two of them reported different numbers for it.

## Flow 11 asks the rows who wrote them instead of asking the log

Vodou has a command called `flows` that grades a handful of end-to-end guarantees from live evidence and answers `ok`, `warn`, `red` or `unknown`. On 2026-08-25 I added Flow 11, extraction honesty. It asks one thing: in the last 24 hours, did fallback output land in long-term memory, and can every untagged row be explained?

It gets its answer from the database. The engine now stamps `heuristic` on anything the fallback writes, so a row names its real author. All three fallback sites log at WARN and use the same wording. The list of populations that are untagged on purpose now lives in one place, inside the grader, and the scripts that used to carry their own copy call the grader instead.

**Diagram (beforeafter)**

Before: three scripts each with their own damage query and a log grep. After: one grader reads provenance from the database and scripts call it.

```text
  BEFORE: Before
    - 3 scripts, 3 copies of the damage SQL
    - fallback counted by grepping a log line
    - 1 of 3 fallback sites wrote that line
    - heuristic rows labeled with the failed provider

  AFTER: After
    - 1 grader owns the predicate
    - rows stamped heuristic by the code that wrote them
    - all 3 fallback sites log at WARN
    - a flow this build lacks exits 2, not 0
```

The public side of the change is in `scripts/bridge/`. `bridge-snapshot.sh` and `bridge-watch.sh` lost their inline SQL and now shell out to `flows --json --flow 11`. If the binary is too old to have Flow 11, they print `absent`, never a reassuring 0. `bridge-guard.sh` and its launchd plist (`com.vodou.bridge.guard.plist`) were deleted. Watching extraction was never the bridge's job.

## 40 doc rows, 36 import rows, and a grep that printed 0 twice

The guard started out simple: alert when untagged chunks show up. It was rewritten three times in one day, and the header comment of the deleted script kept a record of each one.

The first version counted every untagged chunk. It went off on 40 document passages from a file scan. Those are untagged by design. The second version excluded documents and went off on 36 imported chunks that were just waiting for the tag classifier to reach them. For those rows, untagged is a normal temporary state, not damage.

The third rewrite was a bash bug, and I'd bet some version of it is in your repo:

```bash
n=$(grep -c "falling back to heuristic" app.log || echo 0)
echo "[$n]"
# with no matches this prints:
# [0
# 0]
```

`grep -c` prints `0` and also exits 1 when nothing matches, so the `|| echo 0` adds a second zero. The variable becomes `"0\n0"` and every `-gt` comparison after it breaks. The default meant to protect the count was the thing that broke it.

After three rounds the guard finally measured the right population: untagged bullets in the daily log. But its fallback counter was still a grep for one log line, and that line came from only one of the three fallback sites. The other two lanes, which promote short-term memory into daily and weekly summaries, logged their fallback at debug level. You'd only see it with a debug flag set. Both of those lanes could fall back on every single call and the counter would still read 0. It was a metric that total failure would pass.

## The trend line said 81 while the guard said 11

The damage predicate had been copied into three scripts. On 2026-08-25 I fixed the exclusions in one copy and not the others.

**Diagram: One damage metric, two answers, 2026-08-25**

Same day, same database: the trend script reported 81 damaged rows and the guard reported 11

```text
  trend line : 81 rows  ################################ (problem)
  guard      : 11 rows  #### (problem)
```

Neither number was right, and I couldn't tell which one was less wrong without reading all three queries side by side. That is when the predicate moved into the grader. A copied SQL string has no owner, so every fix only reaches the copy you happened to open.

Wiring it up turned up one more false all-clear. If you asked `flows --flow 11` of a build that had no Flow 11, it printed "All flows agree with themselves" and exited 0. An empty set agrees with itself trivially. Any monitor that pinned a flow by number would have read that as healthy. Now it prints which flows the build actually grades and exits 2.

## 47 `[RUN]` bullets turned the new grader yellow

I shipped Flow 11, ran it against the live install and got `warn`: 47 untagged bullets had landed in a daily log. Each one looked like `[RUN] ✓ cpu mcp-monitor·get_cpu_info 31ms`. They were tool run notes, the same content the grader already skipped when it appeared under a `## Run log` heading, just written as bullets instead. The exclusion matched on the heading, so the bullets got through.

**Diagram (timeline)**

Each exclusion in the extraction damage grader was learned from a false alarm

```text
  rewrite 1   ->  40 doc passages flagged, untagged by design (problem)
  rewrite 2   ->  36 import chunks flagged, awaiting classifier (problem)
  rewrite 3   ->  grep -c || echo 0 yields 0 twice (problem)
  2026-08-25  ->  three copies report 81 and 11 (problem)
  2026-08-25  ->  predicate moves into one grader (fixed)
  same day    ->  47 [RUN] bullets flagged, excluded, tested (fixed)
```

I didn't catch this with a test. I caught it by running the grader on real data right after the extraction change, and that's why the test exists now. The grader's own docstring warns about this failure: if an instrument says `warn` about its product's normal output, people learn to ignore it, and then it isn't an instrument anymore. The flow tests pass 18 of 18. The new cases say run notes are not damage, heuristic output that landed is red, an untagged row with no explanation is a separate fault and says so, and a quiet day is `unknown`, never `ok`.

## The invariant: a fallback overwrites the provenance it inherited

This is a class of bug, not a one-off, and you can state it as something that is either true or false of your code:

**Every row written on a fallback path records the fallback as its writer, and the fallback is logged at a level that production actually emits.**

The mechanism was mundane. The backend setting was read once, before the provider call. When the call failed, the heuristic ran with that setting still in scope, and the insert used it. Nothing in the system reads that column to decide anything, so nothing broke when it was wrong. A column no code path depends on can stay wrong forever. The only thing that finds it is a grader that reads the column on purpose.

A second property comes with it: a damage predicate that more than one consumer uses has exactly one definition, and those consumers call it instead of copying its text.

## Break your primary extractor for ten minutes and group by writer

You can check this in five minutes with nothing from my stack. First, point your primary extraction provider at a bad key or an unreachable host, then trigger one extraction cycle. Next, look at what landed:

```sql
SELECT extractor, COUNT(*) AS n
FROM memories
WHERE created_at > datetime('now', '-15 minutes')
GROUP BY extractor;
```

Passing output names the fallback, like `heuristic|9`. Failing output names the provider you just broke, like `openai|9`. If there's no writer column at all, that's also a fail: you can't tell fallback output from model output after the fact.

Next, find the fallback sites that production won't log:

```bash
grep -rnE "fall(ing)? ?back" src/ | grep -viE "warn|error"
```

Passing means no output. Every line that does print is a fallback that can happen as often as it likes while your counters read 0.

Last, count copies of your damage query. Take its most distinctive clause and search every place a number could come from:

```bash
grep -rn "tag IS NULL" scripts/ ops/ dashboards/ | wc -l
```

`1` passes. Anything higher means you'll eventually get your own 81 and 11.

## Self-checks read the output and trust the column that names its author

Most of what's written on agent verification checks the artifact. The [self.md verification loop](https://self.md/guides/agent-verification-loop/) runs its checks between "done" and a write landing. [Neural Pruning](https://neuralpruning.ai/systems/agent-verification/) pairs quick per-skill checks with a monthly audit meant to catch structural drift. [Anna Jey's guide](https://medium.com/toward-next-ai/ai-agent-verification-how-to-make-autonomous-workflows-check-their-own-work-before-production-5daa1544573c) asks whether the agent used the right evidence and called the right tools. All of that is sound. All of it also assumes the record of which tool or model produced the output is true.

The [agent-dialectic-resolver](https://download.plaud.ai/spotrunner1/agent-dialectic-resolver) pattern comes closest: adjudication is "deterministic code that reads typed evidence." That only works if the evidence is honest. My provenance field was typed, structured and wrong, and a deterministic judge reading it would have confidently scored heuristic bullets as model output. None of these sources checks the fallback branch, because it's the branch that runs when nobody is looking.

## Still open: the exclusion list only grows when it cries wolf

Every exclusion Flow 11 has, it learned from a false alarm on real data. Documents, imports, run-log sections and `[RUN]` bullets were each added only after the grader flagged them. Nothing stops the next writer from adding a new untagged population, and the first sign will be another `warn` that is really noise. The grader now separates "heuristic output landed" (red) from "untagged and unexplained" (a separate fault) so that noise doesn't read as damage. But the list is still built reactively, and I don't yet have a way to make a new writer declare its population before it ships.

---

Source: [Your LLM fallback is probably recording the model that failed](https://blog.vodou.ai/llm-fallback-provenance-extraction-grader/) by Chad Priest, from Building Vodou in Public.