# cached_tokens is 0 because your system prompt isn't stable

> Our provider reported cached_tokens 0 of ~16K prompt tokens every turn. The cause: per-turn memory glued into the system prompt. Freeze the prefix instead.

- Author: Chad Priest
- Published: 2026-08-26
- Canonical URL: https://blog.vodou.ai/cached-tokens-is-0-because-your-system-prompt-isnt-stable/
- Tags: llm, caching, typescript, debugging

---

I added a `[cache]` log line next to our usage tracker and read `usage.prompt_tokens_details.cached_tokens` off a warm conversation. It said `0`. Then `1`. Out of roughly 16,000 prompt tokens, every single turn. We were paying full price for the same prefix over and over, and the 273K-token board balloon I'd been blaming on the agent loop was just real money.

## Session affinity was the wrong theory for the first day

My first theory was session affinity. Fireworks' docs say serverless needs a routing hint or turn 2 lands on a replica that never saw turn 1, and we were sending neither `user` nor `x-session-affinity`. Obvious culprit. Cheap fix.

So I wrote a direct probe against `api.fireworks.ai` with a byte-identical 4,034-token prefix:

```
call1 (cold,  user=A): prompt=4034 cached=0      0%
call2 (warm,  user=A): prompt=4034 cached=0      0%   ← still cold
call3 (warm,  user=A): prompt=4034 cached=4033 100%
call4 (no user field): prompt=4034 cached=0      0%
call5 (no user field): prompt=4034 cached=4033 100%
```

Affinity wasn't decisive: calls 4 and 5 hit 100% with no hint at all. Caching was automatic and total. It just needed the prefix to actually be identical, and ours wasn't. Two things were rotting it: the ~24KB workspace bootstrap rode turn 1 and vanished on turn 2, and the ~15K query-dependent memory recall was concatenated straight onto the front of the system message.

## chatWithOpenAICompat freezes the system prompt behind a flag

In `MCP-servers/Vodou-Console/src/llm.ts`, `chatWithOpenAICompat` now has a mode where the system prompt is deliberately frozen:

```ts
if (STABLE_PREFIX) {
  systemPrompt = staticParts;                // frozen → cacheable prefix
  lateContextBlock = memoryForSystem || '';  // volatile → relocated
} else {
  systemPrompt = staticParts + '\n\n---\n\n' + memoryForSystem;
}
```

The relocated memory gets spliced in as a late `system` turn, immediately before the current user message:

```ts
const insertAt = Math.max(1, m.length - 1);
m.splice(insertAt, 0, { role: 'system', content: '### Relevant context for this turn\n\n' + lateContextBlock });
```

Flag on: turn 1 and 2 read 0% (the write lag the probe already warned me about), then 93% → 96% → 97%. Flag off, same conversation: 0 / 81 / 0 / 90 / 95 / 58. The signature of a prefix that busts whenever retrieval returns something new.

## Re-sending the bootstrap every turn cost 2.1x more, not less

I also "fixed" the bootstrap by re-sending it every turn so the provider could cache it. Six turns measured 49K raw / ~33K billed, against 19.6K / ~16K for the old drop-after-turn-1 behavior. About 2.1× worse. Fireworks discounts cached input ~50%, not the ~90% Anthropic-style number I had in my head, so re-sending 6K to cache it loses to simply not sending it. I reverted that half.

The rule that transfers: in a frozen-prefix design, the system prompt is a data structure with an invariant, not a string you can append to. Anything query-dependent goes in the message array. And before you build for a cache, measure its discount: a 50%-off cache and a 90%-off cache reward opposite architectures.

---

Source: [cached_tokens is 0 because your system prompt isn't stable](https://blog.vodou.ai/cached-tokens-is-0-because-your-system-prompt-isnt-stable/) by Chad Priest, from Building Vodou in Public.