# Your prompt snapshot test pins what you believed, not what ran > A byte-level replay gate for LLM prompt assembly only works if fixtures are recorded from the real product and the recorder sees every exit. Ours saw one of three. - Author: Chad Priest - Published: 2026-09-17 - Canonical URL: https://blog.vodou.ai/recorded-prompt-replay-ci-gate/ - Tags: llm, ai-agents, observability, architecture, testing --- Somewhere in your agent there is a function that builds what the model reads: the system prompt, retrieved memory, tool output, a scope note, a trust fence around untrusted text. Somebody edits one line of it, every unit test passes, and the model now gets a different prompt. You won't find out from CI. You'll find out when behavior drifts and nobody can say which commit did it. I had this exact gap in [Vodou](https://vodou.ai/register?utm_source=blog&utm_medium=feature&utm_campaign=replay-p1-a-prompt-change-now-fails-ci-with-a-diff), a local-first system that injects your memory into model calls through several lanes. On 2026-08-28 we recounted what actually writes into a prompt and found five injectors running next to the one assembler, for weeks. Every test agreed with what we believed the assembler did. None of them checked what it did. ## A one-byte change now exits 1 and prints char 19 of 891 The gate is a replay suite. Each fixture is a turn recorded from the running product: the assembler's inputs, the conversation state it found, and the exact bytes it produced. The test feeds those inputs back to the current code and compares byte for byte. The injected context, the user-prefix block, the set of lanes that fired, and each lane's size all have to match. When a byte differs, the failure tells you where. A plain `expected X to be Y` over two 40 KB strings is useless, so the diff finds the first differing character and prints a window around it: ```ts function firstDiff(a: string, b: string): string { if (a === b) return '(identical)'; let i = 0; while (i < Math.min(a.length, b.length) && a[i] === b[i]) i++; return [ `at char ${i} of ${a.length} (recorded ${b.length})`, ` …shared: ${JSON.stringify(a.slice(Math.max(0, i - 40), i))}`, ` NOW : ${JSON.stringify(a.slice(i, i + 70))}`, ` RECORDED: ${JSON.stringify(b.slice(i, i + 70))}`, ].join('\n'); } ``` To prove it, I changed one byte of the ground-truth fence the assembler writes. The suite exited 1, named the assertion (USER PREFIX), and reported char 19 of 891, recorded 890, with NOW and RECORDED on either side. I reverted it and got exit 0. I checked the exit code on purpose. I once pushed two red commits because a test runner piped through `grep` returns grep's exit status, and red text in a pipe is not a gate. **Diagram: Record once, replay on every commit** Recording a real turn into a fixture, then replaying it in CI against the current assembler and diffing the bytes ```text [Real product turn] --> [Fixture JSON] --> [CI replay] --> [Byte compare] [Byte compare] --any byte differs--> [Exit 1 + char offset (problem)] [Byte compare] --identical--> [Exit 0 (fixed)] notes: Real product turn: capture flag on Fixture JSON: input + state at ENTRY + output CI replay: warm conversation, then assemble Byte compare: injected, userPrefix, lane set Fixtures are never hand-written. A red fixture is re-recorded and the diff is reviewed. ``` The rule is that fixtures are recorded, never hand-written. A hand-written fixture pins what its author believed. A recorded one pins what the code did. If a prompt change is intended, you re-record and commit the new fixture with the diff in review. Editing the expectation by hand until the suite goes green is the exact failure the suite exists to catch. ## The recorder reached one of the assembler's three exits The first version looked done. It had fixtures, the diff worked, and the tests were green. Then I used the gate to grade its own recorder and found four defects. First, the assembler has three return paths: a full assembly, a cache hit that reuses an already-built base prompt, and a skill turn. The recorder was hooked into one. The cache-hit and skill shapes could never become fixtures at all. A replay gate that is blind to two of three exits has its hole exactly where the least-exercised code lives. Second, even after hooking the cache-hit exit, the recorder didn't save the cached base. So a cache-hit fixture replayed as an ordinary full assembly and asserted a shape the product never took. It was green, and it was wrong. Third, the recorder read conversation state at the exit, after the assembler had mutated it. That produced a fixture claiming both "bootstrap was sent this turn" and "the conversation was already bootstrapped", which can't both be true. Now state is snapshotted at entry, and the replay warms the conversation into that state before asserting. A continuing turn replayed cold becomes a first turn, with a different lane set. Fourth, the fixtures contained my install root and home directory. They were red on every other machine and in CI, and they committed a personal path into a repo that is open source. Paths are now scrubbed to `` and `` at capture and again at compare. Every other byte is still asserted. Then came the environment. Some lanes carry text the machine supplies: the base system prompt from settings, a bootstrap from a cache file, ground truth containing the local path. For those, the suite asserts that the lane is present but not its size. The bootstrap decision is only asserted when the cache file exists. When it doesn't, the test prints that it skipped the check, so it neither fails nor quietly passes. I verified this by deleting the cache file and running it again. **Diagram (beforeafter)** Before: recorder saw one exit, dropped cachedBase, read state after mutation, pinned a home path. After: all exits, cachedBase kept, state at entry, paths scrubbed ```text BEFORE: First recorder - hooked 1 of 3 exits - cachedBase not saved - state read after mutation - home path in fixture AFTER: Now - cache-hit shape recorded - cachedBase replayed - state snapshotted at entry - ROOT/HOME scrubbed both ends ``` The second round was more humbling. I ran six turn shapes through capture. Two were new: a scoped workbench turn, which pins the fix that got five operator-written instruction sets to reach a model at all, and the cache-hit reuse. Three came back byte-identical to fixtures we already had. That was a finding. Channel envelopes, tool results and document attachments are written into the user message body by a separate function, outside the assembler, and the recorder only hooks the assembler. The gate can't see those lanes. I wrote that down in the plan and didn't fake a fixture for them. ## The invariant: every function that writes model-visible bytes has a recorder at every return Here it is as something you can check. For each function that writes bytes a model will read, count its return paths. Every one of them has to pass through the recorder, and the recorder has to capture the function's inputs as they were on entry, including any cache it was handed. If a code path writes prompt bytes and no recorder covers it, a prompt change on that path ships without a red build. A snapshot suite can have 100% passing tests and still cover one exit out of three. ## Check your own prompt builder in five minutes You need three things: the function that builds your messages, a way to wrap it, and your test runner. In Python: ```python # record.py: wrap your builder; run your app normally for a few turns import json, copy, functools, os, hashlib def record(fn): @functools.wraps(fn) def wrapper(*args, **kwargs): snap = copy.deepcopy({"args": args, "kwargs": kwargs}) # state at ENTRY out = fn(*args, **kwargs) blob = json.dumps({"input": snap, "output": out}, default=str, sort_keys=True) blob = blob.replace(os.path.expanduser("~"), "") name = hashlib.sha1(blob.encode()).hexdigest()[:10] open(f"fixtures/{fn.__name__}-{name}.json", "w").write(blob) return out return wrapper ``` Then count exits and compare them with what you actually recorded: ```bash grep -n "return" path/to/prompt_builder.py | wc -l # exits you have ls fixtures/ | wc -l # shapes you recorded grep -rn "messages.append\|role.*user" src/ | grep -v prompt_builder # writers outside the builder ``` Write a replay test that loads each fixture, calls the builder with its input, and does `assert out == fixture["output"]` using a first-diff message like the one above. Change one character in a template string and run it: ```bash set -o pipefail; pytest tests/test_replay.py | tee out.txt; echo "exit=$?" ``` Passing looks like this: `exit=1` with a char offset while the byte is changed, then `exit=0` after you revert. Failing looks like any of these: `exit=0` with the byte changed (your gate doesn't gate), more `return` statements than distinct fixture shapes (exits nobody recorded), or matches from the third grep (prompt writers the recorder never sees). My first version failed on two of those three. ## Agent CI tools diff behavior, and the prompt bytes come before behavior Good tools already exist. [agentdelta](https://github.com/sandeep-alluru/agentdelta) diffs the reasoning path between runs, [agentprdiff](https://github.com/vnageshwaran-de/agentprdiff) snapshots behaviors like which tool gets called, and [Tracecase](https://github.laiyagushi.com/AgentPostmortem/Tracecase) records runs and fails the merge when a case regresses. The [record, replay, assert write-up on DEV](https://dev.to/galian/record-replay-assert-testing-llm-agents-in-ci-without-paying-for-every-run-4dfp) makes the right call to cut the seam at the SDK call. The [Replay Divergence pattern](https://www.agentpatternscatalog.org/patterns/replay-divergence/) warns that an LLM consumer isn't deterministic, which is why this gate never calls a model. It needs no API key. What they don't talk about is recorder coverage. Seam placement gets discussed as a single point, but a prompt is often built by several writers with several exits, and a recorder that sits at one of them makes a green suite about part of the prompt. Behavioral diffs also can't tell you which byte moved. A deterministic byte gate runs first, costs nothing, and names the offset. ## Still blind: three user-body lanes Tool results, channel envelopes and document attachments still get written outside the function the recorder hooks. A prompt change in those lanes will pass CI today. Skill-turn and eviction shapes aren't recorded yet either. The suite is 5 of 5 green, and it covers what it covers, nothing more. ## Why memory injected into prompts needs a byte-level gate I built this gate because Vodou's main job is putting your memory into model calls. Memory is only yours if you can see exactly what bytes it becomes, and a lane that silently stops firing doesn't change one byte of the rest of the prompt. That's why the lane set is part of the contract. For you, that means your memory lives in a local database on your machine, and nothing leaves it until a turn sends it. The same memory rides into ChatGPT, Claude and Gemini in the browser through the Vodou Bridge extension, and into Claude Code, Cursor and VS Code through MCP and hooks. You don't keep notes for it: facts get extracted from your conversations on their own. When a correction arrives, it supersedes the stale fact instead of sitting next to it, and a precision floor injects nothing rather than noise. The work around it is governed. Permissions, approvals and an audit trail mean nothing outward-facing happens without your say-so, and a scheduler runs the rest while you're away. This post was mined from memory, drafted, graded, scanned by a redaction gate and deployed on that scheduler. The client side is open source: this replay suite and its fixtures sit in the public console server, and you can extend any skill, MCP server or schedule. Vodou is for engineers who run AI across more than one tool and are tired of re-explaining themselves to each one. Start at [vodou.ai](https://vodou.ai). If you want your memory on your own machine, reaching every model you use, built by someone who fails CI on a single byte of what those models get told, you can sign up at [vodou.ai](https://vodou.ai/register?utm_source=blog&utm_medium=feature&utm_campaign=replay-p1-a-prompt-change-now-fails-ci-with-a-diff). --- Source: [Your prompt snapshot test pins what you believed, not what ran](https://blog.vodou.ai/recorded-prompt-replay-ci-gate/) by Chad Priest, from Building Vodou in Public.