building vodou.

Scrubbing recorded fixtures by regex misses user-made data

Recording test fixtures from a live system captures user-made data regex scrubbers cannot see. Allowlist by origin instead, and check it with one diff.

Chad Priest / / 7 min read

On September 1 I found a test fixture in my repo holding 41KB of my own memory. It was a recorded turn for a prompt-replay test. The server that builds the LLM prompt had assembled a full system prompt for one real chat, and I had frozen it as JSON so CI would fail with a diff whenever a prompt change moved it. That assembled prompt includes the assistant’s memory of me. So the fixture held my name, my email, four domains I own, my home address down to the ZIP, family details, the dog’s name, and a private calendar import address.

One of ten private items caught, by a pattern list working as designed

Before anything in the repo goes public, a scan checks the tree against a list of PII patterns. I had been treating that scan as the wall between my memory and GitHub. Counting the private items in the fixture gets you about ten: name, email, four domains, address, family details, the dog, the calendar address. The scan flagged one of them, the email domain.

The scan wasn’t broken. The list is short on purpose, because long pattern lists fire on ordinary prose. A family member’s first name has no shape a regex can know, and neither does a dog’s. Pattern matching finds identifiers that look like identifiers. Most of what made that file private was plain sentences.

Redacting a fixture changes what it tests

Scrubbing the file was harder than it should have been. The fixture records the assembled prompt and asserts its size:

assert len(assembled_prompt) == fixture["chars"]  # 40449

If I had replaced each private string with [REDACTED], the length would have changed. That leaves two bad options. The test fails, or I rewrite chars to fit, and from then on the test compares a scrubbed prompt with a number taken from that same scrubbed prompt. It still goes green, but it no longer pins anything the live system ever produced. I ended up doing a length-preserving substitution by hand so the assertion stayed honest.

That is the second argument against scrubbing, and in my view the stronger one. A replay fixture is valuable because it is exactly what happened. Every redaction turns it into something close to what happened, and nothing tells you when “close” stops being good enough. Dropping a record from a fixture is a decision you can see. Rewriting one quietly changes what the test measures.

A name prefix was the only provenance a recorded skill list had

Recorded prompts were only the first case. Next I looked at a snapshot of the assistant’s skill list. Skills here are reusable instruction files the assistant can run. A list like that looks safe to freeze, and most entries are: they ship publicly under skills/ in the open repo, so recording them leaks nothing. But the same list also holds 34 skills that the system wrote for itself after watching me repeat a workflow. Their names are bland (auto-weekly-recap). Their descriptions and bodies are distilled from my activity: what I repeat, when, and against which accounts.

I had the question wrong. I was asking “does this row contain PII?” The question that matters is “did this row come from the repo, or from the user?” A scrubber can answer the first. Only origin answers the second.

Those generated skills have no column saying where they came from. The only origin marker is the name prefix, and it exists by accident: the generator forbids the model from choosing a name that starts with “auto”, so the system adds the prefix itself. A single refactor of that naming rule would have wiped out the only way to tell my data apart from the product’s. The transferable fix is to record origin as a field when the row is written, even if today a naming convention seems to cover it. A prefix is provenance by accident.

Recorded fixtures mix three kinds of data

Any fixture recorded from a running system holds a mix, and nothing in the recording step separates it:

  1. Repo-shipped: artifacts your project publishes, such as built-in skills, tool definitions and prompt templates. Keep them exactly as recorded.
  2. Third-party-controlled: response shapes and reference data from someone else’s API, such as a Stripe charge object or a weather payload. Ordinary cassettes of these are fine. Scrub the credentials and IDs, which have a shape. This is the case the standard advice was written for.
  3. User-generated: anything a person authored or a system derived from what a person did, such as memory lines, generated skills, chat text, notes, and free-text fields inside a third-party response. Exclude it. Don’t scrub it.

The third class hides inside the other two. It turns up in VCR or Polly.js cassettes of an API whose responses carry user-written text, in LLM eval sets sampled from production traces in LangSmith or Langfuse, in Jest snapshots of an agent’s tool registry that includes user-created tools, and in open-source repos that seed fixtures from a developer’s local SQLite file.

Illya Moskvin’s post on redacting VCR.py cassettes is right that cassettes are meant to be committed and that VCR.py will not clean them for you. opendreams’ anonymize.py is a careful scrubber: it detects the host username, then rewrites paths, emails and key prefixes. Both work well on class 2. Neither can work on class 3, where the private part is prose.

A committed fixture may hold repo-shipped records and third-party records with their identifiers scrubbed; a record that users generated is excluded, not redacted.

Checking your own fixtures

This takes two checks, because registries and free text leak in different ways.

The first check finds free text: any recorded string big enough to be a rendered prompt, a transcript or a memory dump. My 41KB prompt was a single string, so this is the check that would have caught it.

# every string field over 4 KB, with file, path and length
find tests/fixtures -name '*.json' -exec jq -r '
  paths(type == "string" and length > 4096) as $p
  | "\(input_filename)\t\($p | map(tostring) | join("."))\t\(getpath($p) | length)"
' {} +

Passing output is empty. Failing output looks like tests/fixtures/turn.json messages.0.content 40449. False positives will be large third-party bodies, such as an HTML page or a base64 image in a cassette. Open each hit. If it is someone else’s API payload, it belongs to class 2. If it reads like a person, it belongs to class 3.

The second check finds user-made entries in registry-shaped fixtures, meaning lists of named records. Scope the jq path to the list you actually recorded. A bare .. | .name? also returns JSON-schema property names, tool parameter names, header names and model names, which buries the real hits. A typical fixture shape:

{ "skills": [ { "name": "summarize-pdf", "description": "..." },
              { "name": "auto-weekly-recap", "description": "..." } ] }
# names in the recorded registry (change .skills[] to wherever yours lives)
find tests/fixtures -name '*.json' -exec jq -r '.skills[]?.name // empty' {} + \
  | sort -u > /tmp/fixture-names

# names your repo actually ships
git ls-files 'skills/*/SKILL.md' | cut -d/ -f2 | sort -u > /tmp/shipped-names

# recorded but never shipped
comm -23 /tmp/fixture-names /tmp/shipped-names

Passing output is empty. Failing output lists names like auto-weekly-recap or my-invoice-helper. A false positive is a shipped skill that was renamed after the recording, or one registered under an id that differs from its directory name. Re-record or fix the glob for those. For every other hit, read the description. If it reads like a person’s week, no pattern list was ever going to catch it. This diff only covers registry-shaped fixtures. Free text is what the first check is for.

Drop class 3 records in VCR.py or Polly.js beforePersist hooks

Both checks run after the fact, on files that already exist. The better place for the allowlist is the recorder, so a class 3 record never reaches disk. VCR.py already gives you the hook:

import json, pathlib, vcr

SHIPPED = {p.parent.name for p in pathlib.Path("skills").glob("*/SKILL.md")}

def refuse_user_made(response):
    try:
        data = json.loads(response["body"]["string"])
    except ValueError:
        return response
    if isinstance(data, dict):
        for skill in data.get("skills", []):
            if skill.get("name") not in SHIPPED:
                raise ValueError(f"refusing to record non-shipped skill {skill.get('name')!r}")
    return response

recorder = vcr.VCR(before_record_response=refuse_user_made)

Polly.js gives you the same point through a beforePersist handler, and Jest through a custom snapshot serializer that throws. In each case the recording fails loudly instead of committing someone’s week.

Scrubbers are a second line. The first line is an allowlist by origin, enforced in the recorder, because the next recording will capture whatever the system knows that day.