# Your memory recall test is grading the extractor and the chat window

> Memory recall tests lie two ways: the extractor drops fake-looking canaries, and chat history answers for retrieval. Here is how to check your own harness.

- Author: Chad Priest
- Published: 2026-09-18
- Canonical URL: https://blog.vodou.ai/your-memory-recall-test-is-grading-the-extractor-and-the-chat-window/
- Tags: ai, llm, testing, rag

---

My memory recall test gave me two wrong answers, and neither was about retrieval. In the first, the canary never reached the store, so the test failed and I blamed ranking. In the second, the test passed while retrieval did nothing, because the fact was still in the chat window. Both come from the same bug: **the harness was measuring itself instead of the memory store.** Your test can fail both ways if two things are true: a model decides what gets written to memory, and a conversation buffer sits beside retrieval. I'm not going to hand you a list of products with that shape. I haven't checked them one by one, so check yours against those two conditions.

## Failure one: the extractor skipped the line that sounded like a test

I planted a batch of facts in one session. The ordinary ones came out of extraction as memories. The canary didn't. It was a line of the form "my test project codename is …". Nothing errored and nothing logged a rejection. The row just wasn't there. I spent time on ranking before I checked whether the row existed at all.

What I have is one observation. I'm not printing the full string here, so you can't rerun my exact case, and I have no survival rate and no repeat count to give you. My guess is that the extractor made an editorial call: the line announced itself as a test, so it wasn't worth keeping. That's a guess. I didn't prove it.

It also isn't a finding about high-entropy markers. My dropped canary wasn't high-entropy. It was plain words that said "test." The usual advice, for example in [Testing Agent Memory for Cross-User Leakage](https://qaskills.sh/blog/testing-agent-memory-cross-user-leakage), is to use "high-entropy synthetic markers." That advice is written for stores that keep everything you write. I haven't measured what an LLM extractor does with random tokens. If you care, the test is cheap: put ten random-token markers and ten plausible facts through the same extractor, run it several times, and count what survives. Until someone does that, the safe move is a canary that is **plausible but unique**. It should be something a person would actually say, with one detail nobody would guess. "Billing jobs moved to 4:15 on Tuesdays" works. "4:15" won't be in your store by accident, and no model's priors will produce it.

## Failure two: the chat window answered for retrieval

I asked the recall question in the same conversation where I planted the fact, and it passed. It would have passed with the store empty. I only noticed when I read the retrieved items my gateway returned alongside the answer, not the answer itself. Retrieval hadn't supplied the fact. The conversation had.

The same-thread buffer is only the most obvious way a fact can leak into the answer without retrieval. Your probe also has to rule out:

- **Rolling summaries** of earlier turns, which some stacks carry across threads.
- **Profile or "about the user" blocks**, which are built from memory once and then injected on every turn without a search.
- **Provider-side memory**, if the chat model's vendor keeps its own memory of this user or account.
- **Caches or a second index**, such as a vector index kept apart from the SQL table, or a response cache keyed on the question.
- **Model priors.** If the answer is guessable, a correct answer proves nothing. That's the other reason for "4:15."

## Store before probe, probe fails when the canary is absent

**Invariant 1: the fact is in the store before you probe.** If it isn't, a failed recall is measuring the extractor.

**Invariant 2: the probe fails when the fact is absent.** Remove the fact from every layer and hold everything else constant. If the probe still passes, it's measuring history, summaries, or priors.

A recall pass counts only when both hold.

## Plant the canary, count rows, probe: the steps against your own store

Every step talks to your store's own API or your own chat endpoint. None of it needs my stack.

| Step | Contract | Enforces |
|---|---|---|
| 0 | Count matching rows in the store before planting (the baseline) | 1 |
| 1 | Plant the canary through your normal chat path, then wait for extraction | 1 |
| 2 | Query the store directly, both literally and semantically, and compare to the baseline | 1 |
| 3 | Ask from a **new thread ID** and read the retrieved items, not the prose | retrieval |
| 4 | Delete the fact from the store, the index, and any summary, profile, or cache layer, then rerun step 2 and expect the baseline | 2 |
| 5 | Ask again from another new thread and expect retrieval to miss and the model not to know | 2 |

Extractors paraphrase, so "4:15 on Tuesdays" may be stored as "Tue 16:15" or "quarter past four, weekly." Match on several spellings **and** on semantic search. A zero on the literal match means you should read the rows by hand. By itself it isn't a verdict.

**Mem0**, as a direct store check:

```python
from mem0 import Memory
m = Memory()
USER = "canary-user"
SPELLINGS = ["4:15", "16:15", "quarter past four"]

def rows():
    r = m.get_all(user_id=USER)
    return r["results"] if isinstance(r, dict) else r   # newer versions wrap results

def hits():
    return [x for x in rows() if any(s in x["memory"].lower() for s in SPELLINGS)]

baseline = len(hits())                                                     # step 0
m.add([{"role": "user", "content": "Heads up, I moved our billing jobs to 4:15 on Tuesdays."}],
      user_id=USER)                                                        # step 1
print("delta:", len(hits()) - baseline)                                    # step 2: expect >= 1
print(m.search("when do billing jobs run", user_id=USER))                  # step 2: semantic
# step 3: call YOUR chat endpoint with a fresh thread ID; log what it retrieved
for x in hits():
    m.delete(memory_id=x["id"])                                            # step 4
print("delta after delete:", len(hits()) - baseline)                       # expect 0
print(m.search("when do billing jobs run", user_id=USER))                  # expect no billing fact
# step 5: fresh thread ID again; expect no retrieved billing fact and "I don't know"
```

**Chroma**, if your extractor writes to a collection:

```python
import re
SPELL = re.compile(r"4:15|16:15|quarter past four", re.I)
def hits(col):
    got = col.get(include=["documents"])
    return [i for i, d in zip(got["ids"], got["documents"]) if SPELL.search(d or "")]

baseline = len(hits(col))                                                  # step 0
# step 1: plant through your chat path, wait for extraction
print("delta:", len(hits(col)) - baseline)                                 # step 2
print(col.query(query_texts=["when do billing jobs run"], n_results=5))    # step 2: semantic
col.delete(ids=hits(col))                                                  # step 4
print("delta after delete:", len(hits(col)) - baseline)                    # expect 0
```

**Postgres with pgvector**, if memories are rows (use your own table and column names):

```sql
-- steps 0 and 2: run before and after planting, compare
SELECT count(*) FROM memories WHERE content ~* '(4:15|16:15|quarter past four)';
-- step 2, semantic: embed the question with the same model your app uses
SELECT id, content FROM memories ORDER BY embedding <=> :question_embedding LIMIT 5;
-- step 4
DELETE FROM memories WHERE content ~* '(4:15|16:15|quarter past four)';
-- then rerun the count and expect the baseline
```

In step 4, deleting the SQL row is the easy part. If embeddings live in a separate vector store, delete them there too. If your stack keeps a summary, a profile, or a response cache, clear or regenerate it, and turn off provider-side memory for the test account. Otherwise step 5 is testing the layer you forgot.

A pass: step 2's delta is at least 1, step 3 shows the fact among the retrieved items and answers "4:15," step 4 brings the count back to baseline, and step 5 retrieves nothing about billing and the model says it doesn't know. A fail on invariant 1: step 2's delta is 0 and reading the rows by hand turns up nothing, which means your extractor filtered the canary. A fail on invariant 2: step 5 still answers "4:15," which means something other than your store is answering. If your stack can't tell you which items it retrieved, add that before you write another eval. Without it you can't separate retrieval from the chat window, and I couldn't either.

---

Source: [Your memory recall test is grading the extractor and the chat window](https://blog.vodou.ai/your-memory-recall-test-is-grading-the-extractor-and-the-chat-window/) by Chad Priest, from Building Vodou in Public.