# Under a hard project filter, a wrong memory tag is data loss > Per-project agent memory broke on the write side: one tagged prompt labeled a whole batch and hid it. The unanimity rule, a SQL check, and what still fails open. - Author: Chad Priest - Published: 2026-09-15 - Canonical URL: https://blog.vodou.ai/memory-project-filter-wrong-tag-hides-facts/ - Tags: memory, ai-agents, retrieval, architecture, llm --- Sooner or later, every agent memory layer gets a project boundary. Client A's pricing should never show up in Client B's chat. The read side of that boundary is easy to write and easy to test. You add `WHERE project_id = ?` and a unit test that proves the other client's row is gone. The write side is where it breaks. Something has to decide which project a memory belongs to, and it usually decides from a batch of mixed input, well after the conversation ended. I got this wrong in production on the same day I shipped the filter. ## A SQLite vault ranked on project, provenance and duplicates I built this into [Vodou](https://vodou.ai/register?utm_source=blog&utm_medium=capability&utm_campaign=local-memory), which keeps a person's AI memory in a local SQLite vault on their own machine. Conversations get distilled into short tagged bullets. Those bullets are stored as chunks with an FTS5 index and an embedding column. Search runs keyword and meaning together, reranks, then weights each result by where it came from. Nothing leaves the machine until you send it. People use it from the Console Memory tab or from `vodou-core mem search`. On top of plain relevance sit three axes, and each behaves differently on purpose. Project is a hard wall. A chunk tagged with another project is dropped, and a chunk with no project (NULL) is global and passes everywhere. Preferences always stay global because they describe the person, not the client. Provenance is a soft multiplier: first-party memory scores 1.0, sessions captured from other tools like Cursor and Claude Code score 0.925, and one-shot imports score 0.85. Duplicates get a soft demotion. Near-identical chunks form a fact group, one copy is canonical, and the rest score ×0.4. Neither soft axis deletes anything. The design notes are public in `docs/vodou-memory.md`. ![Infographic: your AI memory is a file you own. Import, control and portability in a local Vodou vault.](https://blog.vodou.ai/img/infographics/vodou-owns-your-memory.jpg) **Diagram: Where a memory's project comes from, and where it is enforced** Three write paths feed a tag decision that stamps a project only on unanimous evidence, into a local SQLite vault; recall in project A returns project A plus global memories and never project B ```text [Web chat in a project] --> [Tag decision] [Terminal and IDE prompts] --> [Tag decision] [Imports and captures] --> [Tag decision] [Tag decision] --project_id or NULL--> [Local SQLite vault] [Local SQLite vault] --hard filter, NULL passes--> [Recall inside project A (fixed)] notes: Web chat in a project: one conversation, one project Terminal and IDE prompts: buffered, flushed in windows Imports and captures: lower trust tier Tag decision: unanimous, or global Local SQLite vault: FTS5 + embeddings + project_id Recall inside project A: A plus global, never B The wall is at recall. The bug was at the tag decision. ``` ## One tagged prompt in a flush window stamped every bullet with its project The first version of project tagging reused a rule I already had for scope. Prompts from a terminal or an IDE go into a buffer, and a later flush extracts bullets from the whole window at once. Scope used a dominant-value rule: the bullets got whatever scope most entries carried. That works for scope. Scope is only a 2x soft boost, so a wrong guess moves a fact down a list. I copied that rule to project. Then a flush window mixed one prompt from a project web chat with a run of untagged Claude Code prompts from the terminal. The tagged entry held the only project value in the window, so it won the vote, and every bullet from that window got stamped with that project. The hard filter was on by default, so those memories vanished from every other context. Terminal work that had nothing to do with the project could now only be seen inside it. That changed my mind about the whole design. Under a soft boost, a false-positive tag is a ranking error. Under a hard filter, it hides the memory. That is strictly worse than no tag, because an untagged memory still shows up. Weeks earlier I had logged a decision to defer project isolation unless cross-project bleed became a problem people actually felt. I built it anyway, and the first bug was the opposite of bleed. It was loss. The fix shipped the same day. A window's bullets get a project only when every entry in the window carries the same one. One untagged entry or a second project, and the bullets stay global. Scope kept its dominance rule because it is a boost, not a wall. Web chat tagging never had this problem, since it maps one conversation to one project per extraction cycle and there is nothing to mix. **Diagram (beforeafter)** Before: a dominance vote let one project-tagged prompt stamp a mixed window, hiding its memories everywhere else. After: unanimity leaves a mixed window global and visible in every project ```text BEFORE: Dominance, copied from scope - one project-tagged prompt, several untagged - the only tag wins the vote - every bullet stamped with the project - hard filter hides them everywhere else AFTER: Unanimity - same mixed window - entries disagree - bullets stay global (NULL) - visible in every project ``` ## Newest-wins would have handed canonical copies to a bulk import A week later I made a second wrong call, this time in dedup. Clustering near-duplicates uses two gates, and both must pass: embedding cosine at or above 0.88, and token Jaccard at or above 0.40 after header lines are stripped. With either gate alone, facts that were not the same got collapsed together. Each cluster then elects a canonical copy. My first instinct was to let the newest copy win. But `created_at` on an imported chunk is the import date, not the day the words were said. A bulk import of a year of old chat history would look newer than everything native. It would take canonical status away from memory the person built through real use. The election now ranks provenance first (pinned, first-party, capture, import), then `created_at`, then length. Contradiction resolution changed at the same time. The losing chunk is now superseded under the winner, reversibly, instead of deleted. Entity resolution had a smaller trap of the same kind. I pulled Titlecase name bigrams out with a regex, and a regex consumes its match. "Met Jane Doe" yields "Met Jane" and swallows "Jane Doe". I replaced it with a token window. Surname aliases got a guard too: if a token also appears lowercase somewhere in the corpus, it is a common noun and cannot become an alias. **Diagram (timeline)** Build history from July 2 to August 17: hard project filter ships, dominance tagging hides memories, unanimity fix the same day, provenance-first fact groups and token-window entities on July 11, IDE capture July 12, page axis August 17 ```text Jul 2 -> hard project filter ships Jul 2 -> dominance tagging hides memories (problem) Jul 2 -> unanimity rule, same day (fixed) Jul 11 -> fact groups: provenance before recency (fixed) Jul 11 -> entity aliases: token window replaces regex (fixed) Jul 12 -> Cursor and Claude Code sessions land at capture tier Aug 17 -> page axis: per-site modes and forget ``` ## A key behind a hard filter must be written only on unanimous evidence Here is the failure class, stated as a property you can check: Every write path that assigns a key a read path hard-filters on must write the unfiltered value (NULL, global) in the ambiguous case. And a timestamp used to rank copies must mean the same thing for every source that writes it. Each half is simply true or false of a given codebase. A dominance vote, "last tag seen", "first non-null in the batch": each one fails the first half. A `created_at` that is ingest time for one source and event time for another fails the second. What makes it subtle is that one rule can be right for one key and wrong for another inside the same function. The only difference is what the reader does with the key. A corollary I also had to enforce: the filter must hold on the degraded path. If embeddings are unavailable and search falls back to keyword only, that path needs the same project predicate. Otherwise a process running without its embedding model leaks across clients. ## Find memories tagged from a mixed batch in one SQL query You can run this against your own system in five minutes. You need the table your memories live in and whatever records the inputs that produced them (messages, events, a buffer log). Adjust the names: ```sql -- Memories that carry a project but came from a batch whose inputs did not agree SELECT m.id, m.project_id AS tagged_as, COUNT(DISTINCT COALESCE(e.project_id, '')) AS distinct_input_projects FROM memories m JOIN source_events e ON e.batch_id = m.source_batch_id WHERE m.project_id IS NOT NULL GROUP BY m.id, m.project_id HAVING COUNT(DISTINCT COALESCE(e.project_id, '')) > 1; ``` A pass is zero rows. Every row in a failing result is a memory your filter hides from places it belongs. If you can't write this query because memories don't record which batch produced them, that is your finding: you have no way to audit the label. Next, check the read predicate: ```bash # Filters on project that may have forgotten the global carve-out grep -rnE "project_id\s*=\s*[?:$]" src/ | grep -v "IS NULL" ``` Read every hit. If there's no `OR project_id IS NULL` next to it, global memories quietly disappear inside projects. Last, plant a sentinel fact in project B. Search for it verbatim from project A with vector search turned off (stop the embedding service or remove its key), and confirm you get zero results. A hit means your isolation only exists on the vector path. ## Row-level security guards the read, not the label Most existing work here covers the read side, and it is good. [lucidmem](https://github.com/libr3andr3/lucidmem) makes the project the unit of access and enforces that in Postgres instead of application code, which is the right place for the wall. The [agent-gateway memory design](https://github.com/agent-guide/agent-gateway/blob/main/docs/design/memory.md) treats memory as gateway infrastructure with explicit, auditable tenant boundaries and per-route sharing rules. [projectmem](https://github.com/riponcm/projectmem) avoids the question by keeping memory in a directory inside each repository, so the filesystem is the isolation. None of them defends against a correct wall around a wrong label. Enforcing in the database makes a mislabeled row more reliably invisible, not less. The standard pipeline advice also makes mixed batches more likely. [LangChain's memory guide](https://www.langchain.com/blog/how-to-give-your-agent-memory) describes a background process that extracts and generalizes after the fact. The [Geodocs memory spec](https://geodocs.dev/ai-agents/agent-memory-pattern-spec) says to consolidate into long-term memory at session end. Both are sound, and both put extraction in a batch, which is exactly where one tagged input can speak for its untagged neighbors. Deciding the label needs the same rigor as enforcing the filter. A directory per repo sidesteps the problem, but it leaves no obvious home for facts about the person that should follow them everywhere. ## Unanimity fails open, and open means global The unanimity rule has a cost I haven't removed. A mixed window resolves to global. So real project work flushed in a window that also held a single terminal prompt lands outside the wall and can surface in another client's recall. I chose that direction on purpose: a leaked fact is visible and can be moved, while a hidden one never gets noticed. It is still leakage, and today the only thing limiting it is that most windows aren't mixed. Dedup is also still a scan you run on demand, not a continuous process. Encryption at rest for the vault is a draft and has not shipped. If you want memory from every AI tool you use kept in a SQLite file on your own machine, weighted by where each fact came from and walled off per project without hiding what belongs everywhere, that is what [vodou.ai](https://vodou.ai/register?utm_source=blog&utm_medium=capability&utm_campaign=local-memory) gives you. --- Source: [Under a hard project filter, a wrong memory tag is data loss](https://blog.vodou.ai/memory-project-filter-wrong-tag-hides-facts/) by Chad Priest, from Building Vodou in Public.