Under a hard project filter, a wrong memory tag is data loss

Sooner or later, every agent memory layer gets a project boundary. Client A’s pricing should never show up in Client B’s chat. The read side of that boundary is easy to write and easy to test. You add WHERE project_id = ? and a unit test that proves the other client’s row is gone. The write side is where it breaks. Something has to decide which project a memory belongs to, and it usually decides from a batch of mixed input, well after the conversation ended.

I got this wrong in production on the same day I shipped the filter.

A SQLite vault ranked on project, provenance and duplicates

I built this into Vodou, which keeps a person’s AI memory in a local SQLite vault on their own machine. Conversations get distilled into short tagged bullets. Those bullets are stored as chunks with an FTS5 index and an embedding column. Search runs keyword and meaning together, reranks, then weights each result by where it came from. Nothing leaves the machine until you send it. People use it from the Console Memory tab or from vodou-core mem search.

On top of plain relevance sit three axes, and each behaves differently on purpose. Project is a hard wall. A chunk tagged with another project is dropped, and a chunk with no project (NULL) is global and passes everywhere. Preferences always stay global because they describe the person, not the client. Provenance is a soft multiplier: first-party memory scores 1.0, sessions captured from other tools like Cursor and Claude Code score 0.925, and one-shot imports score 0.85. Duplicates get a soft demotion. Near-identical chunks form a fact group, one copy is canonical, and the rest score ×0.4. Neither soft axis deletes anything. The design notes are public in docs/vodou-memory.md.

One tagged prompt in a flush window stamped every bullet with its project

The first version of project tagging reused a rule I already had for scope. Prompts from a terminal or an IDE go into a buffer, and a later flush extracts bullets from the whole window at once. Scope used a dominant-value rule: the bullets got whatever scope most entries carried. That works for scope. Scope is only a 2x soft boost, so a wrong guess moves a fact down a list.

I copied that rule to project. Then a flush window mixed one prompt from a project web chat with a run of untagged Claude Code prompts from the terminal. The tagged entry held the only project value in the window, so it won the vote, and every bullet from that window got stamped with that project. The hard filter was on by default, so those memories vanished from every other context. Terminal work that had nothing to do with the project could now only be seen inside it.

That changed my mind about the whole design. Under a soft boost, a false-positive tag is a ranking error. Under a hard filter, it hides the memory. That is strictly worse than no tag, because an untagged memory still shows up. Weeks earlier I had logged a decision to defer project isolation unless cross-project bleed became a problem people actually felt. I built it anyway, and the first bug was the opposite of bleed. It was loss.

The fix shipped the same day. A window’s bullets get a project only when every entry in the window carries the same one. One untagged entry or a second project, and the bullets stay global. Scope kept its dominance rule because it is a boost, not a wall. Web chat tagging never had this problem, since it maps one conversation to one project per extraction cycle and there is nothing to mix.

Newest-wins would have handed canonical copies to a bulk import

A week later I made a second wrong call, this time in dedup. Clustering near-duplicates uses two gates, and both must pass: embedding cosine at or above 0.88, and token Jaccard at or above 0.40 after header lines are stripped. With either gate alone, facts that were not the same got collapsed together. Each cluster then elects a canonical copy.

My first instinct was to let the newest copy win. But created_at on an imported chunk is the import date, not the day the words were said. A bulk import of a year of old chat history would look newer than everything native. It would take canonical status away from memory the person built through real use. The election now ranks provenance first (pinned, first-party, capture, import), then created_at, then length. Contradiction resolution changed at the same time. The losing chunk is now superseded under the winner, reversibly, instead of deleted.

Entity resolution had a smaller trap of the same kind. I pulled Titlecase name bigrams out with a regex, and a regex consumes its match. “Met Jane Doe” yields “Met Jane” and swallows “Jane Doe”. I replaced it with a token window. Surname aliases got a guard too: if a token also appears lowercase somewhere in the corpus, it is a common noun and cannot become an alias.

A key behind a hard filter must be written only on unanimous evidence

Here is the failure class, stated as a property you can check:

Every write path that assigns a key a read path hard-filters on must write the unfiltered value (NULL, global) in the ambiguous case. And a timestamp used to rank copies must mean the same thing for every source that writes it.

Each half is simply true or false of a given codebase. A dominance vote, “last tag seen”, “first non-null in the batch”: each one fails the first half. A created_at that is ingest time for one source and event time for another fails the second. What makes it subtle is that one rule can be right for one key and wrong for another inside the same function. The only difference is what the reader does with the key.

A corollary I also had to enforce: the filter must hold on the degraded path. If embeddings are unavailable and search falls back to keyword only, that path needs the same project predicate. Otherwise a process running without its embedding model leaks across clients.

Find memories tagged from a mixed batch in one SQL query

You can run this against your own system in five minutes. You need the table your memories live in and whatever records the inputs that produced them (messages, events, a buffer log). Adjust the names:

-- Memories that carry a project but came from a batch whose inputs did not agree
SELECT m.id,
       m.project_id AS tagged_as,
       COUNT(DISTINCT COALESCE(e.project_id, '<untagged>')) AS distinct_input_projects
FROM memories m
JOIN source_events e ON e.batch_id = m.source_batch_id
WHERE m.project_id IS NOT NULL
GROUP BY m.id, m.project_id
HAVING COUNT(DISTINCT COALESCE(e.project_id, '<untagged>')) > 1;

A pass is zero rows. Every row in a failing result is a memory your filter hides from places it belongs. If you can’t write this query because memories don’t record which batch produced them, that is your finding: you have no way to audit the label.

Next, check the read predicate:

# Filters on project that may have forgotten the global carve-out
grep -rnE "project_id\s*=\s*[?:$]" src/ | grep -v "IS NULL"

Read every hit. If there’s no OR project_id IS NULL next to it, global memories quietly disappear inside projects. Last, plant a sentinel fact in project B. Search for it verbatim from project A with vector search turned off (stop the embedding service or remove its key), and confirm you get zero results. A hit means your isolation only exists on the vector path.

Row-level security guards the read, not the label

Most existing work here covers the read side, and it is good. lucidmem makes the project the unit of access and enforces that in Postgres instead of application code, which is the right place for the wall. The agent-gateway memory design treats memory as gateway infrastructure with explicit, auditable tenant boundaries and per-route sharing rules. projectmem avoids the question by keeping memory in a directory inside each repository, so the filesystem is the isolation.

None of them defends against a correct wall around a wrong label. Enforcing in the database makes a mislabeled row more reliably invisible, not less. The standard pipeline advice also makes mixed batches more likely. LangChain’s memory guide describes a background process that extracts and generalizes after the fact. The Geodocs memory spec says to consolidate into long-term memory at session end. Both are sound, and both put extraction in a batch, which is exactly where one tagged input can speak for its untagged neighbors. Deciding the label needs the same rigor as enforcing the filter. A directory per repo sidesteps the problem, but it leaves no obvious home for facts about the person that should follow them everywhere.

Unanimity fails open, and open means global

The unanimity rule has a cost I haven’t removed. A mixed window resolves to global. So real project work flushed in a window that also held a single terminal prompt lands outside the wall and can surface in another client’s recall. I chose that direction on purpose: a leaked fact is visible and can be moved, while a hidden one never gets noticed. It is still leakage, and today the only thing limiting it is that most windows aren’t mixed. Dedup is also still a scan you run on demand, not a continuous process. Encryption at rest for the vault is a draft and has not shipped.

If you want memory from every AI tool you use kept in a SQLite file on your own machine, weighted by where each fact came from and walled off per project without hiding what belongs everywhere, that is what vodou.ai gives you.