Multi-tenant isolation: the requirement that arrives after the demo

The pattern

The demo runs on one customer's documents. It works. Then the second customer is onboarded, and someone asks the question that should have been asked first: can a user of tenant A ever see a chunk belonging to tenant B?

The most common answer at that point is a filter applied to the retrieved results before they reach the prompt. It looks correct in testing and it is structurally wrong.

Why post-filtering is not isolation

Approximate nearest neighbour search returns the k closest vectors. If you then remove the ones the user may not see, you are left with fewer than k results — and the ones you removed were, by construction, among the most semantically similar. Under a low-permission query the retriever silently degrades to its worst case, and quality becomes a function of how much the user is allowed to see. That is a terrible property to ship.

Worse, the removed content still crossed a trust boundary: it was fetched, scored, and held in memory inside a process that will later render a prompt. A security reviewer is entitled to object to that even if the user never sees it.

What to do instead

Constrain the search. Push the permission predicate into the vector store's filtering layer so that disallowed chunks are never candidates, and keep k honest: you get k results the user is actually allowed to see. This is precisely why payload filtering is a core capability rather than a wrapper in the stores built for this use case.

Then decide where the predicate comes from. Resolving a user's full document ACL on every query is expensive, so most systems precompute a group or label set per user and filter on that. The tradeoff is staleness: a revocation takes effect at the next refresh. Make that window explicit and short enough that you can defend it.

The three things reviewers ask for

First, isolation that holds when the index is corrupt or partially rebuilt — fail closed, not open. Second, an audit trail of what was retrieved for each answer, not just what was generated. Third, per-tenant encryption or a documented argument for why logical separation is sufficient. Have those three answers ready and the review goes from weeks to a meeting.