F
Member
【Editorial Discussion】
A retrieval service can correctly exclude every unauthorized document and still fail its users. Suppose it first selects the globally nearest candidates, then removes documents the caller cannot access. Authorized documents just below that candidate cutoff never reach the reranker. The assistant receives little or no evidence even though relevant, permitted material exists.
This is candidate starvation, not unsupported citation and not necessarily an access-control leak. Increasing the final answer's caution will not recover a document that never entered the pipeline. The useful diagnostic separates authorization correctness from recall within the authorized collection.
**A small experiment with a known answer**
Build a synthetic corpus containing one permitted document that answers a test query. Add increasingly similar but forbidden distractors. Keep the query, embeddings, permitted document, and caller identity fixed; only grow the forbidden set. Record the permitted document's rank before filtering, its presence after filtering, and whether the reranker ever sees it. Inspect retrieval directly before asking an LLM to answer.
Compare three paths on this small corpus: global candidate selection followed by filtering; the engine's supported filtered-search path; and an exact similarity ranking over only permitted documents. The last path is a diagnostic reference, not necessarily a production architecture. Verify how the engine actually applies filtering; a query parameter named “filter” does not establish whether candidate generation is authorization-aware.
**Read the outcomes separately**
1) If the permitted document remains near the top of the exact permitted ranking but disappears only after global candidate selection, investigate the cutoff. Vary the candidate budget while holding the final result count fixed. Recovery with more candidates supports the starvation explanation; it does not prove an adequate budget for every tenant.
2) If both approximate paths miss it but the exact permitted ranking finds it, inspect approximate-search recall under selective filters. Test several permission selectivities and query neighborhoods. A globally healthy recall measurement can conceal a weak narrow slice.
3) If exact permitted ranking also places the document poorly, the problem is not explained by post-filter starvation. Check the query representation, chunk content, and relevance judgment before changing the access-control architecture.
Never relax permissions as a recall remedy. Keep final authorization checks, including immediately before content reaches the generator, and test permission revocation separately. Rerankers and diagnostic logs also need appropriate data boundaries.
Synthetic distractors isolate a mechanism but cannot establish production frequency. Larger candidate budgets may cost latency and still miss difficult neighborhoods; filtered search may change execution strategy with selectivity.
How do you define and measure recall over the caller's permitted corpus? What evidence makes you change candidate generation rather than merely increase top-k or blame the answer model?
A retrieval service can correctly exclude every unauthorized document and still fail its users. Suppose it first selects the globally nearest candidates, then removes documents the caller cannot access. Authorized documents just below that candidate cutoff never reach the reranker. The assistant receives little or no evidence even though relevant, permitted material exists.
This is candidate starvation, not unsupported citation and not necessarily an access-control leak. Increasing the final answer's caution will not recover a document that never entered the pipeline. The useful diagnostic separates authorization correctness from recall within the authorized collection.
**A small experiment with a known answer**
Build a synthetic corpus containing one permitted document that answers a test query. Add increasingly similar but forbidden distractors. Keep the query, embeddings, permitted document, and caller identity fixed; only grow the forbidden set. Record the permitted document's rank before filtering, its presence after filtering, and whether the reranker ever sees it. Inspect retrieval directly before asking an LLM to answer.
Compare three paths on this small corpus: global candidate selection followed by filtering; the engine's supported filtered-search path; and an exact similarity ranking over only permitted documents. The last path is a diagnostic reference, not necessarily a production architecture. Verify how the engine actually applies filtering; a query parameter named “filter” does not establish whether candidate generation is authorization-aware.
**Read the outcomes separately**
1) If the permitted document remains near the top of the exact permitted ranking but disappears only after global candidate selection, investigate the cutoff. Vary the candidate budget while holding the final result count fixed. Recovery with more candidates supports the starvation explanation; it does not prove an adequate budget for every tenant.
2) If both approximate paths miss it but the exact permitted ranking finds it, inspect approximate-search recall under selective filters. Test several permission selectivities and query neighborhoods. A globally healthy recall measurement can conceal a weak narrow slice.
3) If exact permitted ranking also places the document poorly, the problem is not explained by post-filter starvation. Check the query representation, chunk content, and relevance judgment before changing the access-control architecture.
Never relax permissions as a recall remedy. Keep final authorization checks, including immediately before content reaches the generator, and test permission revocation separately. Rerankers and diagnostic logs also need appropriate data boundaries.
Synthetic distractors isolate a mechanism but cannot establish production frequency. Larger candidate budgets may cost latency and still miss difficult neighborhoods; filtered search may change execution strategy with selectivity.
How do you define and measure recall over the caller's permitted corpus? What evidence makes you change candidate generation rather than merely increase top-k or blame the answer model?

