F
Member
【Editorial Discussion】
A shared inference service, a shared prompt cache, or a shared conversation store can leak one tenant’s text into another tenant’s session if the keying is wrong: a cache key that omits tenant, a system prompt assembled in a global variable, a recycled session object, or a log tool that reads the wrong buffer. This is not a theoretical concern only. It is a class of bug that should have tests and a defined kill switch.
1) The cache key and the store key must include the tenant (and the user, if that is the isolation unit), the prompt version, and the relevant documents’ identifiers. A prefix-cache or a response cache that keys only on the token prefix will mix tenants that share a system prompt.
2) Tests should try to force a collision: two tenants with the same first tokens, a reused TCP connection, a reused worker, a retried request. If your tests only use one tenant, you have not tested isolation.
3) Logging, traces, and support tools are part of the boundary. A debug endpoint that can fetch “the last prompt” on a worker is a leak path. Access and retention should be designed, not left as a default.
4) If you suspect a leak, have a procedure to disable the shared cache and to rotate any exposed secrets, not only to apologise. Measure first, then act.
Open question: do you test tenant isolation as a release gate, and has a cache or a session object ever failed that test? What keying do you now consider the minimum? Please discuss designs, not identifiable incidents.
A shared inference service, a shared prompt cache, or a shared conversation store can leak one tenant’s text into another tenant’s session if the keying is wrong: a cache key that omits tenant, a system prompt assembled in a global variable, a recycled session object, or a log tool that reads the wrong buffer. This is not a theoretical concern only. It is a class of bug that should have tests and a defined kill switch.
1) The cache key and the store key must include the tenant (and the user, if that is the isolation unit), the prompt version, and the relevant documents’ identifiers. A prefix-cache or a response cache that keys only on the token prefix will mix tenants that share a system prompt.
2) Tests should try to force a collision: two tenants with the same first tokens, a reused TCP connection, a reused worker, a retried request. If your tests only use one tenant, you have not tested isolation.
3) Logging, traces, and support tools are part of the boundary. A debug endpoint that can fetch “the last prompt” on a worker is a leak path. Access and retention should be designed, not left as a default.
4) If you suspect a leak, have a procedure to disable the shared cache and to rotate any exposed secrets, not only to apologise. Measure first, then act.
Open question: do you test tenant isolation as a release gate, and has a cache or a session object ever failed that test? What keying do you now consider the minimum? Please discuss designs, not identifiable incidents.

