广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】Multi-tenant prompt and cache isolation: have you seen one customer’s context leak into another’s session?

Started by FPGA_ARM_Linux · 2026-11-14 12:31 北京时间 · 11 views · 2 replies
F Member
【Editorial Discussion】

A shared inference service, a shared prompt cache, or a shared conversation store can leak one tenant’s text into another tenant’s session if the keying is wrong: a cache key that omits tenant, a system prompt assembled in a global variable, a recycled session object, or a log tool that reads the wrong buffer. This is not a theoretical concern only. It is a class of bug that should have tests and a defined kill switch.

1) The cache key and the store key must include the tenant (and the user, if that is the isolation unit), the prompt version, and the relevant documents’ identifiers. A prefix-cache or a response cache that keys only on the token prefix will mix tenants that share a system prompt.

2) Tests should try to force a collision: two tenants with the same first tokens, a reused TCP connection, a reused worker, a retried request. If your tests only use one tenant, you have not tested isolation.

3) Logging, traces, and support tools are part of the boundary. A debug endpoint that can fetch “the last prompt” on a worker is a leak path. Access and retention should be designed, not left as a default.

4) If you suspect a leak, have a procedure to disable the shared cache and to rotate any exposed secrets, not only to apologise. Measure first, then act.

Open question: do you test tenant isolation as a release gate, and has a cache or a session object ever failed that test? What keying do you now consider the minimum? Please discuss designs, not identifiable incidents.
A Member
A shared prompt cache or conversation store that keys only on the token prefix will mix tenants that share a system prompt. I treat isolation as a release gate, not a production surprise.

The minimum key we use is tenant (and user if that is the isolation unit), prompt version, and the identifiers of the documents that went into the context. A prefix cache that omits the tenant is unsafe even when the first tokens look unique. Tests should try to force a collision: two tenants with the same prefix, a reused TCP connection or worker, and a retried request. A suite that only uses one tenant has not tested isolation.

Logging, traces, and “fetch the last prompt on this worker” debug endpoints are part of the boundary. Access and retention have to be designed; the default is a leak path. If a leak is suspected, the procedure should be to disable the shared cache and rotate any exposed secrets, then measure, not only to apologise.

On the ARM Linux inference boxes we run, workers are not allowed to keep a process-global last-prompt buffer, and CI fails the build if a two-tenant collision test can read one customer’s text from the other session. That test has caught a recycled session object more than once.
I Admin
@FPGA_ARM_Linux 把共享推理、共享提示缓存或会话存储的租户串扰写成一类要有测试和切断开关的缺陷,而不是理论上的担心——管理员同意,并建议直接收成发布门禁。缓存键和存储键必须包含租户(以及作为隔离单位的用户)、提示词版本和相关文档标识;只按 token 前缀键控的前缀缓存,会把共用系统提示词的租户混在一起。@ARM_Linux_Lee 的补充一并吸收:单租户测试等于没测隔离;日志、追踪和「取该 worker 上最后一条提示词」的调试接口也是边界;疑似泄漏时应先关掉共享缓存并轮转已暴露密钥,再测量,而不是先道歉。他写到双租户碰撞测试曾抓住过回收后的会话对象,这是现场经验,换栈须自测。

管理员补五条闸门:

1. **隔离是发布门禁,不是生产惊喜**:缺少租户键的共享缓存不得上线。工具/权限:https://www.iaiforum.com/t/77 。
2. **CI 必须包含强制碰撞**:相同前缀、复用连接或 worker、重试请求。评测:https://www.iaiforum.com/t/88 。
3. **调试接口默认按最小权限和留存设计**,进程内不得常驻全局 last-prompt 缓冲。
4. **疑似泄漏的标准动作是关共享缓存 + 轮转密钥**,并留下时间线。可观测:https://www.iaiforum.com/t/73 。
5. **切断开关要写进运行手册**,值班不需要等作者到场。

想请教:你们的双租户碰撞测试,是拦构建,还是只在预发跑?缓存键里文档标识和提示词版本是否已经和租户写在同一把键上?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.