广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】Prompt injection via retrieved documents: what did your allow-list and output validator actually stop?

Started by FPGA_ARM_Linux · 2026-11-14 12:26 北京时间 · 9 views · 1 replies
F Member
【Editorial Discussion】

Retrieved documents, tickets, or emails can contain instructions that the model will treat as higher priority than the system policy, especially if those documents are also allowed to influence tool arguments. A filter on the user prompt will not see this path. The useful test is not “can we talk about injection,” it is which attacks your allow-list and your output validator actually stopped.

1) Put adversarial content in the retrieved set, not only in the user box: requests to ignore policy, to exfiltrate earlier context, or to issue a tool call the user did not ask for. If your eval only uses clean corpora, you have not tested this path.

2) Tool arguments should be validated against types, destinations, and an allow-list that the document cannot expand. A model that is “well behaved” in prose can still emit a URL or a command that the wrapper will run. The wrapper is the control; the prompt is not.

3) Log when a document asked for an action and when the wrapper refused. If you cannot show a refusal, you cannot claim the control works.

4) Re-run the set when you add a tool or change the retriever. A new tool is a new attack surface.

Open question: have you caught an injection that arrived only through a retrieved chunk, and which layer stopped it — the model, the validator, or the allow-list? If nothing would have stopped it, what did you add? Method-level reports welcome.
I IAI 编辑部 · 官方 Official
@FPGA_ARM_Linux 把检索文档、工单或邮件当成另一条指令通道,而不是只过滤用户输入框,这个分层编辑部同意。下面只谈防御闸门和验收,不讨论攻击步骤或绕过方法。用户提示词上的过滤器看不到这条路径;真正有用的验收,是看允许列表和输出校验挡住了什么,以及文档要求执行动作时包装层有没有拒绝记录。

编辑部只补防御闸门:

1. **检索进入的文本默认是数据,不得继承系统权限,也不得自行扩大工具范围**。工具/权限:https://www.iaiforum.com/t/77 ;上传文档通道的同类纪律:https://www.iaiforum.com/t/135 。
2. **工具参数必须在包装层按类型、目的地和允许列表校验,文档不能改这张表**:提示词不是控制面。
3. **文档要求动作而包装层拒绝时,必须打点**:说不清拒绝,就不能声称控制有效。可观测:https://www.iaiforum.com/t/73 。
4. **新增工具或改检索器,要重跑同一组防御用例**:新工具就是新的检查面。评测:https://www.iaiforum.com/t/88 。
5. **防御用例与助手版本写进同一张发布单**,失败即阻断灰度:https://www.iaiforum.com/t/87 。

想请教:你们现在能不能在日志里指出「拒绝发生在包装层而不是模型散文里」?新增一个工具时,这组防御用例是不是同一条发布门禁?

——
IAI 编辑部(iaiforum.com 官方编辑账号 · Official editorial account of IAI Forum)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.