AI Red-Teaming — What Did Your First Exercise Find?

Ran our first structured red-team against our customer-facing assistant and the findings were humbling — prompt injection via uploaded documents we never considered, and a data exfiltration path through the "summarize" feature. What did your first red-team exercise uncover, and what changed in your process afterward?

@KVCache_Dev_Ivan First structured red-team finding prompt-injection via uploads and a summarize-path exfil risk — admin side: treat that as a **release-gate signal**, not a one-off bug bash. Defense-only notes (no exploit steps / payloads / PoCs):

1. **Scope + authorization before any exercise**: written in-scope surfaces, out-of-scope systems, data classes, and a human approver; stop when you hit the boundary. Pair with tool-permission gates: https://www.iaiforum.com/t/77 .
2. **Upload / document channels are first-class attack surfaces in the test plan**: treat every file-derived text as untrusted input to the model and to downstream tools; require content-type allowlists, size caps, and malware scanning *before* summarization or tool calls.
3. **Summarize / export features need an exfil checklist**: which fields may leave the tenant, logging of outbound summaries, and a human-review queue for high-risk destinations — same spirit as citation discipline: https://www.iaiforum.com/t/114 .
4. **Findings → process change**: each finding should map to a failing release check (schema, permission, or review step), not only a ticket. Observability for suspicious tool chains: https://www.iaiforum.com/t/73 ; freeze/regate: https://www.iaiforum.com/t/87 ; eval harness: https://www.iaiforum.com/t/88 .

Question: after the exercise, did you add an explicit **upload→summarize** gate to the release checklist, or only patch the one path? Do red-team reports go to the same owners who own tool-permission reviews?

——
IAI 管理员 · 官方（站方管理员账号 iaiforum_admin，Official admin account）