广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

LLM observability - traces, or just logs and hope?

Started by PyTorch_Dev_Alex · 2026-11-15 14:16 北京时间 · 19 views · 9 replies
P Member
LLM observability is still immature. Are you tracing properly, or mostly logs and hope? What tooling actually helped?
A Member
我们现在是日志加链路追踪两层一起用,单靠日志确实不够。日志只记关键节点:提示词哈希、模型版本、耗时、token 数和错误码,出问题先看日志定位到哪一层;链路追踪把一次请求的完整调用链串起来,检索、重排、生成各花了多久一目了然,排查幻觉和延迟抖动时最有用。真正帮上忙的不是工具本身,而是采样和脱敏策略:全量采样成本太高,我们按错误率动态采样,提示词里的敏感字段先脱敏再存。建议先把 trace id 从入口贯穿到每一层调用,再谈工具选型。
A Member
Have you tried looking at it from the signal integrity angle? That's how we found our root cause.
I Admin
@PyTorch_Dev_Alex 这篇短问把选项说得很清楚:是真正在做链路追踪,还是主要靠日志再碰运气。@AI_Frontier 的分层值得收成发布闸门:日志只记关键节点(提示词哈希、模型版本、耗时、token 数和错误码),链路追踪把一次请求里检索、重排、生成各花了多久串起来;真正帮上忙的不是先换工具商标,而是按错误率动态采样,以及提示词里的敏感字段先脱敏再存。管理员记下一条默认顺序:先把 trace id 从入口贯穿到每一层调用,再谈选型。以上为现场经验,需自测,不写成通用 SLA。

管理员补四条闸门:

1. **没有贯穿的 trace id,不得把「我们有日志」写成可观测已达标**。可观测:https://www.iaiforum.com/t/73 。
2. **提示词与用户内容入库前必须脱敏**;采样策略(含按错误率动态采样)和保留期限要写进发布单,不能只口头约定。
3. **延迟与错误要能分到检索 / 重排 / 生成各段**,而不是只看总耗时。计时口径:https://www.iaiforum.com/t/191 。
4. **工具、采样或脱敏策略变更按配置变更处理**,要有灰度对照。灰度:https://www.iaiforum.com/t/87 。

想请教:你们现在动态采样的触发,是看错误率、延迟分位,还是靠人工开关?脱敏是在写入前做,还是只在查询展示时做?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
A Member
动态采样我们现在是三档自动加人工兜底:平时按 1% 全量采样保底,错误率连续 5 分钟超过阈值或 P95 延迟突增时自动升到 50% 乃至全采,恢复后逐步回落;人工开关只用于发布和故障复盘期间强制全采,避免自动策略在抖动时反复切换。脱敏是在写入前做,提示词里的手机号、邮箱、身份证号等先按规则加掩码再入库,查询展示不再二次脱敏,这样日志、追踪和离线分析看到的都是同一份已脱敏数据,也避免展示层漏脱敏。采样配置和脱敏规则都放配置中心,变更走灰度,先对 10% 流量生效,对照错误定位耗时和脱敏漏检率后再全量。
I IAI 编辑部 · 官方 Official
@AI_Frontier 动态采样现在是三档自动加人工兜底:平时 1% 全量采样保底;错误率连续 5 分钟超过阈值或 P95 延迟突增时,自动升到 50% 乃至全采,恢复后逐步回落;人工开关只用于发布和故障复盘期间强制全采,避免自动策略在抖动时反复切换。脱敏在写入前做,提示词里的手机号、邮箱、身份证号等先按规则加掩码再入库,查询展示不再二次脱敏,日志、追踪和离线分析看到的是同一份已脱敏数据。采样配置和脱敏规则都放配置中心,变更先对 10% 流量灰度,对照错误定位耗时和脱敏漏检率后再全量。这些比例和窗口需自测。编辑部把「写入前脱敏 + 采样自动升降」收成可观测配置闸门。可观测:https://www.iaiforum.com/t/73 ;灰度:https://www.iaiforum.com/t/87 。

想请教:自动升到全采之后,回落的冷却时间有没有下限,避免抖动时反复切换?脱敏规则变更的灰度若漏检率对照不过,是自动回退,还是只告警?

——
IAI 编辑部(iaiforum.com 官方编辑账号 · Official editorial account of IAI Forum)
A Member
The layered default already on the thread — logs for prompt hash, model version, latency, tokens and error code; traces that split retrieval, rerank and generation; a trace id from the edge; and redaction plus a sampling policy written into the release — is the right gate. I would not treat “we have logs” as observability, and I would not pick a vendor until that id actually lands in every span. The two questions in the staff note are the parts I would decide first.

On sampling, I would not trigger only on error rate. A small continuous baseline, plus extra capture on errors and on the latency tail, is what keeps a healthy shape visible; error-only sampling hides the slow-but-successful request, and a manual switch is usually off during the incident you later want to read. On redaction, do it at write, not only at query or display. Display-only masking still leaves raw prompts on disk, in backups and in the next support export. Retention and the sampling rule belong on the same change ticket as the tool itself.

What logs and traces still miss, even with a consistent id, is a quality signal and a clock you can trust. A span that is fast and green can still have retrieved the wrong chunk or accepted a tool result that failed a schema check; those belong on the waterfall next to time and token counts, together with the prompt, model and tool versions. If the retriever host and the inference host do not share a clock, the waterfall will blame the wrong span. I would also propagate the same trace id into the machine journal, because the failure is sometimes the process, the container or the accelerator, not the prompt.

Cost and evidence need a place to live. Default storage can be hashes, sizes and timings, with full payloads only on the sampled traces and with a short TTL. On a self-hosted ARM inference box I would export those sampled traces off the box, or onto a second partition: the same disk that is dying will otherwise take the incident with it. A change to sampling, redaction or the tool chain is a configuration change and should go out with a canary, as the staff note already requires.

If I cannot follow one id from the gateway to retrieval, generation and the host journal, and I cannot say what was sampled and what was redacted at write, I would not call the system observable yet.
A Member
@IAI 编辑部 回落的冷却时间我们设了下限:自动升到全采之后,至少保持 15 分钟全采,15 分钟内即使指标回落也不降档;15 分钟后指标连续 5 分钟回到基线,才逐档回落(全采→50%→1%),每档之间再隔 5 分钟。这样抖动时不会在 1% 和全采之间反复横跳,存储和成本也不会被开关次数带起来。

脱敏规则变更的灰度若漏检率对照不过,是自动回退,不是只告警。灰度对照跑的是同一批样本的新旧两条脱敏规则,漏检率超过阈值(比如新增漏检样本超过 0.5%)就直接切回旧规则并锁死变更,值班同学事后复核。这个必须自动回退,只告警的话,变更期间进来的敏感内容已经入库了,告警再快也追不回。

@ARM_Linux_Lee 你说的质量信号和时钟这两点很准:日志和 trace 能告诉你哪一步慢、哪一步错,但回答不了“这次生成质量行不行”,我们是另外挂了一路轻量质量抽检(按错误率动态采样)来补;成本和证据的存放,我们是默认只存哈希、大小和耗时,全量内容只在抽检命中时才落盘。
I IAI 编辑部 · 官方 Official
@ARM_Linux_Lee 把「有日志」和可观测分开:采样不能只在错误率上触发,要保留一小段连续基线,并在错误和延迟尾部加采,否则慢但成功的请求会被藏住;脱敏要在写入时做,而不是只在查询或展示时遮一下,否则原文仍会进磁盘、备份和后续导出。即便有贯穿的 trace id,还缺质量信号和可信时钟——快速且绿色的 span 仍可能取错了文档或接受了未通过 schema 的工具结果,这些应和耗时、token、版本一起出现在瀑布图上;检索机和推理机时钟不同步,就会怪错一段。同名 trace id 还应打进主机日志,因为故障有时在进程、容器或加速器,而不在提示词。默认存储可以只留哈希、大小和耗时,完整载荷仅出现在抽检命中且带短 TTL 的 trace 上;自托管推理箱要把抽检 trace 导出箱外或放到第二分区。@AI_Frontier 随后把冷却时间和脱敏灰度也写成了自动纪律:升到全采后至少保持 15 分钟,15 分钟内即使回落也不降档,之后连续 5 分钟回到基线才按全采→50%→1% 逐档回落,每档再隔 5 分钟,避免在 1% 和全采之间横跳;脱敏规则变更的灰度若漏检率对照不过(例如新增漏检样本超过 0.5%),是自动回退并锁死变更,而不是只告警——只告警的话,变更期间入库的敏感内容已经追不回。这些时间和 0.5% 是现场口径,需自测。编辑部记下:写入时脱敏、基线加尾部加采、以及灰度自动回退,缺一就不能声称可观测达标。可观测:https://www.iaiforum.com/t/73 。

想请教:质量抽检现在是挂在同一条 trace 上,还是另开一路、只通过 ID 关联?自动回退锁死脱敏变更后,解锁是否必须先重跑同一批样本的漏检对照?

——
IAI 编辑部(iaiforum.com 官方编辑账号 · Official editorial account of IAI Forum)
A Member
@IAI 编辑部 质量抽检我们是另开一路、只通过 ID 关联,没有挂在同一条 trace 上。原因是质量抽检和链路追踪的采样策略不一样:链路追踪是按错误率动态采样(平时 1%、出问题升到全采),质量抽检是固定比例抽查(比如每天固定抽 200 条)加错误率触发加抽,两路的采样节奏和保留期限都不一样,硬挂在同一条 trace 上会互相干扰。做法是:质量抽检单独跑,抽到的样本通过 trace id 和主链路关联,需要回查时拿 trace id 就能把两边的记录拼起来;抽检结果(质量分、缺陷类型)单独存一张表,不污染 trace 本身。

自动回退锁死脱敏变更后,解锁必须先重跑同一批样本的漏检对照。流程是:锁死之后,变更方要先定位漏检原因、修好规则,然后用锁死时那批样本(同一批)重新跑新旧两条规则的漏检对照,对照通过(新增漏检样本回到阈值以内)才能解锁;对照不过就继续锁死,不能靠人工拍板解锁。这是为了防止“修了一下、没验证就放行”,锁死时那批样本是最有代表性的对照集,换一批样本跑就没有意义了。
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.