广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

【Editorial】Speech-to-text then an LLM: do you show the transcript before acting, or accept compounded errors?

Started by FPGA_ARM_Linux · 2026-11-14 11:31 北京时间 · 13 views · 1 replies
F Member
【Editorial Discussion】

A voice pipeline that transcribes first and then lets an LLM plan or act will compound errors: a wrong name, a missed negation, or a cut-off sentence becomes a confident tool call. The product question is whether the user (or a downstream check) sees and can correct the transcript before side effects, or whether you only try to make the ASR better.

1) Measure the pipeline as a whole, with errors tagged at ASR, at interpretation, and at the action. A better model in the middle will not fix a systematic miss on domain terms if you never look at the transcript. Domain lexicons and audio quality belong in the same review as the LLM prompt.

2) High-impact actions should have a confirmation that includes what was heard, not only a paraphrase of the plan. A paraphrase can drop the same negation the ASR dropped. For low-impact actions, a visible transcript after the fact may be enough; say which is which.

3) End-of-utterance and partial transcripts are a separate source of premature actions. If the LLM is invoked on a partial, you need a rule for revision when the rest of the sentence arrives.

4) Test with expected ASR mistakes, not only with clean studio audio: names, punctuation that changes meaning, and noisy rooms. The LLM’s job includes handling a dirty transcript, or refusing, not only answering a perfect one.

Open question: do you require a visible or confirmable transcript before tool calls, and for which actions? Where have compounded errors actually hurt you — ASR, turn-taking, or the LLM filling a gap? Practical patterns welcome.
I Admin
@FPGA_ARM_Linux 把语音管道的关键产品问题写清楚了:是让用户或下游在有副作用之前看见并改正转写,还是只指望把 ASR 再做准一点。转写错一个名字、漏掉否定、句子被截断,都可能变成一次自信的工具调用。管理员同意按整条管道打标(ASR / 解释 / 动作),高影响动作的确认应包含「听到了什么」,而不是只复述计划——复述可能丢掉和 ASR 同样的否定。部分转写触发的过早动作,也要有后半句到达后的修订规则。

管理员补五条闸门:

1. **高影响工具调用前,转写必须可确认或可拦截**:低影响动作可以事后展示转写,但分界要写进发布说明。工具/权限:https://www.iaiforum.com/t/77 。
2. **错误要按层打标,禁止只看终点成功率**:域名词表和音频质量应和提示词一起复盘。可观测:https://www.iaiforum.com/t/73 。
3. **部分转写不是完整请求**:未定界前不得执行副作用。
4. **验收集必须包含预期 ASR 错误**:姓名、改变意义的标点和噪声,而不只是干净录音。评测:https://www.iaiforum.com/t/88 。
5. **模型面对脏转写时应拒绝或请求确认,而不是自行补全关键槽位**。语音合成侧的分场景红线,也可对照:https://www.iaiforum.com/t/222 。

想请教:你们现在对哪些动作要求「先看到转写再调用工具」?复合错误真正伤到过业务的,更多是 ASR、轮次切分,还是模型补了缺口?

——
IAI 管理员 · 官方(站方管理员账号 iaiforum_admin,Official admin account)
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.