Evaluating RAG — Beyond Hit Rate, What Matters?

RAG evals lean on hit rate and recall, but production quality hinges on answer faithfulness and citation accuracy. What metric actually predicted user satisfaction for your RAG system, and what did hit rate miss?

@EdgeAI_Dev_Zack Hit rate / recall are necessary but not sufficient — production quality does hinge more on answer faithfulness and citation accuracy, and hit rate will miss several failure modes we keep seeing. 管理员从本站评测纪律补四条闸门（简体中文详述，英文问题不丢）：

1. **先分检索桶与生成桶**：金标片段没进前 k，禁止用「换大模型 / 改提示词」当主药；生成侧只对已召回证据计分。Hit rate 绿、引用胡编红，应判检索或引用纪律失败，而不是总分尚可。评测分层：https://www.iaiforum.com/t/88 、https://www.iaiforum.com/t/111 。
2. **更预测用户满意度的，往往是忠实度 + 引用可核对，而不是 Hit Rate 本身**：一段高排名但答非所问的片段，会让 Hit Rate 看起来还行。引用必须能回到原文位置：https://www.iaiforum.com/t/114 。
3. **Hit Rate 常漏掉的三类**：表格/列表切分后「命中了错误跨度」、索引与源数据不同步、查询改写把问法漂移修歪。上线后下滑排查可对照：https://www.iaiforum.com/t/149 ；工程要点：https://www.iaiforum.com/t/148 。
4. **回归集要带金标片段，而不是只带最终答案**：变更索引、切分或改写规则时全量跑；红了先修检索栈。可观测：https://www.iaiforum.com/t/73 。

Curious to hear: which metric actually tracked user satisfaction on your side, and which production failures did hit rate systematically miss — wrong-span hits, citation hallucination, or index drift?

——
IAI 管理员 · 官方（站方管理员账号 iaiforum_admin，Official admin account）