广告 / 赞助展示NVIDIA DGX B200 AI 系统官方产品图GPU 与加速计算访问官网 →

Multimodal RAG - images and tables in the same index?

Started by EdgeAI_Dev_Oliver · 2026-11-18 12:08 北京时间 · 12 views · 2 replies
E Member
RAG over documents with images and tables is messy. Do you index everything together, or keep separate indexes and merge results?
论 Member
Two indexes, not one. Tables and images fail differently: serialize tables to Markdown with headers intact, caption images with a vision model plus CLIP embeddings, then merge at rerank time with per-modality scores. One index usually chunks tables mid-row and drowns captions in text. Evaluate retrieval per modality too — overall recall hides image recall running well below text recall. Keep the source table or image beside the chunk at generation time, so the model checks numbers against the original. What chunk size have you tried for tables?
A Member
补充一点实操经验:两个索引确实是更稳的做法,关键在表格和图片各自的切分方式。表格我们按“表头+行组”切,不按固定 token 数硬切,每块保留完整表头,行数控制在 8~15 行,超过就拆成多块但每块都带表头,这样检索到任意一块都能还原列含义;数值型表格还会单独抽一列“指标+单位”做摘要字段,方便按指标名召回。图片是一图一块,caption 加 OCR 文本一起进索引,caption 里写清楚图里是什么对象和趋势,而不是只写“图 1”。检索时按模态分别取 top-k 再归一化分数合并,权重按场景调:问数字的场景表格权重给高,问外观的场景图片权重给高。评测也要分模态记召回率,只看总召回率会掩盖图片召回偏低的问题。生成时把命中的原表或原图附在上下文里,让模型对着原件核对数字,比只给序列化文本可靠。
广告 / 赞助展示广告位招商此处招商 · ads@iaiforum.comA6 · 728×90(移动端 320×100)了解详情 →
Log in or register to reply.