Abstract
We evaluate multimodal large language models (MLLMs) for topic-aligned captioning in financial short-form videos (SVs) by testing joint reasoning over transcripts (T), audio (A), and video (V). Using 624 annotated YouTube SVs, we assess all seven modality combinations (T, A, V, TA, TV, AV, TAV) across five topics: main recommendation, sentiment analysis, video purpose, visual analysis, and financial entity recognition. Video alone performs strongly on four of five topics, underscoring its value for capturing visual context and effective cues such as emotions, gestures, and body language. Selective pairs such as TV or AV often surpass TAV, implying that too many modalities may introduce noise. These results establish the first baselines for financial short-form video captioning and illustrate the potential and challenges of grounding complex visual cues in this domain. All code and data can be found on our Github under the CC-BY-NC-SA 4.0 license.
Abstract (translated)
我们通过测试跨转录文本（T）、音频（A）和视频（V）的联合推理，评估了多模态大型语言模型（MLLMs）在金融短视频（SVs）中进行主题一致字幕生成的能力。使用624个标注过的YouTube SV数据集，我们在五个话题上对所有七种模态组合（T、A、V、TA、TV、AV、TAV）进行了评估：主要推荐、情感分析、视频目的、视觉分析和金融实体识别。单独的视频在五项中的四项表现突出，强调了捕捉视觉环境和有效线索（如情绪、手势和肢体语言）的重要性。而选择性的模态组合，例如TV或AV，经常超过TAV的表现，表明过多的模态可能引入噪声。这些结果为金融短视频字幕生成建立了首个基准，并展示了在该领域将复杂视觉提示转化为实际应用的潜力与挑战。所有代码和数据可以在我们的GitHub上找到（遵循CC-BY-NC-SA 4.0许可）。
URL
https://arxiv.org/abs/2509.25745
