Abstract
Contextual advertising serves ads that are aligned to the content that the user is viewing. The rapid growth of video content on social platforms and streaming services, along with privacy concerns, has increased the need for contextual advertising. Placing the right ad in the right context creates a seamless and pleasant ad viewing experience, resulting in higher audience engagement and, ultimately, better ad monetization. From a technology standpoint, effective contextual advertising requires a video retrieval system capable of understanding complex video content at a very granular level. Current text-to-video retrieval models based on joint multimodal training demand large datasets and computational resources, limiting their practicality and lacking the key functionalities required for ad ecosystem integration. We introduce ContextIQ, a multimodal expert-based video retrieval system designed specifically for contextual advertising. ContextIQ utilizes modality-specific experts-video, audio, transcript (captions), and metadata such as objects, actions, emotion, etc.-to create semantically rich video representations. We show that our system, without joint training, achieves better or comparable results to state-of-the-art models and commercial solutions on multiple text-to-video retrieval benchmarks. Our ablation studies highlight the benefits of leveraging multiple modalities for enhanced video retrieval accuracy instead of using a vision-language model alone. Furthermore, we show how video retrieval systems such as ContextIQ can be used for contextual advertising in an ad ecosystem while also addressing concerns related to brand safety and filtering inappropriate content.
Abstract (translated)
情境广告投放的是与用户正在查看的内容相匹配的广告。随着社交平台和流媒体服务上的视频内容快速增长，以及隐私问题的关注增加，对情境广告的需求也越来越大。在正确的情境中放置合适的广告可以创造一个无缝且愉悦的广告观看体验，从而提高观众参与度，并最终实现更好的广告变现效果。从技术角度来看，有效的场景化广告投放需要一个能够以非常精细的粒度理解复杂视频内容的视频检索系统。当前基于联合多模态训练的文本到视频检索模型要求大量数据集和计算资源，这限制了其实用性，并且缺乏将广告生态系统整合所需的关键功能。 我们介绍了一种名为ContextIQ的多模态专家型视频检索系统，专门用于情境广告投放。ContextIQ利用特定模式的专家——包括视频、音频、字幕（字幕）、以及对象、动作、情感等元数据——来创建语义丰富的视频表示形式。我们的研究表明，在不进行联合训练的情况下，该系统在多个文本到视频检索基准上达到了优于或与最先进的模型和商用解决方案相当的结果。我们的消融研究强调了利用多种模态提高视频检索准确性的好处，而不仅仅是使用单一的视觉语言模型。此外，我们展示了像ContextIQ这样的视频检索系统如何在广告生态系统中用于情境广告投放，同时也解决了品牌安全以及过滤不适当内容的相关问题。
URL
https://arxiv.org/abs/2410.22233
