Abstract
Tracking by natural language specification aims to locate the referred target in a sequence based on the natural language description. Existing algorithms solve this issue in two steps, visual grounding and tracking, and accordingly deploy the separated grounding model and tracking model to implement these two steps, respectively. Such a separated framework overlooks the link between visual grounding and tracking, which is that the natural language descriptions provide global semantic cues for localizing the target for both two steps. Besides, the separated framework can hardly be trained end-to-end. To handle these issues, we propose a joint visual grounding and tracking framework, which reformulates grounding and tracking as a unified task: localizing the referred target based on the given visual-language references. Specifically, we propose a multi-source relation modeling module to effectively build the relation between the visual-language references and the test image. In addition, we design a temporal modeling module to provide a temporal clue with the guidance of the global semantic information for our model, which effectively improves the adaptability to the appearance variations of the target. Extensive experimental results on TNL2K, LaSOT, OTB99, and RefCOCOg demonstrate that our method performs favorably against state-of-the-art algorithms for both tracking and grounding. Code is available at this https URL.
Abstract (translated)
自然语言指定的跟踪旨在根据自然语言描述在序列中定位提及的目标。现有算法采取了两个步骤来解决这个问题:视觉grounding和跟踪,并相应地部署分开的grounding模型和跟踪模型来实现这两个步骤。这种分开的框架忽略了视觉grounding和跟踪之间的联系,也就是自然语言描述为这两个步骤提供了全球语义线索。此外,分开的框架很难进行端到端的训练。为了解决这些问题,我们提出了一个联合的视觉grounding和跟踪框架,将其重新定义为一种统一的任务:根据给定的视觉语言引用定位提及的目标。具体来说,我们提出了一个多源关系建模模块,以有效地构建视觉语言引用和测试图像之间的关系。此外,我们设计了时间建模模块,以提供给我们的模型一个时间线索,以提供时间线索,并有效地改善其对目标外观变化适应性。在TNL2K、LaSOT、OTB99和RefCOCOg等实验中的结果表明,我们的方法和现有跟踪和grounding算法在这两个方面的性能都很优秀。代码可在这个https URL上获取。
URL
https://arxiv.org/abs/2303.12027
