Video_Indexing
-
Efficient Self-Supervised Video Hashing with Selective State Spaces
2024-12-19 04:33:22Jinpeng Wang, Niu Lian, Jun Li, Yuting Wang, Yan Feng, Bin Chen, Yongbing Zhang, Shu-Tao XiaAbstract
Self-supervised video hashing (SSVH) is a practical task in video indexing and retrieval. Although Transformers are predominant in SSVH for their impressive temporal modeling capabilities, they often suffer from computational and memory inefficiencies. Drawing inspiration from Mamba, an advanced state-space model, we explore its potential in SSVH to achieve a better balance between efficacy and efficiency. We introduce S5VH, a Mamba-based video hashing model with an improved self-supervised learning paradigm. Specifically, we design bidirectional Mamba layers for both the encoder and decoder, which are effective and efficient in capturing temporal relationships thanks to the data-dependent selective scanning mechanism with linear complexity. In our learning strategy, we transform global semantics in the feature space into semantically consistent and discriminative hash centers, followed by a center alignment loss as a global learning signal. Our self-local-global (SLG) paradigm significantly improves learning efficiency, leading to faster and better convergence. Extensive experiments demonstrate S5VH's improvements over state-of-the-art methods, superior transferability, and scalable advantages in inference efficiency. Code is available at this https URL.
Abstract (translated)
自我监督视频哈希（SSVH）是在视频索引和检索中的一个实用任务。尽管Transformer因出色的时序建模能力而在SSVH中占据主导地位，但它们往往存在计算和内存效率低下的问题。受到先进状态空间模型Mamba的启发，我们探索其在SSVH中的潜力，以实现效果与效率之间的更好平衡。我们引入了S5VH，一种基于Mamba的视频哈希模型，并采用了改进的自我监督学习范式。具体来说，我们在编码器和解码器中设计了双向Mamba层，这些层能够有效地捕捉时间关系，得益于线性复杂度的数据依赖选择性扫描机制。在我们的学习策略中，我们将特征空间中的全局语义转换为语义一致且具有判别性的哈希中心，并随后采用中心对齐损失作为全局学习信号。我们的自局部-全局（SLG）范式显著提升了学习效率，实现了更快和更好的收敛效果。广泛的实验表明，S5VH在性能上超越了最先进的方法，拥有优越的迁移能力和推理效率可扩展优势。代码可在该链接https URL获得。
URL
https://arxiv.org/abs/2412.14518
PDF
-
Contrastive Graph Multimodal Model for Text Classification in Videos
2022-06-06 04:06:21Ye Liu, Changchong Lu, Chen Lin, Di Yin, Bo RenAbstract
The extraction of text information in videos serves as a critical step towards semantic understanding of videos. It usually involved in two steps: (1) text recognition and (2) text classification. To localize texts in videos, we can resort to large numbers of text recognition methods based on OCR technology. However, to our knowledge, there is no existing work focused on the second step of video text classification, which will limit the guidance to downstream tasks such as video indexing and browsing. In this paper, we are the first to address this new task of video text classification by fusing multimodal information to deal with the challenging scenario where different types of video texts may be confused with various colors, unknown fonts and complex layouts. In addition, we tailor a specific module called CorrelationNet to reinforce feature representation by explicitly extracting layout information. Furthermore, contrastive learning is utilized to explore inherent connections between samples using plentiful unlabeled videos. Finally, we construct a new well-defined industrial dataset from the news domain, called TI-News, which is dedicated to building and evaluating video text recognition and classification applications. Extensive experiments on TI-News demonstrate the effectiveness of our method.
Abstract (translated)
URL
https://arxiv.org/abs/2206.02343
PDF
-
Deep Learning-based Automatic Player Identification and Logging in American Football Videos
2022-04-26 02:59:03Hongshan Liu, Colin Aderon, Noah Wagon, Huapu Liu, Steven MacCall, Yu GanAbstract
American football games attract significant worldwide attention every year. Game analysis systems generate crucial information that can help analyze the games by providing fans and coaches with a convenient means to track and evaluate player performance. Identifying participating players in each play is also important for the video indexing of player participation per play. Processing football game video presents challenges such as crowded setting, distorted objects, and imbalanced data for identifying players, especially jersey numbers. In this work, we propose a deep learning-based football video analysis system to automatically track players and index their participation per play. It is a multi-stage network design to highlight area of interest and identify jersey number information with high accuracy. First, we utilize an object detection network, a detection transformer, to tackle the player detection problem in crowded context. Second, we identify players using jersey number recognition with a secondary convolutional neural network, then synchronize it with a game clock subsystem. Finally, the system outputs a complete log in a database for play indexing. We demonstrate the effectiveness and reliability of player identification and the logging system by analyzing the qualitative and quantitative results on football videos. The proposed system shows great potential for implementation in and analysis of football broadcast video.
Abstract (translated)
URL
https://arxiv.org/abs/2204.13809
PDF
-
NEWSKVQA: Knowledge-Aware News Video Question Answering
2022-02-08 17:31:31Pranay Gupta, Manish GuptaAbstract
Answering questions in the context of videos can be helpful in video indexing, video retrieval systems, video summarization, learning management systems and surveillance video analysis. Although there exists a large body of work on visual question answering, work on video question answering (1) is limited to domains like movies, TV shows, gameplay, or human activity, and (2) is mostly based on common sense reasoning. In this paper, we explore a new frontier in video question answering: answering knowledge-based questions in the context of news videos. To this end, we curate a new dataset of 12K news videos spanning across 156 hours with 1M multiple-choice question-answer pairs covering 8263 unique entities. We make the dataset publicly available. Using this dataset, we propose a novel approach, NEWSKVQA (Knowledge-Aware News Video Question Answering) which performs multi-modal inferencing over textual multiple-choice questions, videos, their transcripts and knowledge base, and presents a strong baseline.
Abstract (translated)
URL
https://arxiv.org/abs/2202.04015
PDF
-
Sound Event Detection Transformer: An Event-based End-to-End Model for Sound Event Detection
2021-10-05 12:56:23Zhirong Ye, Xiangdong Wang, Hong Liu, Yueliang Qian, Rui Tao, Long Yan, Kazushige OuchiAbstract
Sound event detection (SED) has gained increasing attention with its wide application in surveillance, video indexing, etc. Existing models in SED mainly generate frame-level predictions, converting it into a sequence multi-label classification problem, which inevitably brings a trade-off between event boundary detection and audio tagging when using weakly labeled data to train the model. Besides, it needs post-processing and cannot be trained in an end-to-end way. This paper firstly presents the 1D Detection Transformer (1D-DETR), inspired by Detection Transformer. Furthermore, given the characteristics of SED, the audio query and a one-to-many matching strategy for fine-tuning the model are added to 1D-DETR to form the model of Sound Event Detection Transformer (SEDT), which generates event-level predictions, end-to-end detection. Experiments are conducted on the URBAN-SED dataset and the DCASE2019 Task4 dataset, and both experiments have achieved competitive results compared with SOTA models. The application of SEDT on SED shows that it can be used as a framework for one-dimensional signal detection and may be extended to other similar tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2110.02011
PDF
-
DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval
2021-06-24 18:34:24Giorgos Kordopatis-Zilos, Christos Tzelepis, Symeon Papadopoulos, Ioannis Kompatsiaris, Ioannis PatrasAbstract
In this paper, we address the problem of high performance and computationally efficient content-based video retrieval in large-scale datasets. Current methods typically propose either: (i) fine-grained approaches employing spatio-temporal representations and similarity calculations, achieving high performance at a high computational cost or (ii) coarse-grained approaches representing/indexing videos as global vectors, where the spatio-temporal structure is lost, providing low performance but also having low computational cost. In this work, we propose a Knowledge Distillation framework, which we call Distill-and-Select (DnS), that starting from a well-performing fine-grained Teacher Network learns: a) Student Networks at different retrieval performance and computational efficiency trade-offs and b) a Selection Network that at test time rapidly directs samples to the appropriate student to maintain both high retrieval performance and high computational efficiency. We train several students with different architectures and arrive at different trade-offs of performance and efficiency, i.e., speed and storage requirements, including fine-grained students that store index videos using binary representations. Importantly, the proposed scheme allows Knowledge Distillation in large, unlabelled datasets -- this leads to good students. We evaluate DnS on five public datasets on three different video retrieval tasks and demonstrate a) that our students achieve state-of-the-art performance in several cases and b) that our DnS framework provides an excellent trade-off between retrieval performance, computational speed, and storage space. In specific configurations, our method achieves similar mAP with the teacher but is 20 times faster and requires 240 times less storage space. Our collected dataset and implementation are publicly available: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2106.13266
PDF
-
View-invariant action recognition
2020-09-01 18:08:46Yogesh S Rawat, Shruti VyasAbstract
Human action recognition is an important problem in computer vision. It has a wide range of applications in surveillance, human-computer interaction, augmented reality, video indexing, and retrieval. The varying pattern of spatio-temporal appearance generated by human action is key for identifying the performed action. We have seen a lot of research exploring this dynamics of spatio-temporal appearance for learning a visual representation of human actions. However, most of the research in action recognition is focused on some common viewpoints, and these approaches do not perform well when there is a change in viewpoint. Human actions are performed in a 3-dimensional environment and are projected to a 2-dimensional space when captured as a video from a given viewpoint. Therefore, an action will have a different spatio-temporal appearance from different viewpoints. The research in view-invariant action recognition addresses this problem and focuses on recognizing human actions from unseen viewpoints.
Abstract (translated)
URL
https://arxiv.org/abs/2009.00638
PDF
-
Learning to Rank for Active Learning: A Listwise Approach
2020-07-31 21:05:16Minghan Li, Xialei Liu, Joost van de Weijer, Bogdan RaducanuAbstract
Active learning emerged as an alternative to alleviate the effort to label huge amount of data for data hungry applications (such as image/video indexing and retrieval, autonomous driving, etc.). The goal of active learning is to automatically select a number of unlabeled samples for annotation (according to a budget), based on an acquisition function, which indicates how valuable a sample is for training the model. The learning loss method is a task-agnostic approach which attaches a module to learn to predict the target loss of unlabeled data, and select data with the highest loss for labeling. In this work, we follow this strategy but we define the acquisition function as a learning to rank problem and rethink the structure of the loss prediction module, using a simple but effective listwise approach. Experimental results on four datasets demonstrate that our method outperforms recent state-of-the-art active learning approaches for both image classification and regression tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2008.00078
PDF
-
Content-based Video Indexing and Retrieval Using Corr-LDA
2019-11-19 22:47:51Rahul Radhakrishnan Iyer, Sanjeel Parekh, Vikas Mohandoss, Anush Ramsurat, Bhiksha Raj, Rita SinghAbstract
Existing video indexing and retrieval methods on popular web-based multimedia sharing websites are based on user-provided sparse tagging. This paper proposes a very specific way of searching for video clips, based on the content of the video. We present our work on Content-based Video Indexing and Retrieval using the Correspondence-Latent Dirichlet Allocation (corr-LDA) probabilistic framework. This is a model that provides for auto-annotation of videos in a database with textual descriptors, and brings the added benefit of utilizing the semantic relations between the content of the video and text. We use the concept-level matching provided by corr-LDA to build correspondences between text and multimedia, with the objective of retrieving content with increased accuracy. In our experiments, we employ only the audio components of the individual recordings and compare our results with an SVM-based approach.
Abstract (translated)
URL
https://arxiv.org/abs/1602.08581
PDF
-
Investigating Correlations of Inter-coder Agreement and Machine Annotation Performance for Historical Video Data
2019-07-24 13:50:20Kader Pustu-Iren, Markus Mühling, Nikolaus Korfhage, Joanna Bars, Sabrina Bernhöft, Angelika Hörth, Bernd Freisleben, Ralph EwerthAbstract
Video indexing approaches such as visual concept classification and person recognition are essential to enable fine-grained semantic search in large-scale video archives such as the historical video collection of former German Democratic Republic (GDR) maintained by the German Broadcasting Archive (DRA). Typically, a lexicon of visual concepts has to be defined for semantic search. However, the definition of visual concepts can be more or less subjective due to individually differing judgments of annotators, which may have an impact on annotation quality and subsequently training of supervised machine learning methods. In this paper, we analyze the inter-coder agreement for historical TV data of the former GDR for visual concept classification and person recognition. The inter-coder agreement is evaluated for a group of expert as well as non-expert annotators in order to determine differences in annotation homogeneity. Furthermore, correlations between visual recognition performance and inter-annotator agreement are measured. In this context, information about image quantity and agreement are used to predict average precision for concept classification. Finally, the influence of expert vs. non-expert annotations acquired in the study are used to evaluate person recognition.
Abstract (translated)
URL
https://arxiv.org/abs/1907.10450
PDF
-
Video OCR for Video Indexing
2011-09-30 15:51:12Sankirti S., P. M. KamadeAbstract
Video OCR is a technique that can greatly help to locate the topics of interest in video via the automatic extraction and reading of captions and annotations. Text in video can provide key indexing information. Recognizing such text for search application is critical. Major difficult problem for character recognition for videos is degraded and deformated characters, low resolution characters or very complex background. To tackle the problem preprocessing on text image plays vital role. Most of the OCR engines are working on the binary image so to find a better binarization procedure for image to get a desired result is important.Accurate binarization process minimizes the error rate of video OCR.
Abstract (translated)
URL
https://arxiv.org/abs/1109.6862
PDF
