Video_Classification
-
MMTA: Benchmarking Multimodal Temporal Analysis with Time Series, Text, and Vision
2026-10-14 05:39:34Ziyang Zhang, Shenyi Li, Yilin Wang, Ziyun Cui, Bowen Zhou, Wen Wu, Chao ZhangAbstract
Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce MMTA (multimodal temporal analysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains -- robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate -- under video+TS and image+TS layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce TimeOmni-v, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.
Abstract (translated)
URL
https://arxiv.org/abs/2610.17856
PDF
-
Towards Cost-Efficient Video Dataset Construction with Large Language Models
2026-10-12 14:26:44Mohammed Shabot, Jonathan Adams, Jan Van den Stock, Joost VennekensAbstract
Research on video classification models has witnessed significant growth recently, accompanied by a significant increase in the demand for datasets to train such models. Existing video dataset curation approaches share a common bottleneck: candidate videos must be downloaded before any relevance judgment is made, which results in storage, time and annotation costs. In this work, we present a filtering step, using Large Language Models (LLMs) to pre-screen videos based on their metadata, before actually downloading these videos, thereby saving on these costs. We evaluate the LLMs against a strict human majority vote derived from four independent annotators and validate the reliability of the annotations through inter-annotator agreement analysis (Gwet's AC1). We compare the LLM pre-filter against two no-LLM baselines, namely two keyword heuristics. Via a Leave-One-Rater-Out (LOO) mechanism and an Add-One-Rater-In (AORI), we evaluate the performance of the LLMs as virtual annotators. Finally, we report cost comparisons relative to existing approaches. To ensure reproducability, we made our implementation available on github.
Abstract (translated)
URL
https://arxiv.org/abs/2610.16857
PDF
-
VCR-Bench: A Modular Open-Source Benchmark for Video Classification Robustness
2026-10-06 18:03:58Maksim Plinskiy, Aleksandr Gushchin, Sergey Lavrushkin, Dmitriy S. Vatolin, Anastasia AntsiferovaAbstract
Robustness of image classification has several benchmarks, but their video counterparts are absent. In video classification temporal dimension introduces additional degrees of freedom for adversarial attacks, defenses, and preprocessing. Temporal sampling, perturbation budgets, and metric aggregation also interact in ways with no direct analogue in the image setting. Therefore, robustness for video classifiers is studied across scattered, incompatible implementations, making reported numbers hard to reproduce and analyze. We introduce VCR-Bench, a modular open-source benchmark framework that standardizes video loading, wrappers for classifiers, adversarial attacks and defenses, perceptual metrics, configuration presets, and result logging. VCR-Bench currently integrates 30 video classification models, 14 adversarial attacks, and 10 defense wrappers under a common evaluation protocol. We evaluate representative video classifiers, attacks, and defenses on Kinetics-400 subset, reporting clean accuracy, attack success rate, perceptual quality, runtime, and memory usage. VCR-Bench is released with documented installation, reproducible run presets, component-extension interfaces, and scripts for reproducing the reported results at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2610.08936
PDF
-
Hierarchy-Aware and Anatomy-Guided Learning for Lung Ultrasound Video Classification
2026-07-20 04:55:28Alya Almsouti, Lotfi Mecharbat, Noha Aboukhater, Yousef Alabrach, Siddiq Anwar, Andre Kumar, Ibrahim Almakky, Mohammad YaqubAbstract
Lung ultrasound (LUS) is a bedside tool for assessing pulmonary edema in patients at risk due to heart failure or impaired kidney function. However, automated LUS analysis remains challenging because of speckle noise, imaging artifacts, and operator-dependent acquisition variability. In this work, we present a deep learning framework for multi-class LUS video classification that explores two components: hierarchy-aware training, and anatomy-guided learning. Starting from a strong baseline, we introduce hierarchical training strategies and then introduce pleural line mask supervision to guide model attention toward anatomically relevant regions. We study four clinically relevant classes--healthy, B-lines, consolidations, and mixed B-lines with consolidations--using an open-access dataset of 1,886 videos from 219 patients, evaluated with patient-level five-fold cross-validation. Results show that hierarchy-aware training improves pathological separation relative to flat classification, while mask-guided attention supervision achieves the highest mean macro-F1 of 65.7\% and produces more localized attention patterns. Transfer experiments on the external COVID-BLUeS dataset further show competitive and parameter-efficient adaptation while preserving pleural-focused attention behavior. These findings suggest that combining clinically structured objectives with anatomy-guided supervision is a practical approach to robust, interpretable LUS video analysis. Code and model implementations are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2607.17551
PDF
-
Learning to Unify Deformable Shape and Texture Representations for Cardiac Video Classification
2026-07-08 15:15:16Tonmoy Hossain, Miaomiao ZhangAbstract
Deformable shape representations have proven to be robust complements to texture features in cardiac image classification, offering geometric priors that are invariant to imaging artifacts and intensity variations. However, existing deep networks perform simple concatenation to combine these distinct feature representations, which neither fully exploits their complementary nature nor learns cross-modal feature dependencies. Furthermore, this results in uniform attention across all timepoints; hence ignoring the varying diagnostic importance across the cardiac phases. In this paper, we propose a novel cardiac video classification model that, for the first time, learns temporal features in an integrated space of deformable shape and image texture representations. In particular, we design a bi-directional cross-attention in the latent space to fuse latent deformable shape and image features, allowing each modality to adaptively weight the other based on spatio-temporal correspondence. In contrast to current methods that apply uniform weighting across all the cardiac phases, our approach learns to dynamically adjust the contributions of shape and texture representations, derived from images, over time. We demonstrate state-of-the-art classification performance on a cine cardiac magnetic resonance (CMR) video dataset, achieving improved interpretability from attention mechanisms that identify diagnostically critical cardiac phases and modality contributions.
Abstract (translated)
URL
https://arxiv.org/abs/2607.07518
PDF
-
Gen4U: Unifying Video Generation and Understanding via Diffusion
2026-07-07 23:17:26Michael King, Aravindh Mahendran, Matthew Koichi Grimes, Fedor Kitashov, Adham Elarabawy, Pedro Velez, Maks Ovsjanikov, Viorica P\u{a}tr\u{a}uceanAbstract
Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning). Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.
Abstract (translated)
URL
https://arxiv.org/abs/2607.06856
PDF
-
Forget, Anticipate and Adapt: Test Time Training for Long Videos
2026-06-25 01:40:10Rajat Modi, Sebastian Noel, Xin Liang, Yogesh Singh RawatAbstract
Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute. We present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a surprise metric: how much new information the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFNs empirical effectiveness on dense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos.
Abstract (translated)
URL
https://arxiv.org/abs/2606.26515
PDF
-
Query-Efficient Video Adversarial Attack with Stylized Logo on Service Computing
2026-06-16 07:37:18Duoxun Tang, Yuxin Cao, Xi Xiao, Derui Wang, Sheng Wen, Tianqing ZhuAbstract
In service computing, video classification has become fundamental to many intelligent applications. While Deep Neural Networks (DNNs) have demonstrated excellent performance in recognizing video content, recent studies have shown that DNNs are highly vulnerable to adversarial examples. Thus, understanding adversarial attacks can better respond to emergency situations. In order to improve attack performance, many style-transfer-based attacks and patch-based attacks have been proposed. However, the global perturbation of the former will bring unnatural global colors, while the latter is difficult to achieve success in targeted attacks due to the limited perturbation space. Moreover, compared to a plethora of methods targeting image classifiers, video adversarial attacks remain relatively underexplored. Therefore, to generate adversarial examples with a low budget and to provide them with a higher verisimilitude, we propose a novel black-box video attack framework, called Stylized Logo Attack (SLA). SLA is conducted through three stages. The first stage involves building a style reference set for logos, which can not only make the generated examples more natural, but also carry more target class features in targeted attacks. Then, Reinforcement Learning is employed to determine the style reference and position parameters of the logo within the video, which ensures that the stylized logo is placed in the video with optimal attributes. Finally, perturbations are optimized in a step-by-step manner so as to improve the fooling rate. Experimental results indicate that SLA can achieve better performance than state-of-the-art methods and still maintain good deception effects when facing various defense methods. We believe SLA can raise awareness among the security community about the reliability and security of video classification systems and serve as a memorandum of possible attack methods.
Abstract (translated)
URL
https://arxiv.org/abs/2408.12099
PDF
-
Spatio-Temporal Fusion Model for Standard View Classification of Echocardiographic Videos
2026-06-16 02:37:43Bo Gou, Jicheng Zhang, Jianlong Xiong, Tao He, Bentian Liu, Hai Wu, Yijiao Wang, Yu Zhang, Yujia Yang, Yun Dai, Jian Liu, Jie WangAbstract
Automated classification of standard echocardiographic views is crucial for efficient clinical workflow but faces three main challenges. First, publicly available datasets are scarce and limited in scale and view coverage. Second, the performance of some modern video-level architectures for echocardiographic view classification remains underexplored. Third, some view categories exhibit highly similar spatial appearances, making single-frame features insufficient for discrimination, while heterogeneous frame quality complicates robust temporal information fusion. To address these challenges, we release the Echocardiographic Videos of Nine Views (EV9V) dataset, comprising 5,138 videos, 910,579 frames, and 9 standard views, which is, to the best of our knowledge, the largest publicly available echocardiography video dataset. Using EV9V, we systematically benchmark representative video classification architectures, including Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers. Furthermore, we propose a Spatio-Temporal Fusion Model (STFM), an efficient dual-stream CNN-LSTM (Long Short-Term Memory) framework that jointly captures spatial anatomical structures and temporal cardiac dynamics. The proposed framework leverages uncertainty-aware learning to preferentially sample representative video segments during training and evidence-based fusion during inference, improving robustness to variations in frame quality across echocardiographic videos. Extensive experiments demonstrate that our method achieves competitive performance across diverse video classification models, validating the effectiveness of uncertainty-aware spatio-temporal learning for echocardiographic view classification. The code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2606.17437
PDF
-
Interpretable Temporal Facial-Region Motion Analysis for In-the-Wild Parkinson's Disease Video Classification
2026-06-08 19:15:39Riyadh Almushrafy (Majmaah University, Saudi Arabia)Abstract
Reduced facial expressivity is a common motor manifestation of Parkinson's disease (PD), often described as hypomimia or facial bradykinesia. This paper examines whether temporal motion descriptors extracted from facial-region keypoints can support in-the-wild PD-related video classification on the YouTubePD benchmark. Each video is represented using geometric descriptors from 14 predefined facial regions. Static geometry, normalized geometry, velocity-based descriptors, relative-velocity descriptors, and a GRU sequence baseline are compared under the same binary classification protocol. To assess stability and interpretability, the study includes seed-robustness analysis, region-level ablation, and permutation importance. The best result is obtained with normalized velocity descriptors and a Random Forest classifier, reaching a balanced accuracy of 0.826 and an AUROC of 0.855 on the held-out test split. Across 10 random seeds, this representation remains stable, with balanced accuracy of 0.810 +/- 0.018 and AUROC of 0.855 +/- 0.005. Overall, the results suggest that normalized facial-region motion is a lightweight and interpretable representation for YouTubePD video classification. The study is framed as a benchmark-level analysis and does not claim clinical severity assessment or MDS-UPDRS facial-expression scoring.
Abstract (translated)
URL
https://arxiv.org/abs/2606.10088
PDF
-
Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videos
2026-05-18 18:04:54Yuqi Tang, Yang Shi, Zhuoran Zhang, Qixun Wang, Xuehai Bai, Yue Ding, Ruizhe Chen, Bohan Zeng, Xinlong Chen, Xuanyu Zhu, Bozhou Li, Yuran Wang, Yifan Dai, Chengzhuo Tong, Xinyu Liu, Yiyan Ji, Yujie Wei, Yuhao Dong, Shilin Yan, Fengxiang Wang, Yi-Fan Zhang, Haotian Wang, Yuanxing Zhang, Pengfei WanAbstract
Recent video generative models have greatly improved the realism of AI-generated videos, yet their outputs still exhibit artifacts such as temporal inconsistencies, structural distortions, and semantic incoherence. While Multimodal Large Language Models (MLLMs) show strong visual understanding capabilities, their ability to perceive and reason about such artifacts remains unclear. Existing benchmarks often lack systematic evaluation of artifact-aware perception and fine-grained diagnostic reasoning, especially across diverse AI-generated video domains beyond photorealistic content. To address this gap, we introduce Artifact-Bench, a comprehensive benchmark for evaluating MLLMs on AI-generated video artifact detection and analysis. We first establish a three-level hierarchical taxonomy of realism artifacts, covering photorealistic, animated, and CG-style videos. Based on this taxonomy, Artifact-Bench defines three complementary tasks: real vs. AI-generated video classification, pairwise realism comparison, and fine-grained artifact identification. Experiments on 19 leading MLLMs reveal substantial limitations in artifact perception and reasoning, with many models approaching random or even below-random performance in challenging settings. We further observe significant misalignment between MLLM judgments and human perceptual preferences, highlighting their limited reliability as general evaluators for AI-generated video realism.
Abstract (translated)
URL
https://arxiv.org/abs/2605.18984
PDF
-
Appearance-free Action Recognition: Zero-shot Generalization in Humans and a Two-Pathway Model
2026-04-17 20:13:52Prerana Kumar (Hertie Institute, University of Tuebingen, IMPRS-IS), Martin A. Giese (Hertie Institute, University of Tuebingen)Abstract
Action recognition is a fundamental ability for social species. Yet, its underlying computations are not well understood. Classical psychophysical studies using simplified stimuli have shown that humans can perceive body motion even under degradation of relevant shape cues. Recent work using real-world action videos and their appearance-free counterparts (that preserve motion but lack static shape cues) included explicit training of humans and models on the appearance-free videos. Whether humans and vision models generalize in a zero-shot manner to appearance-free transformations of real-world action videos is not yet known. To measure this generalization in humans, we conducted a laboratory-based psychophysics experiment. 22 participants were trained to recognize five action categories using naturalistic videos (UCF5 dataset), and tested zero-shot on two types of appearance-free transformations: (i) dense-noise motion videos from an existing dataset (AFD5) and (ii) random-dot appearance-free videos. We find that participants recognize actions in both types of appearance-free videos well above chance, albeit with reduced accuracy compared to naturalistic videos. To model this behavior, we developed a two-pathway 3D CNN-based model combining an RGB (form) stream and an optical flow (motion) stream, including a coherence-gating mechanism inspired by Gestalt common-fate grouping. Our model generalizes to both appearance-free datasets and outperforms contemporary video classification models, narrowing the gap to human performance. We find that the motion pathway is critical for generalization to appearance-free videos, while the form pathway improves performance on naturalistic videos. Our findings highlight the importance of motion-based representations for generalization to appearance-free videos, and support the use of multi-stream architectures to model video-based action recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2604.16675
PDF
-
LOLGORITHM: Funny Comment Generation Agent For Short Videos
2026-04-09 11:58:54Xuan Ouyang, Senan Wang, Bouzhou Wang, Siyuan Xiahou, Jinrong Zhou, Yuekang LiAbstract
Short-form video platforms have become central to multimedia information dissemination, where comments play a critical role in driving engagement, propagation, and algorithmic feedback. However, existing approaches -- including video summarization and live-streaming danmaku generation -- fail to produce authentic comments that conform to platform-specific cultural and linguistic norms. In this paper, we present LOLGORITHM, a novel modular multi-agent framework for stylized short-form video comment generation. LOLGORITHM supports six controllable comment styles and comprises three core modules: video content summarization, video classification, and comment generation with semantic retrieval and hot meme augmentation. We further construct a bilingual dataset of 3,267 videos and 16,335 comments spanning five high-engagement categories across YouTube and Douyin. Evaluation combining automatic scoring and large-scale human preference analysis demonstrates that LOLGORITHM consistently outperforms baseline methods, achieving human preference selection rates of 80.46\% on YouTube and 84.29\% on Douyin across 107 respondents. Ablation studies confirm that these gains are attributable to the framework architecture rather than the choice of backbone LLM, underscoring the robustness and generalizability of our approach.
Abstract (translated)
URL
https://arxiv.org/abs/2604.09729
PDF
-
vAccSOL: Efficient and Transparent AI Vision Offloading for Mobile Robots
2026-03-17 15:44:44Adam Zahir, Michele Gucciardom Falk Selker, Anastasios Nanos, Kostis Papazafeiropoulos, Carlos J. Bernardos, Nicolas Weber, Roberto GonzalezAbstract
Mobile robots are increasingly deployed for inspection, patrol, and search-and-rescue operations, relying on computer vision for perception, navigation, and autonomous decision-making. However, executing modern vision workloads onboard is challenging due to limited compute resources and strict energy constraints. While some platforms include embedded accelerators, these are typically tied to proprietary software stacks, leaving user-defined workloads to run on resource-constrained companion computers. We present vAccSOL, a framework for efficient and transparent execution of AI-based vision workloads across heterogeneous robotic and edge platforms. vAccSOL integrates two components: SOL, a neural network compiler that generates optimized inference libraries with minimal runtime dependencies, and vAccel, a lightweight execution framework that transparently dispatches inference locally on the robot or to nearby edge infrastructure. This combination enables hardware-optimized inference and flexible execution placement without requiring modifications to robot applications. We evaluate vAccSOL on a real-world testbed with a commercial quadruped robot and twelve deep learning models covering image classification, video classification, and semantic segmentation. Compared to a PyTorch compiler baseline, SOL achieves comparable or better inference performance. With edge offloading, vAccSOL reduces robot-side power consumption by up to 80% and edge-side power by up to 60% compared to PyTorch, while increasing vision pipeline frame rate by up to 24x, extending the operating lifetime of battery-powered robots.
Abstract (translated)
URL
https://arxiv.org/abs/2603.16685
PDF
-
HSEmotion Team at ABAW-10 Competition: Facial Expression Recognition, Valence-Arousal Estimation, Action Unit Detection and Fine-Grained Violence Classification
2026-03-13 06:13:46Andrey V. Savchenko, Kseniia TsypliakovaAbstract
This article presents our results for the 10th Affective Behavior Analysis in-the-Wild (ABAW) competition. For frame-wise facial emotion understanding tasks (frame-wise facial expression recognition, valence-arousal estimation, action unit detection), we propose a fast approach based on facial embedding extraction with pre-trained EfficientNet-based emotion recognition models. If the latter model's confidence exceeds a threshold, its prediction is used. Otherwise, we feed embeddings into a simple multi-layered perceptron trained on the AffWild2 dataset. Estimated class-level scores are smoothed in a sliding window of fixed size to mitigate noise in frame-wise predictions. For the fine-grained violence detection task, we examine several pre-trained architectures for frame embeddings and their aggregation for video classification. Experimental results on four tasks from the ABAW challenge demonstrate that our approach significantly improves validation metrics over existing baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2603.12693
PDF
-
Contrastive learning-based video quality assessment-jointed video vision transformer for video recognition
2026-03-11 16:51:55Jian Sun, Mohammad H. MahoorAbstract
Video quality significantly affects video classification. We found this problem when we classified Mild Cognitive Impairment well from clear videos, but worse from blurred ones. From then, we realized that referring to Video Quality Assessment (VQA) may improve video classification. This paper proposed Self-Supervised Learning-based Video Vision Transformer combined with No-reference VQA for video classification (SSL-V3) to fulfill the goal. SSL-V3 leverages Combined-SSL mechanism to join VQA into video classification and address the label shortage of VQA, which commonly occurs in video datasets, making it impossible to provide an accurate Video Quality Score. In brief, Combined-SSL takes video quality score as a factor to directly tune the feature map of the video classification. Then, the score, as an intersected point, links VQA and classification, using the supervised classification task to tune the parameters of VQA. SSL-V3 achieved robust experimental results on two datasets. For example, it reached an accuracy of 94.87% on some interview videos in the I-CONECT (a facial video-involved healthcare dataset), verifying SSL-V3's effectiveness.
Abstract (translated)
视频质量显著影响视频分类的效果。我们在对轻度认知障碍（Mild Cognitive Impairment）进行分类时发现，从清晰的视频中可以较好地识别，但从模糊的视频中则表现较差。因此我们意识到借鉴视频质量评估（VQA）的方法可能有助于改进视频分类任务。本文提出了一种基于自监督学习的视频视觉变换器结合无参考VQA的视频分类方法（SSL-V3），以实现这一目标。 SSL-V3利用了Combined-SSL机制，将VQA与视频分类结合起来，并解决了VQA中常见的标签短缺问题，在大多数视频数据集中难以提供准确的视频质量分数。简而言之，Combined-SSL通过将视频质量评分作为一个因素直接调整视频分类模型中的特征图来实现这一点。同时，该评分作为交点位置，连接了VQA和分类任务，使用监督学习下的分类任务来调整VQA参数。 在两个数据集上，SSL-V3取得了稳健的实验结果。例如，在I-CONECT（一个涉及面部视频的医疗健康数据集）中的某些访谈视频上，其准确率达到了94.87%，证明了SSL-V3的有效性。
URL
https://arxiv.org/abs/2603.10965
PDF
-
From Imitation to Intuition: Intrinsic Reasoning for Open-Instance Video Classification
2026-03-11 00:50:01Ke Zhang, Xiangchen Zhao, Yunjie Tian, Jiayu Zheng, Vishal M. Patel, Di FuAbstract
Conventional video classification models, acting as effective imitators, excel in scenarios with homogeneous data distributions. However, real-world applications often present an open-instance challenge, where intra-class variations are vast and complex, beyond existing benchmarks. While traditional video encoder models struggle to fit these diverse distributions, vision-language models (VLMs) offer superior generalization but have not fully leveraged their reasoning capabilities (intuition) for such tasks. In this paper, we bridge this gap with an intrinsic reasoning framework that evolves open-instance video classification from imitation to intuition. Our approach, namely DeepIntuit, begins with a cold-start supervised alignment to initialize reasoning capability, followed by refinement using Group Relative Policy Optimization (GRPO) to enhance reasoning coherence through reinforcement learning. Crucially, to translate this reasoning into accurate classification, DeepIntuit then introduces an intuitive calibration stage. In this stage, a classifier is trained on this intrinsic reasoning traces generated by the refined VLM, ensuring stable knowledge transfer without distribution mismatch. Extensive experiments demonstrate that for open-instance video classification, DeepIntuit benefits significantly from transcending simple feature imitation and evolving toward intrinsic reasoning. Our project is available at this https URL.
Abstract (translated)
传统的视频分类模型在数据分布同质化的情景下表现出色，然而，在现实世界的应用中经常遇到开放实例挑战。在这种情况下，同一类别的内部变化是广泛且复杂的，超出了现有的基准测试范围。尽管传统视频编码器模型难以适应这些多样的分布情况，视觉-语言模型（VLMs）提供了更好的泛化能力，但它们尚未完全利用其推理能力来处理此类任务。本文提出了一种内在推理框架，旨在将开放实例视频分类从模仿进化到直觉理解。我们为此开发的方法被称为DeepIntuit。 DeepIntuit方法首先通过冷启动监督对齐初始化推理能力，然后使用组相对策略优化（GRPO）进行细化，以增强通过强化学习实现的推理一致性。至关重要的是，为了将这种推理转化为准确的分类，DeepIntuit引入了一个直观校准阶段，在这个阶段中，一个分类器会在经过改进的VLM生成的内在推理痕迹上进行训练，确保知识传递稳定且无分布错配。 广泛的实验表明，对于开放实例视频分类而言，DeepIntuit方法通过超越简单的特征模仿并进化到内在推理获益颇丰。我们的项目可在此链接访问：[此URL]（请将括号内的内容替换为实际的网址）。
URL
https://arxiv.org/abs/2603.10300
PDF
-
Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video
2026-01-22 09:14:11Pascal Benschop, Justin Dauwels, Jan van GemertAbstract
Spatial reasoning in vision language models (VLMs) remains fragile when semantics hinge on subtle temporal or geometric cues. We introduce a synthetic benchmark that probes two complementary skills: situational awareness (recognizing whether an interaction is harmful or benign) and spatial awareness (tracking who does what to whom, and reasoning about relative positions and motion). Through minimal video pairs, we test three challenges: distinguishing violence from benign activity, binding assailant roles across viewpoints, and judging fine-grained trajectory alignment. While we evaluate recent VLMs in a training-free setting, the benchmark is applicable to any video classification model. Results show performance only slightly above chance across tasks. A simple aid, stable color cues, partly reduces assailant role confusions but does not resolve the underlying weakness. By releasing data and code, we aim to provide reproducible diagnostics and seed exploration of lightweight spatial priors to complement large-scale pretraining.
Abstract (translated)
在视觉语言模型（VLMs）中，空间推理依然脆弱，尤其是在语义依赖于微妙的时间或几何线索时。我们引入了一个合成基准测试，该测试考察两种互补技能：情境意识（识别互动是否具有危害性或无害）和空间意识（追踪谁对谁做了什么，并推断相对位置和运动）。通过最小化视频配对，我们测试了三个挑战：区分暴力行为与无害活动、在不同视角下绑定施暴者角色以及判断细微的轨迹对齐。尽管我们在零训练设置中评估了最近的VLMs，但该基准适用于任何视频分类模型。结果表明，在各个任务上的表现仅略高于随机猜测水平。一种简单的辅助手段——稳定颜色线索——部分减少了施暴者角色混淆，但仍未能解决根本弱点。通过发布数据和代码，我们旨在提供可重复诊断工具，并促进对轻量级空间先验的探索，以补充大规模预训练。
URL
https://arxiv.org/abs/2601.15780
PDF
-
Not all Blends are Equal: The BLEMORE Dataset of Blended Emotion Expressions with Relative Salience Annotations
2026-01-19 16:59:45Tim Lachmann, Alexandra Israelsson, Christina Tornberg, Teimuraz Saghinadze, Michal Balazia, Philipp M\"uller, Petri LaukkaAbstract
Humans often experience not just a single basic emotion at a time, but rather a blend of several emotions with varying salience. Despite the importance of such blended emotions, most video-based emotion recognition approaches are designed to recognize single emotions only. The few approaches that have attempted to recognize blended emotions typically cannot assess the relative salience of the emotions within a blend. This limitation largely stems from the lack of datasets containing a substantial number of blended emotion samples annotated with relative salience. To address this shortcoming, we introduce BLEMORE, a novel dataset for multimodal (video, audio) blended emotion recognition that includes information on the relative salience of each emotion within a blend. BLEMORE comprises over 3,000 clips from 58 actors, performing 6 basic emotions and 10 distinct blends, where each blend has 3 different salience configurations (50/50, 70/30, and 30/70). Using this dataset, we conduct extensive evaluations of state-of-the-art video classification approaches on two blended emotion prediction tasks: (1) predicting the presence of emotions in a given sample, and (2) predicting the relative salience of emotions in a blend. Our results show that unimodal classifiers achieve up to 29% presence accuracy and 13% salience accuracy on the validation set, while multimodal methods yield clear improvements, with ImageBind + WavLM reaching 35% presence accuracy and HiCMAE 18% salience accuracy. On the held-out test set, the best models achieve 33% presence accuracy (VideoMAEv2 + HuBERT) and 18% salience accuracy (HiCMAE). In sum, the BLEMORE dataset provides a valuable resource to advancing research on emotion recognition systems that account for the complexity and significance of blended emotion expressions.
Abstract (translated)
人类在某一时刻通常体验的不是单一的基本情绪，而是一种混合了多种不同强度情感的状态。尽管这种复合情感的重要性不容忽视，但大多数基于视频的情绪识别方法仅设计用于识别单一情绪。少数尝试识别复合情感的方法通常也无法评估这些情感之间的相对显著性。这一局限主要源于缺少包含大量注有相对显著性的复合情感样本的数据集。 为了解决这个问题，我们引入了BLEMORE数据集，这是一个新的多模态（视频、音频）混合情绪识别数据集，包含了每种情绪在混合中的相对显著信息。BLEMORE包括超过3000个片段，来自58名演员的表演，涵盖了6种基本情感和10种不同的复合情感组合，其中每一种复合情感有三种不同的情感强度配置（50/50, 70/30 和 30/70）。利用这个数据集，我们对最先进的视频分类方法进行了广泛的评估，针对两个混合情绪预测任务：(1) 预测给定样本中是否存在特定情绪；(2) 预测情感混合中的相对显著性。我们的结果显示，在验证集中，单模态分类器在存在准确性上可达到最高29%，在显著性准确性上可达13%；而多模态方法则明显表现出改进，其中ImageBind + WavLM达到了35%的存在准确性，HiCMAE的显著性准确率为18%。在未公开的测试集中，最佳模型实现了VideoMAEv2 + HuBERT的33%存在准确性和HiCMAE的18%显著性准确度。 综上所述，BLEMORE数据集为推进考虑情感表达复杂性和重要性的混合情绪识别系统研究提供了宝贵的资源。
URL
https://arxiv.org/abs/2601.13225
PDF
-
Effects of Different Attention Mechanisms Applied on 3D Models in Video Classification
2026-01-15 20:45:56Mohammad Rasras, Iuliana Marin, Serban Radu, Irina MocanuAbstract
Human action recognition has become an important research focus in computer vision due to the wide range of applications where it is used. 3D Resnet-based CNN models, particularly MC3, R3D, and R(2+1)D, have different convolutional filters to extract spatiotemporal features. This paper investigates the impact of reducing the captured knowledge from temporal data, while increasing the resolution of the frames. To establish this experiment, we created similar designs to the three originals, but with a dropout layer added before the final classifier. Secondly, we then developed ten new versions for each one of these three designs. The variants include special attention blocks within their architecture, such as convolutional block attention module (CBAM), temporal convolution networks (TCN), in addition to multi-headed and channel attention mechanisms. The purpose behind that is to observe the extent of the influence each of these blocks has on performance for the restricted-temporal models. The results of testing all the models on UCF101 have shown accuracy of 88.98% for the variant with multiheaded attention added to the modified R(2+1)D. This paper concludes the significance of missing temporal features in the performance of the newly created increased resolution models. The variants had different behavior on class-level accuracy, despite the similarity of their enhancements to the overall performance.
Abstract (translated)
人类行为识别由于其广泛的应用场景，已成为计算机视觉领域的重要研究焦点。基于3D Resnet的CNN模型（如MC3、R3D和R(2+1)D）通过不同的卷积滤波器来提取空间-时间特征。本文探讨了减少捕获的时间数据知识的影响，并增加帧分辨率的效果。为了进行这项实验，我们创建了与原始设计相似的设计，但在最终分类器之前添加了一个dropout层。其次，为每种设计方案开发了十个新版本，这些变体在其架构中加入了特殊的注意模块，如卷积块注意力模块（CBAM）、时间卷积网络（TCN），以及多头和通道注意机制。 引入这些模块的目的是观察它们在限制时间特征的新建高分辨率模型性能中的影响程度。所有模型在UCF101数据集上的测试结果表明，在修改后的R(2+1)D中添加了多头注意力机制的变体获得了88.98%的准确率。本文得出结论，缺失的时间特性对新创建的、增加分辨率的模型性能具有重要意义。尽管这些变体对其整体性能进行了类似的增强，但它们在类别级别的准确性上表现出不同的行为。
URL
https://arxiv.org/abs/2601.10854
PDF
