Abstract
Engagement analysis finds various applications in healthcare, education, advertisement, services. Deep Neural Networks, used for analysis, possess complex architecture and need large amounts of input data, computational power, inference time. These constraints challenge embedding systems into devices for real-time use. To address these limitations, we present a novel two-stream feature fusion "Tensor-Convolution and Convolution-Transformer Network" (TCCT-Net) architecture. To better learn the meaningful patterns in the temporal-spatial domain, we design a "CT" stream that integrates a hybrid convolutional-transformer. In parallel, to efficiently extract rich patterns from the temporal-frequency domain and boost processing speed, we introduce a "TC" stream that uses Continuous Wavelet Transform (CWT) to represent information in a 2D tensor form. Evaluated on the EngageNet dataset, the proposed method outperforms existing baselines, utilizing only two behavioral features (head pose rotations) compared to the 98 used in baseline models. Furthermore, comparative analysis shows TCCT-Net's architecture offers an order-of-magnitude improvement in inference speed compared to state-of-the-art image-based Recurrent Neural Network (RNN) methods. The code will be released at this https URL.
Abstract (translated)
翻译： Engagement分析在医疗、教育、广告和服务等领域发现了各种应用。用于分析的Deep Neural Networks具有复杂的架构，需要大量的输入数据、计算能力和推理时间。这些限制将嵌入系统推向实时使用设备。为解决这些限制，我们提出了一个名为"Tensor-Convolution and Convolution-Transformer Network"（TCCT-Net）的新颖架构。为了更好地学习时间域中的有意义模式，我们设计了一个"CT"流，该流整合了一个混合卷积-Transformer。并行地，为了有效地从时间频域中提取丰富模式并提高处理速度，我们引入了一个"TC"流，该流使用连续波形变换（CWT）将信息表示为二维张量的形式。在EngageNet数据集上评估，与基线模型相比，所提出的方法仅使用了两个行为特征（头姿态旋转），但性能优于98个基线模型。此外，比较分析显示，TCCT-Net的架构比最先进的基于图像的循环神经网络（RNN）方法具有数量级的改进。代码将在这个https URL上发布。
URL
https://arxiv.org/abs/2404.09474
