Abstract
The development of various sensing technologies is improving measurements of stress and the well-being of individuals. Although progress has been made with single signal modalities like wearables and facial emotion recognition, integrating multiple modalities provides a more comprehensive understanding of stress, given that stress manifests differently across different people. Multi-modal learning aims to capitalize on the strength of each modality rather than relying on a single signal. Given the complexity of processing and integrating high-dimensional data from limited subjects, more research is needed. Numerous research efforts have been focused on fusing stress and emotion signals at an early stage, e.g., feature-level fusion using basic machine learning methods and 1D-CNN Methods. This paper proposes a multi-modal learning approach for stress detection that integrates facial landmarks and biometric signals. We test this multi-modal integration with various early-fusion and late-fusion techniques to integrate the 1D-CNN model from biometric signals and 2-D CNN using facial landmarks. We evaluate these architectures using a rigorous test of models' generalizability using the leave-one-subject-out mechanism, i.e., all samples related to a single subject are left out to train the model. Our findings show that late-fusion achieved 94.39\% accuracy, and early-fusion surpassed it with a 98.38\% accuracy rate. This research contributes valuable insights into enhancing stress detection through a multi-modal approach. The proposed research offers important knowledge in improving stress detection using a multi-modal approach.
Abstract (translated)
各种传感技术的不断发展提高了对压力和个体健康状况的测量。虽然单信号模态（如可穿戴设备和面部表情识别）的进步已经取得，但整合多个模态提供了对压力更全面的了解，因为压力在每个人身上表现方式不同。多模态学习旨在利用每个模态的优势，而不是依赖单一信号。考虑到处理和整合高维数据的精复杂度，需要进行更多的研究。已经有很多研究将压力和情感信号在早期阶段进行融合，例如使用基本机器学习方法和1D-CNN方法进行特征级融合。本文提出了一种多模态学习方法来进行压力检测，整合面部特征和生物特征信号。我们测试了这些早期融合和晚期融合技术对1D-CNN模型和2D CNN模型的整合效果。我们使用严格的模型泛化测试来评估这些架构，即所有与单一主题相关的样本都被排除，以训练模型。我们的研究结果表明，晚期融合获得了94.39%的准确率，而早期融合超过了98.38%的准确率。这项研究为通过多模态方法增强压力检测提供了宝贵的洞见。所提出的研究为通过多模态方法提高压力检测提供了重要的知识。
URL
https://arxiv.org/abs/2311.03606
