Multi_Modal
-
WiFi based Human Fall and Activity Recognition using Transformer based Encoder Decoder and Graph Neural Networks
2025-04-23 12:22:24Younggeol Cho, Elisa Motta, Olivia Nocentini, Marta Lagomarsino, Andrea Merello, Marco Crepaldi, Arash AjoudaniAbstract
Human pose estimation and action recognition have received attention due to their critical roles in healthcare monitoring, rehabilitation, and assistive technologies. In this study, we proposed a novel architecture named Transformer based Encoder Decoder Network (TED Net) designed for estimating human skeleton poses from WiFi Channel State Information (CSI). TED Net integrates convolutional encoders with transformer based attention mechanisms to capture spatiotemporal features from CSI signals. The estimated skeleton poses were used as input to a customized Directed Graph Neural Network (DGNN) for action recognition. We validated our model on two datasets: a publicly available multi modal dataset for assessing general pose estimation, and a newly collected dataset focused on fall related scenarios involving 20 participants. Experimental results demonstrated that TED Net outperformed existing approaches in pose estimation, and that the DGNN achieves reliable action classification using CSI based skeletons, with performance comparable to RGB based systems. Notably, TED Net maintains robust performance across both fall and non fall cases. These findings highlight the potential of CSI driven human skeleton estimation for effective action recognition, particularly in home environments such as elderly fall detection. In such settings, WiFi signals are often readily available, offering a privacy preserving alternative to vision based methods, which may raise concerns about continuous camera monitoring.
Abstract (translated)
人体姿态估计和动作识别因其在医疗监护、康复及辅助技术中的关键作用而受到了广泛关注。本研究提出了一种名为基于变换器的编码解码网络（TED Net）的新架构，用于从WiFi信道状态信息（CSI）中估算人体骨骼姿势。TED Net结合了卷积编码器与基于变换器的注意力机制，以捕捉来自CSI信号的时空特征。所估计的人体姿态作为输入被送入一种定制化的有向图神经网络（DGNN），用于动作识别。我们在两个数据集上验证了我们的模型：一个是公开可用的多模态数据集，用于评估一般姿势估计；另一个是新收集的数据集，重点关注涉及20名参与者的与跌倒相关的场景。实验结果显示，TED Net在姿态估计方面超越了现有方法，并且DGNN能够使用基于CSI的人体骨骼进行可靠的动作分类，性能与基于RGB的方法相当。值得注意的是，TED Net在跌倒和非跌倒情况下都保持了稳健的性能。这些发现突显了CSI驱动的人体骨架估算在有效动作识别中的潜力，尤其是在如老人跌倒检测这样的家庭环境中。在这种环境下，WiFi信号往往易于获取，并提供了一种隐私保护的方法来替代基于视觉的方法，后者可能因持续摄像监控而引起担忧。
URL
https://arxiv.org/abs/2504.16655
PDF
-
Advancing Chronic Tuberculosis Diagnostics Using Vision-Language Models: A Multi modal Framework for Precision Analysis
2025-03-17 13:49:29Praveen Shastry, Sowmya Chowdary Muthulur, Naveen Kumarasami, Anandakumar D, Mounigasri M, Keerthana R, Kishore Prasath Venkatesh, Bargava Subramanian, Kalyan Sivasailam, Revathi Ezhumalai, Abitha MarimuthuAbstract
Background This study proposes a Vision-Language Model (VLM) leveraging the SIGLIP encoder and Gemma-3b transformer decoder to enhance automated chronic tuberculosis (TB) screening. By integrating chest X-ray images with clinical data, the model addresses the challenges of manual interpretation, improving diagnostic consistency and accessibility, particularly in resource-constrained settings. Methods The VLM architecture combines a Vision Transformer (ViT) for visual encoding and a transformer-based text encoder to process clinical context, such as patient histories and treatment records. Cross-modal attention mechanisms align radiographic features with textual information, while the Gemma-3b decoder generates comprehensive diagnostic reports. The model was pre-trained on 5 million paired medical images and texts and fine-tuned using 100,000 chronic TB-specific chest X-rays. Results The model demonstrated high precision (94 percent) and recall (94 percent) for detecting key chronic TB pathologies, including fibrosis, calcified granulomas, and bronchiectasis. Area Under the Curve (AUC) scores exceeded 0.93, and Intersection over Union (IoU) values were above 0.91, validating its effectiveness in detecting and localizing TB-related abnormalities. Conclusion The VLM offers a robust and scalable solution for automated chronic TB diagnosis, integrating radiographic and clinical data to deliver actionable and context-aware insights. Future work will address subtle pathologies and dataset biases to enhance the model's generalizability, ensuring equitable performance across diverse populations and healthcare settings.
Abstract (translated)
背景：这项研究提出了一种利用SIGLIP编码器和Gemma-3b解码器的视觉语言模型（VLM），以增强自动慢性结核病（TB）筛查。通过整合胸部X光影像与临床数据，该模型旨在解决手动解读的挑战，提高诊断的一致性和可访问性，特别是在资源有限的情况下。 方法：VLM架构结合了用于视觉编码的Vision Transformer（ViT）和基于变压器的文本编码器以处理如患者病史和治疗记录等临床背景信息。跨模态注意机制将放射学特征与文本信息对齐，而Gemma-3b解码器则生成全面的诊断报告。该模型在500万张配对的医学影像和文本数据上进行了预训练，并通过10万张特定于慢性TB的胸部X光片进行了微调。 结果：该模型对于检测慢性结核病的关键病理特征，如纤维化、钙化的肉芽肿和支气管扩张等表现出了高精度（94%）和召回率（94%）。在曲线下面积（AUC）评分超过0.93，交并比（IoU）值高于0.91，验证了其在检测和定位与结核病相关的异常方面具有有效性。 结论：VLM为自动慢性结核诊断提供了一种稳健且可扩展的解决方案，通过整合放射学和临床数据来交付实际可行且情境感知的洞察。未来的工作将解决细微病理特征和数据集偏差的问题，以增强模型在多样人口和医疗环境中的泛化能力，确保其性能的公平性。
URL
https://arxiv.org/abs/2503.14536
PDF
-
X-SG$^2$S: Safe and Generalizable Gaussian Splatting with X-dimensional Watermarks
2025-02-13 17:59:15Zihang Cheng, Huiping Zhuang, Chun Li, Xin Meng, Ming Li, Fei Richard YuAbstract
3D Gaussian Splatting (3DGS) has been widely used in 3D reconstruction and 3D generation. Training to get a 3DGS scene often takes a lot of time and resources and even valuable inspiration. The increasing amount of 3DGS digital asset have brought great challenges to the copyright protection. However, it still lacks profound exploration targeted at 3DGS. In this paper, we propose a new framework X-SG$^2$S which can simultaneously watermark 1 to 3D messages while keeping the original 3DGS scene almost unchanged. Generally, we have a X-SG$^2$S injector for adding multi-modal messages simultaneously and an extractor for extract them. Specifically, we first split the watermarks into message patches in a fixed manner and sort the 3DGS points. A self-adaption gate is used to pick out suitable location for watermarking. Then use a XD(multi-dimension)-injection heads to add multi-modal messages into sorted 3DGS points. A learnable gate can recognize the location with extra messages and XD-extraction heads can restore hidden messages from the location recommended by the learnable gate. Extensive experiments demonstrated that the proposed X-SG$^2$S can effectively conceal multi modal messages without changing pretrained 3DGS pipeline or the original form of 3DGS parameters. Meanwhile, with simple and efficient model structure and high practicality, X-SG$^2$S still shows good performance in hiding and extracting multi-modal inner structured or unstructured messages. X-SG$^2$S is the first to unify 1 to 3D watermarking model for 3DGS and the first framework to add multi-modal watermarks simultaneous in one 3DGS which pave the wave for later researches.
Abstract (translated)
3D高斯点阵（3D Gaussian Splatting，简称3DGS）在三维重建和生成领域得到了广泛应用。为了获得一个3DGS场景，通常需要消耗大量时间和资源，并且有时还需要有价值的创意灵感。随着3DGS数字资产数量的不断增加，版权保护也面临着巨大的挑战。然而，针对3DGS的研究仍处于初步阶段。在此论文中，我们提出了一种新的框架X-SG$^2$S，它可以同时嵌入1到3维的信息，而几乎不改变原始的3DGS场景。总的来说，我们的框架包含一个X-SG$^2$S注入器来添加多模态信息，并且有一个提取器用于从中提取这些信息。 具体来说，我们首先以固定的方式将水印分割成消息补丁，并对3DGS点进行排序。使用自适应门选择适合嵌入水印的位置。然后利用XD（多维）-注射头向已排序的3DGS点中添加多模态信息。一个可学习的门能够识别带有额外信息的位置，而XD提取头可以从由该可学习门推荐的位置中恢复隐藏的信息。 广泛的实验表明，所提出的X-SG$^2$S能够在不改变预训练的3DGS流程或原始参数形式的情况下有效隐藏多模态消息。同时，由于模型结构简单且高效，并具有高度实用性，X-SG$^2$S在隐藏和提取多模式内部结构化或非结构化信息方面表现良好。 作为首个统一1到3维水印嵌入模型的框架以及首个在同一3DGS中同时添加多模态水印的框架，X-SG$^2$S为后续研究铺平了道路。
URL
https://arxiv.org/abs/2502.10475
PDF
-
Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning
2025-02-08 07:32:48Subin Kim, Hoonrae Kim, Heejin Do, Gary Geunbae LeeAbstract
Previous research has revealed the potential of large language models (LLMs) to support cognitive reframing therapy; however, their focus was primarily on text-based methods, often overlooking the importance of non-verbal evidence crucial in real-life therapy. To alleviate this gap, we extend the textual cognitive reframing to multimodality, incorporating visual clues. Specifically, we present a new dataset called Multi Modal-Cognitive Support Conversation (M2CoSC), which pairs each GPT-4-generated dialogue with an image that reflects the virtual client's facial expressions. To better mirror real psychotherapy, where facial expressions lead to interpreting implicit emotional evidence, we propose a multi-hop psychotherapeutic reasoning approach that explicitly identifies and incorporates subtle evidence. Our comprehensive experiments with both LLMs and vision-language models (VLMs) demonstrate that the VLMs' performance as psychotherapists is significantly improved with the M2CoSC dataset. Furthermore, the multi-hop psychotherapeutic reasoning method enables VLMs to provide more thoughtful and empathetic suggestions, outperforming standard prompting methods.
Abstract (translated)
之前的研究已经揭示了大型语言模型（LLM）在认知重构疗法中的潜力；然而，这些研究主要集中在文本方法上，往往忽略了非言语证据在现实生活中治疗中的重要性。为了解决这一差距，我们将基于文本的认知重构扩展到了多模态领域，引入视觉线索。具体来说，我们提出了一个新的数据集——多模态认知支持对话（M2CoSC），该数据集中每一段由GPT-4生成的对话都配有一张反映虚拟客户面部表情的图片。为了更好地模拟现实生活中的心理治疗，在这种情境中，面部表情用于解读隐含的情感线索，我们提出了一种多层次的心理治疗方法，这种方法能够明确识别并整合细微证据。我们的全面实验包括LLM和视觉语言模型（VLM），结果表明，使用M2CoSC数据集可以显著提升VLM作为心理咨询师的表现。此外，多层次心理治疗推理方法使VLM能提供更体贴且富有同情心的建议，优于标准提示法的效果。
URL
https://arxiv.org/abs/2502.06873
PDF
-
Modality Unified Attack for Omni-Modality Person Re-Identification
2025-01-22 09:54:43Yuan Bian, Min Liu, Yunqi Yi, Xueping Wang, Yunfeng Ma, Yaonan WangAbstract
Deep learning based person re-identification (re-id) models have been widely employed in surveillance systems. Recent studies have demonstrated that black-box single-modality and cross-modality re-id models are vulnerable to adversarial examples (AEs), leaving the robustness of multi-modality re-id models unexplored. Due to the lack of knowledge about the specific type of model deployed in the target black-box surveillance system, we aim to generate modality unified AEs for omni-modality (single-, cross- and multi-modality) re-id models. Specifically, we propose a novel Modality Unified Attack method to train modality-specific adversarial generators to generate AEs that effectively attack different omni-modality models. A multi-modality model is adopted as the surrogate model, wherein the features of each modality are perturbed by metric disruption loss before fusion. To collapse the common features of omni-modality models, Cross Modality Simulated Disruption approach is introduced to mimic the cross-modality feature embeddings by intentionally feeding images to non-corresponding modality-specific subnetworks of the surrogate model. Moreover, Multi Modality Collaborative Disruption strategy is devised to facilitate the attacker to comprehensively corrupt the informative content of person images by leveraging a multi modality feature collaborative metric disruption loss. Extensive experiments show that our MUA method can effectively attack the omni-modality re-id models, achieving 55.9%, 24.4%, 49.0% and 62.7% mean mAP Drop Rate, respectively.
Abstract (translated)
基于深度学习的人再识别（re-id）模型在监控系统中得到了广泛的应用。近期的研究表明，单模态和跨模态的黑盒人再识别模型容易受到对抗样本(AEs)的影响，而多模态人再识别模型的鲁棒性尚不清楚。由于目标黑盒监视系统中所部署的具体类型未知，我们的研究旨在生成适用于全模态（单模、跨模和多模）re-id模型的统一对抗样本。 为此，我们提出了一种名为“Modality Unified Attack”(MUA)的新方法，通过训练特定于每个模态的对抗生成器来生成能够有效攻击不同全模态模型的对抗性示例。在我们的方案中，采用了一个多模态模型作为替代模型，在该模型中，每一种模态的数据特征都经过了度量破坏损失函数（metric disruption loss）的扰动处理，并在融合之前进行。 为了使各种全模态模型共有的特征失效，“Cross Modality Simulated Disruption”方法被引入，通过故意将图像输入到替代模型中的非对应特定模态子网络中来模仿跨模态特征嵌入。此外，“Multi Modality Collaborative Disruption”策略旨在帮助攻击者全面破坏人像图片中有用的信息内容，利用多模态特征协作度量破坏损失函数。 广泛的实验表明，我们的MUA方法能够有效攻击全模态的人再识别模型，在单模、跨模和多模模型上分别实现了55.9%，24.4% 和 62.7%的平均mAP下降率。此外，在包含所有三种类型的综合测试集上的均值mAP下降率为49.0%。 综上所述，我们的研究为提升人再识别模型的安全性和鲁棒性提供了一种全新的视角和方法论。
URL
https://arxiv.org/abs/2501.12761
PDF
-
SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild
2025-01-06 12:09:08Jiawei Liu, Yuanzhi Zhu, Feiyu Gao, Zhibo Yang, Peng Wang, Junyang Lin, Xinggang Wang, Wenyu LiuAbstract
Generating visual text in natural scene images is a challenging task with many unsolved problems. Different from generating text on artificially designed images (such as posters, covers, cartoons, etc.), the text in natural scene images needs to meet the following four key criteria: (1) Fidelity: the generated text should appear as realistic as a photograph and be completely accurate, with no errors in any of the strokes. (2) Reasonability: the text should be generated on reasonable carrier areas (such as boards, signs, walls, etc.), and the generated text content should also be relevant to the scene. (3) Utility: the generated text can facilitate to the training of natural scene OCR (Optical Character Recognition) tasks. (4) Controllability: The attribute of the text (such as font and color) should be controllable as this http URL this paper, we propose a two stage method, SceneVTG++, which simultaneously satisfies the four aspects mentioned above. SceneVTG++ consists of a Text Layout and Content Generator (TLCG) and a Controllable Local Text Diffusion (CLTD). The former utilizes the world knowledge of multi modal large language models to find reasonable text areas and recommend text content according to the nature scene background images, while the latter generates controllable multilingual text based on the diffusion model. Through extensive experiments, we respectively verified the effectiveness of TLCG and CLTD, and demonstrated the state-of-the-art text generation performance of SceneVTG++. In addition, the generated images have superior utility in OCR tasks like text detection and text recognition. Codes and datasets will be available.
Abstract (translated)
在自然场景图像中生成视觉文本是一项充满挑战的任务，许多问题尚未解决。与在人工设计的图像（如海报、封面、卡通等）上生成文字不同，在自然场景图像中的文字需要满足以下四个关键标准：(1) 真实性：生成的文字应该看起来像一张照片一样逼真，并且完全准确，没有任何笔画错误。(2) 合理性：文本应当出现在合理的载体区域（如板子、标识牌、墙壁等），并且所生成的文本内容也应与场景相关。(3) 实用性：生成的文本能够促进自然场景OCR（光学字符识别）任务的训练。(4) 可控性：文字属性（如字体和颜色）应该可以控制。 在这篇论文中，我们提出了一种两阶段方法——SceneVTG++，它同时满足上述四个方面的需求。SceneVTG++由文本布局及内容生成器(TLCG) 和可控局部文本扩散(CLTD)组成。前者利用多模态大型语言模型的世界知识来寻找合理的文字区域并根据自然场景背景图像推荐文字内容，而后者则基于扩散模型生成可控制的多语言文本。通过广泛的实验，我们分别验证了TLCG和CLTD的有效性，并展示了SceneVTG++在文本生成性能方面的先进水平。此外，所生成的图像在OCR任务（如文本检测、文本识别）中具有极高的实用性。代码及数据集将会公开提供。
URL
https://arxiv.org/abs/2501.02962
PDF
-
IRisPath: Enhancing Off-Road Navigation with Robust IR-RGB Fusion for Improved Day and Night Traversability
2024-12-04 09:53:09Saksham Sharma, Akshit Raizada, Suresh SundaramAbstract
Autonomous off-road navigation is required for applications in agriculture, construction, search and rescue and defence. Traditional on-road autonomous methods struggle with dynamic terrains, leading to poor vehicle control on off-road. Recent deep-learning models have used perception sensors along with kinesthetic feedback for navigation on such terrains. However, this approach has out-of-domain uncertainty. Factors like change in weather and time of day impacts the performance of the model. We propose a multi modal fusion network FuseIsPath capable of using LWIR and RGB images to provide robustness against dynamic weather and light conditions. To aid further works in this domain, we also open-source a day-night dataset with LWIR and RGB images along with pseudo-labels for traversability. In order to co-register the two images we developed a novel method for targetless extrinsic calibration of LWIR, LiDAR and RGB cameras with translation accuracy of 1.7cm and rotation accuracy of 0.827degree.
Abstract (translated)
URL
https://arxiv.org/abs/2412.03173
PDF
-
The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare
2024-11-05 17:36:32Souren Pashangpour, Goldie NejatAbstract
The potential use of large language models (LLMs) in healthcare robotics can help address the significant demand put on healthcare systems around the world with respect to an aging demographic and a shortage of healthcare professionals. Even though LLMs have already been integrated into medicine to assist both clinicians and patients, the integration of LLMs within healthcare robots has not yet been explored for clinical settings. In this perspective paper, we investigate the groundbreaking developments in robotics and LLMs to uniquely identify the needed system requirements for designing health specific LLM based robots in terms of multi modal communication through human robot interactions (HRIs), semantic reasoning, and task planning. Furthermore, we discuss the ethical issues, open challenges, and potential future research directions for this emerging innovative field.
Abstract (translated)
大型语言模型（LLMs）在医疗机器人中的潜在应用，可以帮助应对全球范围内因人口老龄化和医护人员短缺而对 healthcare 系统提出的重大需求。尽管 LLMs 已经被整合到医学中以帮助临床医生和患者，但将 LLMs 集成到医疗机器人中的方法尚未在临床环境中得到探索。在这篇视角论文中，我们探讨了机器人技术与 LLMs 的突破性发展，以独特地确定设计特定于健康领域的基于 LLM 机器人的系统需求，特别是在通过人机交互（HRIs）实现多模态通信、语义推理和任务规划方面的需求。此外，我们还讨论了这一新兴创新领域中的伦理问题、开放挑战以及潜在的未来研究方向。 注：原文中的 "healthcare systems" 已经被直接翻译为“医疗系统”，而 "address the significant demand put on healthcare systems" 被解释为应对医疗系统的重大需求。此外，考虑到中文表达习惯，“health specific” 直译可能不够自然，因此调整为“特定于健康领域”。
URL
https://arxiv.org/abs/2411.03287
PDF
-
Multi Modal Information Fusion of Acoustic and Linguistic Data for Decoding Dairy Cow Vocalizations in Animal Welfare Assessment
2024-11-01 09:48:30Bubacarr Jobarteh, Madalina Mincu, Gavojdian Dinu, Suresh NeethirajanAbstract
Understanding animal vocalizations through multi-source data fusion is crucial for assessing emotional states and enhancing animal welfare in precision livestock farming. This study aims to decode dairy cow contact calls by employing multi-modal data fusion techniques, integrating transcription, semantic analysis, contextual and emotional assessment, and acoustic feature extraction. We utilized the Natural Language Processing model to transcribe audio recordings of cow vocalizations into written form. By fusing multiple acoustic features frequency, duration, and intensity with transcribed textual data, we developed a comprehensive representation of cow vocalizations. Utilizing data fusion within a custom-developed ontology, we categorized vocalizations into high frequency calls associated with distress or arousal, and low frequency calls linked to contentment or calmness. Analyzing the fused multi dimensional data, we identified anxiety related features indicative of emotional distress, including specific frequency measurements and sound spectrum results. Assessing the sentiment and acoustic features of vocalizations from 20 individual cows allowed us to determine differences in calling patterns and emotional states. Employing advanced machine learning algorithms, Random Forest, Support Vector Machine, and Recurrent Neural Networks, we effectively processed and fused multi-source data to classify cow vocalizations. These models were optimized to handle computational demands and data quality challenges inherent in practical farm environments. Our findings demonstrate the effectiveness of multi-source data fusion and intelligent processing techniques in animal welfare monitoring. This study represents a significant advancement in animal welfare assessment, highlighting the role of innovative fusion technologies in understanding and improving the emotional wellbeing of dairy cows.
Abstract (translated)
通过多源数据融合理解动物的发声对于在精准畜牧业中评估情绪状态和提高动物福利至关重要。本研究旨在通过运用多模态数据融合技术解码奶牛叫声，整合转录、语义分析、情境与情感评估以及声学特征提取。我们利用自然语言处理模型将奶牛叫声的音频记录转换为书面形式。通过结合多种声学特征（频率、持续时间和强度）与转录的文字数据，我们开发了对奶牛叫声的全面表示。在自定义开发的本体论中使用数据融合，我们将叫声分类为高频呼叫（与压力或唤醒状态相关）和低频呼叫（与满足感或平静情绪相关）。通过分析融合的多维数据，我们识别出表明情感困扰的具体频率测量和声音光谱结果等焦虑相关特征。评估20头单独奶牛叫声的情感和声学特征使我们能够确定叫唤模式和情感状态之间的差异。利用先进的机器学习算法——随机森林、支持向量机和循环神经网络，我们有效地处理并融合了多源数据以分类奶牛叫声。这些模型经过优化，可以应对实际农场环境中固有的计算需求和数据质量挑战。我们的研究结果表明，在动物福利监测中使用多源数据融合与智能处理技术的有效性。本研究表明在动物福利评估方面的一个重大进展，强调创新融合技术在理解和改善奶牛情感福祉中的作用。
URL
https://arxiv.org/abs/2411.00477
PDF
-
M2M-Gen: A Multimodal Framework for Automated Background Music Generation in Japanese Manga Using Large Language Models
2024-10-13 17:15:59Megha Sharma, Muhammad Taimoor Haseeb, Gus Xia, Yoshimasa TsuruokaAbstract
This paper introduces M2M Gen, a multi modal framework for generating background music tailored to Japanese manga. The key challenges in this task are the lack of an available dataset or a baseline. To address these challenges, we propose an automated music generation pipeline that produces background music for an input manga book. Initially, we use the dialogues in a manga to detect scene boundaries and perform emotion classification using the characters faces within a scene. Then, we use GPT4o to translate this low level scene information into a high level music directive. Conditioned on the scene information and the music directive, another instance of GPT 4o generates page level music captions to guide a text to music model. This produces music that is aligned with the mangas evolving narrative. The effectiveness of M2M Gen is confirmed through extensive subjective evaluations, showcasing its capability to generate higher quality, more relevant and consistent music that complements specific scenes when compared to our baselines.
Abstract (translated)
URL
https://arxiv.org/abs/2410.09928
PDF
-
Contrastive Learning-based Multi Modal Architecture for Emoticon Prediction by Employing Image-Text Pairs
2024-08-05 15:45:59Ananya Pandey, Dinesh Kumar VishwakarmaAbstract
The emoticons are symbolic representations that generally accompany the textual content to visually enhance or summarize the true intention of a written message. Although widely utilized in the realm of social media, the core semantics of these emoticons have not been extensively explored based on multiple modalities. Incorporating textual and visual information within a single message develops an advanced way of conveying information. Hence, this research aims to analyze the relationship among sentences, visuals, and emoticons. For an orderly exposition, this paper initially provides a detailed examination of the various techniques for extracting multimodal features, emphasizing the pros and cons of each method. Through conducting a comprehensive examination of several multimodal algorithms, with specific emphasis on the fusion approaches, we have proposed a novel contrastive learning based multimodal architecture. The proposed model employs the joint training of dual-branch encoder along with the contrastive learning to accurately map text and images into a common latent space. Our key finding is that by integrating the principle of contrastive learning with that of the other two branches yields superior results. The experimental results demonstrate that our suggested methodology surpasses existing multimodal approaches in terms of accuracy and robustness. The proposed model attained an accuracy of 91% and an MCC-score of 90% while assessing emoticons using the Multimodal-Twitter Emoticon dataset acquired from Twitter. We provide evidence that deep features acquired by contrastive learning are more efficient, suggesting that the proposed fusion technique also possesses strong generalisation capabilities for recognising emoticons across several modes.
Abstract (translated)
表情符号是一种象征性的表示形式，通常与文本内容一起使用，以视觉上增强或总结书面信息的确切意图。尽管在社交媒体领域得到了广泛应用，但基于多种模式对这些表情符号的核心语义进行了深入探讨还是不足为继。将文本和视觉信息集成到一个消息中，发展了一种高级传达信息的方法。因此，这项研究旨在分析句子、视觉信息和表情符号之间的关系。为进行有序的阐述，本文首先对各种提取多模态特征的技术进行了详细调查，强调每种方法的优缺点。通过全面评估多个多模态算法，特别是融合方法，我们提出了一个新颖的基于多模态学习的架构。所提出的模型采用双分支编码器与对比学习相结合来准确地将文本和图像映射到共同的潜在空间。我们的关键发现是，将对比学习原理与其他两个分支相结合产生了更好的结果。实验结果表明，我们所提出的方法在准确性和鲁棒性方面超过了现有的多模态方法。在使用Twitter Multimodal Emoticon数据集评估表情符号时，所提出的模型获得了91%的准确性和90%的MCC分数。我们提供了证据，表明通过对比学习获得的深度特征更加有效，表明所提出的融合技术也对识别多种模式下的表情符号具有很强的泛化能力。
URL
https://arxiv.org/abs/2408.02571
PDF
-
Diag2Diag: Multi modal super resolution for physics discovery with application to fusion
2024-05-09 17:06:51Azarakhsh Jalalvand, Max Curie, SangKyeun Kim, Peter Steiner, Jaemin Seo, Qiming Hu, Andrew Oakleigh Nelson, Egemen KolemenAbstract
This paper introduces a groundbreaking multi-modal neural network model designed for resolution enhancement, which innovatively leverages inter-diagnostic correlations within a system. Traditional approaches have primarily focused on uni-modal enhancement strategies, such as pixel-based image enhancement or heuristic signal interpolation. In contrast, our model employs a novel methodology by harnessing the diagnostic relationships within the physics of fusion plasma. Initially, we establish the correlation among diagnostics within the tokamak. Subsequently, we utilize these correlations to substantially enhance the temporal resolution of the Thomson Scattering diagnostic, which assesses plasma density and temperature. By increasing its resolution from conventional 200Hz to 500kHz, we facilitate a new level of insight into plasma behavior, previously attainable only through computationally intensive simulations. This enhancement goes beyond simple interpolation, offering novel perspectives on the underlying physical phenomena governing plasma dynamics.
Abstract (translated)
本文提出了一种在分辨率增强方面具有突破性的多模态神经网络模型,该模型创新地利用了系统内诊断关系。传统方法主要集中在单模态增强策略,例如基于像素的图像增强或启发式信号插值。相比之下,我们的模型通过利用融合 plasma 物理学中诊断关系的方法来创新性地实现了一种新的方法。首先,我们在 tokamak 中建立了诊断之间的关系。接着,我们利用这些关系大大增强了汤姆逊散射诊断的时域分辨率,该诊断评估了 plasma 密度和温度。通过将分辨率从传统的 200Hz 提高到 500kHz,我们促进了对 plasma 行为的深入洞察,这一般仅通过计算密集型模拟才能实现。这种增强超越了简单的插值,提供了一种新颖的视角,揭示了控制 plasma 动力学背后的物理现象。
URL
https://arxiv.org/abs/2405.05908
PDF
-
Solution for Emotion Prediction Competition of Workshop on Emotionally and Culturally Intelligent AI
2024-03-26 13:14:18Shengdong Xu, Zhouyang Chi, Yang YangAbstract
This report provide a detailed description of the method that we explored and proposed in the WECIA Emotion Prediction Competition (EPC), which predicts a person's emotion through an artistic work with a comment. The dataset of this competition is ArtELingo, designed to encourage work on diversity across languages and cultures. The dataset has two main challenges, namely modal imbalance problem and language-cultural differences problem. In order to address this issue, we propose a simple yet effective approach called single-multi modal with Emotion-Cultural specific prompt(ECSP), which focuses on using the single modal message to enhance the performance of multimodal models and a well-designed prompt to reduce cultural differences problem. To clarify, our approach contains two main blocks: (1)XLM-R\cite{conneau2019unsupervised} based unimodal model and X$^2$-VLM\cite{zeng2022x} based multimodal model (2) Emotion-Cultural specific prompt. Our approach ranked first in the final test with a score of 0.627.
Abstract (translated)
本报告详细描述了我们参加WECIA情感预测竞赛（EPC）时所探索和提出的方法，该竞赛通过一件艺术作品来预测一个人的情感。比赛的數據集是ArtELingo，旨在鼓励跨語言和文化的作品。比赛數據集有两个主要挑戰，即模态不平衡問題和語言-文化差異問題。为了应对这个问题，我们提出了一个简单而有效的方案，称为情感文化特定提示（ECSP）单一模态与多模态模型。该方案重点使用单一模态信息来提高多模态模型的性能，并设计了一个精心设计的提示来减少文化差异问题。为了明确，我们的方法包含两个主要部分：（1）基于unimodal的XLM-R模型和基于multimodal的X$^2$-VLM模型（2）情感-文化特定提示。我们的方法在决赛测试中排名第一，得分为0.627。
URL
https://arxiv.org/abs/2403.17683
PDF
-
Data Augmentation using LLMs: Data Perspectives, Learning Paradigms and Challenges
2024-03-05 14:11:54Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, Shafiq JotyAbstract
In the rapidly evolving field of machine learning (ML), data augmentation (DA) has emerged as a pivotal technique for enhancing model performance by diversifying training examples without the need for additional data collection. This survey explores the transformative impact of Large Language Models (LLMs) on DA, particularly addressing the unique challenges and opportunities they present in the context of natural language processing (NLP) and beyond. From a data perspective and a learning perspective, we examine various strategies that utilize Large Language Models for data augmentation, including a novel exploration of learning paradigms where LLM-generated data is used for further training. Additionally, this paper delineates the primary challenges faced in this domain, ranging from controllable data augmentation to multi modal data augmentation. This survey highlights the paradigm shift introduced by LLMs in DA, aims to serve as a foundational guide for researchers and practitioners in this field.
Abstract (translated)
在迅速发展的机器学习（ML）领域，数据增强（DA）已成为通过扩展训练示例来提高模型性能的关键技术，而无需进行额外的数据收集。本调查探讨了大型语言模型（LLMs）对DA的变革性影响，特别是在自然语言处理（NLP）及其它领域的独特挑战和机遇。从数据视角和学习视角出发，我们检查了各种利用LLM进行数据增强的策略，包括一种新的探索学习范式，其中LLM生成的数据用于进一步训练。此外，本文概述了该领域面临的主要挑战，从可控制的数据增强到多模态数据增强。本调查突出了LLM在DA领域引入的范式转变，旨在为该领域的研究人员和实践者提供基础指导。
URL
https://arxiv.org/abs/2403.02990
PDF
-
A Multimodal Approach for Advanced Pest Detection and Classification
2023-12-18 05:54:20Jinli Duan, Haoyu Ding, Sung KimAbstract
This paper presents a novel multi modal deep learning framework for enhanced agricultural pest detection, combining tiny-BERT's natural language processing with R-CNN and ResNet-18's image processing. Addressing limitations of traditional CNN-based visual methods, this approach integrates textual context for more accurate pest identification. The R-CNN and ResNet-18 integration tackles deep CNN issues like vanishing gradients, while tiny-BERT ensures computational efficiency. Employing ensemble learning with linear regression and random forest models, the framework demonstrates superior discriminate ability, as shown in ROC and AUC analyses. This multi modal approach, blending text and image data, significantly boosts pest detection in agriculture. The study highlights the potential of multi modal deep learning in complex real-world scenarios, suggesting future expansions in diversity of datasets, advanced data augmentation, and cross-modal attention mechanisms to enhance model performance.
Abstract (translated)
本文提出了一种新颖的多模态深度学习框架,用于增强农业害虫检测,将 tiny-BERT 的自然语言处理与 R-CNN 和 ResNet-18 的图像处理相结合。该方法解决了传统 CNN 视觉方法的局限性,并引入了文本上下文以实现更精确的害虫识别。R-CNN 和 ResNet-18 的集成解决了深度 CNN 问题,如消失的梯度,而 tiny-BERT 保证了计算效率。通过线性回归和随机森林模型的集成,该框架展示了卓越的判别能力,如图论和 AUC 分析结果所示。这种多模态方法结合了文本和图像数据,显著提高了农业中的害虫检测。本研究突出了多模态深度学习在复杂现实场景中的潜力,建议在未来增加数据集的多样性、高级数据增强和跨模态关注机制,以提高模型性能。
URL
https://arxiv.org/abs/2312.10948
PDF
-
Versatile Audio-Visual Learning for Handling Single and Multi Modalities in Emotion Regression and Classification Tasks
2023-05-12 03:13:37Lucas Goncalves, Seong-Gyun Leem, Wei-Cheng Lin, Berrak Sisman, Carlos BussoAbstract
Most current audio-visual emotion recognition models lack the flexibility needed for deployment in practical applications. We envision a multimodal system that works even when only one modality is available and can be implemented interchangeably for either predicting emotional attributes or recognizing categorical emotions. Achieving such flexibility in a multimodal emotion recognition system is difficult due to the inherent challenges in accurately interpreting and integrating varied data sources. It is also a challenge to robustly handle missing or partial information while allowing direct switch between regression and classification tasks. This study proposes a \emph{versatile audio-visual learning} (VAVL) framework for handling unimodal and multimodal systems for emotion regression and emotion classification tasks. We implement an audio-visual framework that can be trained even when audio and visual paired data is not available for part of the training set (i.e., audio only or only video is present). We achieve this effective representation learning with audio-visual shared layers, residual connections over shared layers, and a unimodal reconstruction task. Our experimental results reveal that our architecture significantly outperforms strong baselines on both the CREMA-D and MSP-IMPROV corpora. Notably, VAVL attains a new state-of-the-art performance in the emotional attribute prediction task on the MSP-IMPROV corpus. Code available at: this https URL
Abstract (translated)
当前大多数音频和视觉情感识别模型缺乏实际应用所需的灵活性。我们构想了一种多模态系统，即使在只有一个模态可用的情况下也能工作，并且可以相互替代地用于预测情感属性或识别分类情感。在实现多模态情感识别系统的灵活性方面，由于准确解释和整合多种数据源的内在挑战，非常困难。此外，在处理缺失或部分信息的同时，允许直接进行回归和分类任务的挑战也非常困难。本文提出了一个 \emph{多功能音频-视觉学习} (VAVL)框架，用于处理情感回归和情感分类任务中的单模态和多模态系统。我们实现了一个音频和视觉共享层的框架，在共享层上保留连接，并实现了单模态重建任务。我们的实验结果显示，我们的架构在CREMA-D和MSP-IMPROV corpora上的 strong baselines 显著超越了它们。值得注意的是，在MSP-IMPROV corpora上的情感属性预测任务中，VAVL取得了新的先进技术表现。代码可在 this https URL 上获取。
URL
https://arxiv.org/abs/2305.07216
PDF
-
Medical Image Analysis using Deep Relational Learning
2023-03-28 16:10:12Zhihua LiuAbstract
In the past ten years, with the help of deep learning, especially the rapid development of deep neural networks, medical image analysis has made remarkable progress. However, how to effectively use the relational information between various tissues or organs in medical images is still a very challenging problem, and it has not been fully studied. In this thesis, we propose two novel solutions to this problem based on deep relational learning. First, we propose a context-aware fully convolutional network that effectively models implicit relation information between features to perform medical image segmentation. The network achieves the state-of-the-art segmentation results on the Multi Modal Brain Tumor Segmentation 2017 (BraTS2017) and Multi Modal Brain Tumor Segmentation 2018 (BraTS2018) data sets. Subsequently, we propose a new hierarchical homography estimation network to achieve accurate medical image mosaicing by learning the explicit spatial relationship between adjacent frames. We use the UCL Fetoscopy Placenta dataset to conduct experiments and our hierarchical homography estimation network outperforms the other state-of-the-art mosaicing methods while generating robust and meaningful mosaicing result on unseen frames.
Abstract (translated)
在过去的十年中，借助深度学习，特别是深度神经网络的迅速发展，医学图像分析取得了显著进展。然而，如何有效地利用医学图像中各种组织和器官之间的隐含关系仍然是一个极具挑战性的问题，并尚未得到充分研究。在本文中，我们提出了基于深度学习关系深度学习的两个创新解决方案。首先，我们提出了一种具有上下文意识的全卷积神经网络，有效地模型了特征之间的隐含关系信息，以进行医学图像分割。该网络在Multimodal Brain Tumor Segmentation 2017( BraTS2017)和Multimodal Brain Tumor Segmentation 2018( BraTS2018)数据集上取得了最先进的分割结果。随后，我们提出了一种新的层级基元估计网络，以通过学习相邻帧之间的明确空间关系实现准确的医学图像拼贴，我们使用UCL鲸鱼超声波 dataset进行了实验，我们的层级基元估计网络在未观测帧上的拼贴结果表现优异，同时生成稳健且有意义的拼贴结果。
URL
https://arxiv.org/abs/2303.16099
PDF
-
Human Fall Detection- Multimodality Approach
2023-02-01 04:05:14Xi Wang, Ramya Penta, Bhavya Sehgal, Dale Chen-SongAbstract
Falls have become more frequent in recent years, which has been harmful for senior citizens.Therefore detecting falls have become important and several data sets and machine learning model have been introduced related to fall detection. In this project report, a human fall detection method is proposed using a multi modality approach. We used the UP-FALL detection data set which is collected by dozens of volunteers using different sensors and two cameras. We use wrist sensor with acclerometer data keeping labels to binary classification, namely fall and no fall from the data set.We used fusion of camera and sensor data to increase performance. The experimental results shows that using only wrist data as compared to multi sensor for binary classification did not impact the model prediction performance for fall detection.
Abstract (translated)
Falls近年来变得越来越普遍,这对老年人来说是一种危害。因此,检测跌倒变得非常重要,并引入了多个数据集和机器学习模型与跌倒检测相关。在本报告中,提出了一种使用多模态方法的人跌倒检测方法。我们使用了由数十个志愿者使用多种传感器和两只摄像机收集的UP-Fall检测数据集。我们使用带计步器的手腕传感器将计步器数据作为标签进行二元分类,即跌倒和未跌倒从数据集中选取。我们使用了相机和传感器数据的集成来提高性能。实验结果显示,与使用多个传感器进行二元分类相比,仅使用手腕数据对于跌倒检测模型预测性能没有影响。
URL
https://arxiv.org/abs/2302.00224
PDF
-
AlignVE: Visual Entailment Recognition Based on Alignment Relations
2022-11-16 07:52:24Biwei Cao, Jiuxin Cao, Jie Gui, Jiayun Shen, Bo Liu, Lei He, Yuan Yan Tang, James Tin-Yau KwokAbstract
Visual entailment (VE) is to recognize whether the semantics of a hypothesis text can be inferred from the given premise image, which is one special task among recent emerged vision and language understanding tasks. Currently, most of the existing VE approaches are derived from the methods of visual question answering. They recognize visual entailment by quantifying the similarity between the hypothesis and premise in the content semantic features from multi modalities. Such approaches, however, ignore the VE's unique nature of relation inference between the premise and hypothesis. Therefore, in this paper, a new architecture called AlignVE is proposed to solve the visual entailment problem with a relation interaction method. It models the relation between the premise and hypothesis as an alignment matrix. Then it introduces a pooling operation to get feature vectors with a fixed size. Finally, it goes through the fully-connected layer and normalization layer to complete the classification. Experiments show that our alignment-based architecture reaches 72.45\% accuracy on SNLI-VE dataset, outperforming previous content-based models under the same settings.
Abstract (translated)
URL
https://arxiv.org/abs/2211.08736
PDF
-
A Rich Recipe Representation as Plan to Support Expressive Multi Modal Queries on Recipe Content and Preparation Process
2022-03-31 15:29:38Vishal Pallagani, Priyadharsini Ramamurthy, Vedant Khandelwal, Revathy Venkataramanan, Kausik Lakkaraju, Sathyanarayanan N. Aakur, Biplav SrivastavaAbstract
Food is not only a basic human necessity but also a key factor driving a society's health and economic well-being. As a result, the cooking domain is a popular use-case to demonstrate decision-support (AI) capabilities in service of benefits like precision health with tools ranging from information retrieval interfaces to task-oriented chatbots. An AI here should understand concepts in the food domain (e.g., recipes, ingredients), be tolerant to failures encountered while cooking (e.g., browning of butter), handle allergy-based substitutions, and work with multiple data modalities (e.g. text and images). However, the recipes today are handled as textual documents which makes it difficult for machines to read, reason and handle ambiguity. This demands a need for better representation of the recipes, overcoming the ambiguity and sparseness that exists in the current textual documents. In this paper, we discuss the construction of a machine-understandable rich recipe representation (R3), in the form of plans, from the recipes available in natural language. R3 is infused with additional knowledge such as information about allergens and images of ingredients, possible failures and tips for each atomic cooking step. To show the benefits of R3, we also present TREAT, a tool for recipe retrieval which uses R3 to perform multi-modal reasoning on the recipe's content (plan objects - ingredients and cooking tools), food preparation process (plan actions and time), and media type (image, text). R3 leads to improved retrieval efficiency and new capabilities that were hither-to not possible in textual representation.
Abstract (translated)
URL
https://arxiv.org/abs/2203.17109
PDF
