Emotion
-
TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM
2024-12-04 07:50:17Huiying Cao, Yiqun Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yifei ZhangAbstract
Empathetic conversation is a crucial characteristic in daily conversations between individuals. Nowadays, Large Language models (LLMs) have shown outstanding performance in generating empathetic responses. Knowledge bases like COMET can assist LLMs in mitigating illusions and enhancing the understanding of users' intentions and emotions. However, models remain heavily reliant on fixed knowledge bases and unrestricted incorporation of external knowledge can introduce noise. Tool learning is a flexible end-to-end approach that assists LLMs in handling complex problems. In this paper, we propose Emotional Knowledge Tool Calling (EKTC) framework, which encapsulates the commonsense knowledge bases as empathetic tools, enabling LLMs to integrate external knowledge flexibly through tool calling. In order to adapt the models to the new task, we construct a novel dataset TOOL-ED based on the EMPATHETICMPATHETIC DIALOGUE (ED) dataset. We validate EKTC on the ED dataset, and the experimental results demonstrate that our framework can enhance the ability of LLMs to generate empathetic responses effectively.
Abstract (translated)
URL
https://arxiv.org/abs/2412.03096
PDF
-
Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models
Abstract
The rapid advancements in large language models (LLMs) have significantly improved their ability to generate natural language, making texts generated by LLMs increasingly indistinguishable from human-written texts. Recent research has predominantly focused on using LLMs to classify text as either human-written or machine-generated. In our study, we adopt a different approach by profiling texts spanning four domains based on 250 distinct linguistic features. We select the M4 dataset from the Subtask B of SemEval 2024 Task 8. We automatically calculate various linguistic features with the LFTK tool and additionally measure the average syntactic depth, semantic similarity, and emotional content for each document. We then apply a two-dimensional PCA reduction to all the calculated features. Our analyses reveal significant differences between human-written texts and those generated by LLMs, particularly in the variability of these features, which we find to be considerably higher in human-written texts. This discrepancy is especially evident in text genres with less rigid linguistic style constraints. Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs. These insights underscore the need for incorporating meaningful linguistic features to enhance the understanding of textual outputs of LLMs.
Abstract (translated)
URL
https://arxiv.org/abs/2412.03025
PDF
-
Dynamic Graph Neural Ordinary Differential Equation Network for Multi-modal Emotion Recognition in Conversation
2024-12-04 01:07:59Yuntao Shou, Tao Meng, Wei Ai, Keqin LiAbstract
Multimodal emotion recognition in conversation (MERC) refers to identifying and classifying human emotional states by combining data from multiple different modalities (e.g., audio, images, text, video, etc.). Most existing multimodal emotion recognition methods use GCN to improve performance, but existing GCN methods are prone to overfitting and cannot capture the temporal dependency of the speaker's emotions. To address the above problems, we propose a Dynamic Graph Neural Ordinary Differential Equation Network (DGODE) for MERC, which combines the dynamic changes of emotions to capture the temporal dependency of speakers' emotions, and effectively alleviates the overfitting problem of GCNs. Technically, the key idea of DGODE is to utilize an adaptive mixhop mechanism to improve the generalization ability of GCNs and use the graph ODE evolution network to characterize the continuous dynamics of node representations over time and capture temporal dependencies. Extensive experiments on two publicly available multimodal emotion recognition datasets demonstrate that the proposed DGODE model has superior performance compared to various baselines. Furthermore, the proposed DGODE can also alleviate the over-smoothing problem, thereby enabling the construction of a deep GCN network.
Abstract (translated)
URL
https://arxiv.org/abs/2412.02935
PDF
-
MLD-EA: Check and Complete Narrative Coherence by Introducing Emotions and Actions
2024-12-03 23:01:21Jinming Zhang, Yunfei LongAbstract
Narrative understanding and story generation are critical challenges in natural language processing (NLP), with much of the existing research focused on summarization and question-answering tasks. While previous studies have explored predicting plot endings and generating extended narratives, they often neglect the logical coherence within stories, leaving a significant gap in the field. To address this, we introduce the Missing Logic Detector by Emotion and Action (MLD-EA) model, which leverages large language models (LLMs) to identify narrative gaps and generate coherent sentences that integrate seamlessly with the story's emotional and logical flow. The experimental results demonstrate that the MLD-EA model enhances narrative understanding and story generation, highlighting LLMs' potential as effective logic checkers in story writing with logical coherence and emotional consistency. This work fills a gap in NLP research and advances border goals of creating more sophisticated and reliable story-generation systems.
Abstract (translated)
URL
https://arxiv.org/abs/2412.02897
PDF
-
GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot
2024-12-03 17:41:24Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, Jie TangAbstract
We introduce GLM-4-Voice, an intelligent and human-like end-to-end spoken chatbot. It supports both Chinese and English, engages in real-time voice conversations, and varies vocal nuances such as emotion, intonation, speech rate, and dialect according to user instructions. GLM-4-Voice uses an ultra-low bitrate (175bps), single-codebook speech tokenizer with 12.5Hz frame rate derived from an automatic speech recognition (ASR) model by incorporating a vector-quantized bottleneck into the encoder. To efficiently transfer knowledge from text to speech modalities, we synthesize speech-text interleaved data from existing text pre-training corpora using a text-to-token model. We continue pre-training from the pre-trained text language model GLM-4-9B with a combination of unsupervised speech data, interleaved speech-text data, and supervised speech-text data, scaling up to 1 trillion tokens, achieving state-of-the-art performance in both speech language modeling and spoken question answering. We then fine-tune the pre-trained model with high-quality conversational speech data, achieving superior performance compared to existing baselines in both conversational ability and speech quality. The open models can be accessed through this https URL and this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2412.02612
PDF
-
Towards Rich Emotions in 3D Avatars: A Text-to-3D Avatar Generation Benchmark
2024-12-03 15:39:05Haidong Xu, Meishan Zhang, Hao Ju, Zhedong Zheng, Hongyuan Zhu, Erik Cambria, Min Zhang, Hao FeiAbstract
Producing emotionally dynamic 3D facial avatars with text derived from spoken words (Emo3D) has been a pivotal research topic in 3D avatar generation. While progress has been made in general-purpose 3D avatar generation, the exploration of generating emotional 3D avatars remains scarce, primarily due to the complexities of identifying and rendering rich emotions from spoken words. This paper reexamines Emo3D generation and draws inspiration from human processes, breaking down Emo3D into two cascading steps: Text-to-3D Expression Mapping (T3DEM) and 3D Avatar Rendering (3DAR). T3DEM is the most crucial step in determining the quality of Emo3D generation and encompasses three key challenges: Expression Diversity, Emotion-Content Consistency, and Expression Fluidity. To address these challenges, we introduce a novel benchmark to advance research in Emo3D generation. First, we present EmoAva, a large-scale, high-quality dataset for T3DEM, comprising 15,000 text-to-3D expression mappings that characterize the aforementioned three challenges in Emo3D generation. Furthermore, we develop various metrics to effectively evaluate models against these identified challenges. Next, to effectively model the consistency, diversity, and fluidity of human expressions in the T3DEM step, we propose the Continuous Text-to-Expression Generator, which employs an autoregressive Conditional Variational Autoencoder for expression code generation, enhanced with Latent Temporal Attention and Expression-wise Attention mechanisms. Finally, to further enhance the 3DAR step on rendering higher-quality subtle expressions, we present the Globally-informed Gaussian Avatar (GiGA) model. GiGA incorporates a global information mechanism into 3D Gaussian representations, enabling the capture of subtle micro-expressions and seamless transitions between emotional states.
Abstract (translated)
URL
https://arxiv.org/abs/2412.02508
PDF
-
VR Based Emotion Recognition Using Deep Multimodal Fusion With Biosignals Across Multiple Anatomical Domains
2024-12-03 08:59:12Pubudu L. Indrasiri, Bipasha Kashyap, Chandima Kolambahewage, Bahareh Nakisa, Kiran Ijaz, Pubudu N. PathiranaAbstract
Emotion recognition is significantly enhanced by integrating multimodal biosignals and IMU data from multiple domains. In this paper, we introduce a novel multi-scale attention-based LSTM architecture, combined with Squeeze-and-Excitation (SE) blocks, by leveraging multi-domain signals from the head (Meta Quest Pro VR headset), trunk (Equivital Vest), and peripheral (Empatica Embrace Plus) during affect elicitation via visual stimuli. Signals from 23 participants were recorded, alongside self-assessed valence and arousal ratings after each stimulus. LSTM layers extract features from each modality, while multi-scale attention captures fine-grained temporal dependencies, and SE blocks recalibrate feature importance prior to classification. We assess which domain's signals carry the most distinctive emotional information during VR experiences, identifying key biosignals contributing to emotion detection. The proposed architecture, validated in a user study, demonstrates superior performance in classifying valance and arousal level (high / low), showcasing the efficacy of multi-domain and multi-modal fusion with biosignals (e.g., TEMP, EDA) with IMU data (e.g., accelerometer) for emotion recognition in real-world applications.
Abstract (translated)
URL
https://arxiv.org/abs/2412.02283
PDF
-
FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait
2024-12-02 02:50:07Taekyung Ki, Dongchan Min, Gyoungsu ChaeAbstract
With the rapid advancement of diffusion-based generative models, portrait image animation has achieved remarkable results. However, it still faces challenges in temporally consistent video generation and fast sampling due to its iterative sampling nature. This paper presents FLOAT, an audio-driven talking portrait video generation method based on flow matching generative model. We shift the generative modeling from the pixel-based latent space to a learned motion latent space, enabling efficient design of temporally consistent motion. To achieve this, we introduce a transformer-based vector field predictor with a simple yet effective frame-wise conditioning mechanism. Additionally, our method supports speech-driven emotion enhancement, enabling a natural incorporation of expressive motions. Extensive experiments demonstrate that our method outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency.
Abstract (translated)
URL
https://arxiv.org/abs/2412.01064
PDF
-
Does chat change LLM's mind? Impact of Conversation on Psychological States of LLMs
2024-12-01 13:19:32Junhyuk Choi, Yeseon Hong, Minju Kim, Bugeun KimAbstract
The recent growth of large language models (LLMs) has enabled more authentic, human-centered interactions through multi-agent systems. However, investigation into how conversations affect the psychological states of LLMs is limited, despite the impact of these states on the usability of LLM-based systems. In this study, we explored whether psychological states change during multi-agent interactions, focusing on the effects of conversation depth, topic, and speaker. We experimentally investigated the behavior of 10 LLMs in open-domain conversations. We employed 14 questionnaires and a topic-analysis method to examine the behavior of LLMs across four aspects: personality, interpersonal relationships, motivation, and emotion. The results revealed distinct psychological trends influenced by conversation depth and topic, with significant variations observed between different LLM families and parameter sizes.
Abstract (translated)
URL
https://arxiv.org/abs/2412.00804
PDF
-
Pairwise Discernment of AffectNet Expressions with ArcFace
2024-12-01 10:18:55Dylan Waldner, Shyamal MitraAbstract
This study takes a preliminary step toward teaching computers to recognize human emotions through Facial Emotion Recognition (FER). Transfer learning is applied using ResNeXt, EfficientNet models, and an ArcFace model originally trained on the facial verification task, leveraging the AffectNet database, a collection of human face images annotated with corresponding emotions. The findings highlight the value of congruent domain transfer learning, the challenges posed by imbalanced datasets in learning facial emotion patterns, and the effectiveness of pairwise learning in addressing class imbalances to enhance model performance on the FER task.
Abstract (translated)
URL
https://arxiv.org/abs/2412.01860
PDF
-
Forma mentis networks predict creativity ratings of short texts via interpretable artificial intelligence in human and GPT-simulated raters
2024-11-30 16:33:48Edith Haim, Natalie Fischer, Salvatore Citraro, Giulio Rossetti, Massimo StellaAbstract
Creativity is a fundamental skill of human cognition. We use textual forma mentis networks (TFMN) to extract network (semantic/syntactic associations) and emotional features from approximately one thousand human- and GPT3.5-generated stories. Using Explainable Artificial Intelligence (XAI), we test whether features relative to Mednick's associative theory of creativity can explain creativity ratings assigned by humans and GPT-3.5. Using XGBoost, we examine three scenarios: (i) human ratings of human stories, (ii) GPT-3.5 ratings of human stories, and (iii) GPT-3.5 ratings of GPT-generated stories. Our findings reveal that GPT-3.5 ratings differ significantly from human ratings not only in terms of correlations but also because of feature patterns identified with XAI methods. GPT-3.5 favours 'its own' stories and rates human stories differently from humans. Feature importance analysis with SHAP scores shows that: (i) network features are more predictive for human creativity ratings but also for GPT-3.5's ratings of human stories; (ii) emotional features played a greater role than semantic/syntactic network structure in GPT-3.5 rating its own stories. These quantitative results underscore key limitations in GPT-3.5's ability to align with human assessments of creativity. We emphasise the need for caution when using GPT-3.5 to assess and generate creative content, as it does not yet capture the nuanced complexity that characterises human creativity.
Abstract (translated)
URL
https://arxiv.org/abs/2412.00530
PDF
-
Improving speaker verification robustness with synthetic emotional utterances
2024-11-30 02:18:26Nikhil Kumar Koditala, Chelsea Jui-Ting Ju, Ruirui Li, Minho Jin, Aman Chadha, Andreas StolckeAbstract
A speaker verification (SV) system offers an authentication service designed to confirm whether a given speech sample originates from a specific speaker. This technology has paved the way for various personalized applications that cater to individual preferences. A noteworthy challenge faced by SV systems is their ability to perform consistently across a range of emotional spectra. Most existing models exhibit high error rates when dealing with emotional utterances compared to neutral ones. Consequently, this phenomenon often leads to missing out on speech of interest. This issue primarily stems from the limited availability of labeled emotional speech data, impeding the development of robust speaker representations that encompass diverse emotional states. To address this concern, we propose a novel approach employing the CycleGAN framework to serve as a data augmentation method. This technique synthesizes emotional speech segments for each specific speaker while preserving the unique vocal identity. Our experimental findings underscore the effectiveness of incorporating synthetic emotional data into the training process. The models trained using this augmented dataset consistently outperform the baseline models on the task of verifying speakers in emotional speech scenarios, reducing equal error rate by as much as 3.64% relative.
Abstract (translated)
URL
https://arxiv.org/abs/2412.00319
PDF
-
SDR-GNN: Spectral Domain Reconstruction Graph Neural Network for Incomplete Multimodal Learning in Conversational Emotion Recognition
2024-11-29 16:31:50Fangze Fu, Wei Ai, Fan Yang, Yuntao Shou, Tao Meng, Keqin LiAbstract
Multimodal Emotion Recognition in Conversations (MERC) aims to classify utterance emotions using textual, auditory, and visual modal features. Most existing MERC methods assume each utterance has complete modalities, overlooking the common issue of incomplete modalities in real-world scenarios. Recently, graph neural networks (GNNs) have achieved notable results in Incomplete Multimodal Emotion Recognition in Conversations (IMERC). However, traditional GNNs focus on binary relationships between nodes, limiting their ability to capture more complex, higher-order information. Moreover, repeated message passing can cause over-smoothing, reducing their capacity to preserve essential high-frequency details. To address these issues, we propose a Spectral Domain Reconstruction Graph Neural Network (SDR-GNN) for incomplete multimodal learning in conversational emotion recognition. SDR-GNN constructs an utterance semantic interaction graph using a sliding window based on both speaker and context relationships to model emotional dependencies. To capture higher-order and high-frequency information, SDR-GNN utilizes weighted relationship aggregation, ensuring consistent semantic feature extraction across utterances. Additionally, it performs multi-frequency aggregation in the spectral domain, enabling efficient recovery of incomplete modalities by extracting both high- and low-frequency information. Finally, multi-head attention is applied to fuse and optimize features for emotion recognition. Extensive experiments on various real-world datasets demonstrate that our approach is effective in incomplete multimodal learning and outperforms current state-of-the-art methods.
Abstract (translated)
URL
https://arxiv.org/abs/2411.19822
PDF
-
GPT as ghostwriter at the White House
2024-11-27 14:12:36Jacques SavoyAbstract
Recently several large language models (LLMs) have demonstrated their capability to generate a message in response to a user request. Such scientific breakthroughs promote new perspectives but also some fears. The main focus of this study is to analyze the written style of one LLM called ChatGPT 3.5 by comparing its generated messages with those of the recent US presidents. To achieve this objective, we compare the State of the Union addresses written by Reagan to Obama with those automatically produced by ChatGPT. We found that ChatGPT tends to overuse the lemma "we" as well as nouns and commas. On the other hand, the generated speeches employ less verbs and include, in mean, longer sentences. Even when imposing a given style to ChatGPT, the resulting speech remains distinct from messages written by the target author. Moreover, ChatGPT opts for a neutral tone with mainly positive emotional expressions and symbolic terms (e.g., freedom, nation). Finally, we show that the GPT's style exposes distinct features compared to real presidential addresses.
Abstract (translated)
最近，几个大型语言模型（LLMs）展示了它们根据用户请求生成消息的能力。这些科学突破促进了新视角的形成，但也引发了一些担忧。本研究的主要目的是通过比较一个名为ChatGPT 3.5的LLM生成的消息与近期美国总统发布的消息来分析其写作风格。为此，我们将里根到奥巴马的国情咨文演讲与其由ChatGPT自动生成的版本进行了对比。我们发现，ChatGPT倾向于过度使用词根“we”以及名词和逗号。另一方面，它生成的演讲中动词较少，并且平均而言句子更长。即使对ChatGPT设定特定风格，其产生的演讲仍然与目标作者撰写的文本有区别。此外，ChatGPT选择了一种主要包含积极情感表达和象征性词汇（如自由、国家）的中立语气。最后，我们展示了GPT的写作风格与真实的总统演讲相比存在显著差异。
URL
https://arxiv.org/abs/2411.18365
PDF
-
Wearable intelligent throat enables natural speech in stroke patients with dysarthria
2024-11-27 12:03:52Chenyu Tang, Shuo Gao, Cong Li, Wentian Yi, Yuxuan Jin, Xiaoxue Zhai, Sixuan Lei, Hongbei Meng, Zibo Zhang, Muzi Xu, Shengbo Wang, Xuhang Chen, Chenxi Wang, Hongyun Yang, Ningli Wang, Wenyu Wang, Jin Cao, Xiaodong Feng, Peter Smielewski, Yu Pan, Wenhui Song, Martin Birchall, Luigi G. OcchipintAbstract
Wearable silent speech systems hold significant potential for restoring communication in patients with speech impairments. However, seamless, coherent speech remains elusive, and clinical efficacy is still unproven. Here, we present an AI-driven intelligent throat (IT) system that integrates throat muscle vibrations and carotid pulse signal sensors with large language model (LLM) processing to enable fluent, emotionally expressive communication. The system utilizes ultrasensitive textile strain sensors to capture high-quality signals from the neck area and supports token-level processing for real-time, continuous speech decoding, enabling seamless, delay-free communication. In tests with five stroke patients with dysarthria, IT's LLM agents intelligently corrected token errors and enriched sentence-level emotional and logical coherence, achieving low error rates (4.2% word error rate, 2.9% sentence error rate) and a 55% increase in user satisfaction. This work establishes a portable, intuitive communication platform for patients with dysarthria with the potential to be applied broadly across different neurological conditions and in multi-language support systems.
Abstract (translated)
可穿戴的无声语音系统在恢复有言语障碍患者沟通能力方面具有巨大潜力。然而，无缝且连贯的言语仍然难以实现，临床效果尚未得到证实。在这里，我们介绍了一种由AI驱动的智能喉(IT)系统，该系统将喉咙肌肉振动和颈动脉脉冲信号传感器与大型语言模型(LLM)处理相结合，以实现流畅、富有情感表达的沟通。该系统利用超敏感的纺织应变传感器捕捉颈部区域的高质量信号，并支持令牌级处理，从而实现实时连续语音解码，使无缝无延迟的交流成为可能。在对五名患有构音障碍的中风患者进行测试时，IT系统的LLM代理智能地纠正了令牌错误并丰富了句子层面的情感和逻辑连贯性，达到了低误码率（4.2%的单词误码率，2.9%的句子误码率），并且用户满意度提高了55%。这项工作为患有构音障碍的患者建立了一个便携且直观的沟通平台，并具有在不同神经条件下的广泛应用潜力以及多语言支持系统的应用可能性。
URL
https://arxiv.org/abs/2411.18266
PDF
-
SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment
2024-11-27 09:18:26Jie Wang, Yichen Wang, Zhilin Zhang, Jianhao Zeng, Kaidi Wang, Zhiyang ChenAbstract
With strong expressive capabilities in Large Language Models(LLMs), generative models effectively capture sentiment structures and deep semantics, however, challenges remain in fine-grained sentiment classification across multi-lingual and complex contexts. To address this, we propose the Sentiment Cross-Lingual Recognition and Logic Framework (SentiXRL), which incorporates two modules,an emotion retrieval enhancement module to improve sentiment classification accuracy in complex contexts through historical dialogue and logical reasoning,and a self-circulating analysis negotiation mechanism (SANM)to facilitates autonomous decision-making within a single model for classification this http URL have validated SentiXRL's superiority on multiple standard datasets, outperforming existing models on CPED and CH-SIMS,and achieving overall better performance on MELD,Emorynlp and IEMOCAP. Notably, we unified labels across several fine-grained sentiment annotation datasets and conducted category confusion experiments, revealing challenges and impacts of class imbalance in standard datasets.
Abstract (translated)
强大的语言模型（LLMs）在表达能力方面表现出色，生成型模型能够有效捕捉情感结构和深层语义。然而，在多语言和复杂背景下进行细粒度情感分类仍存在挑战。为此，我们提出了情感跨语言识别与逻辑框架（SentiXRL），该框架包含两个模块：情感检索增强模块通过历史对话和逻辑推理来提高复杂背景下的情感分类准确性；自我循环分析谈判机制（SANM）促进单个模型内部的自主决策以进行分类。我们在多个标准数据集上验证了SentiXRL的优势，它在CPED和CH-SIMS上的表现超过了现有模型，并且在MELD、EmoryNLP和IEMOCAP上也表现出总体更好的性能。值得注意的是，我们统一了几种细粒度情感标注数据集的标签并进行了类别混淆实验，揭示了标准数据集中类别不平衡带来的挑战及其影响。
URL
https://arxiv.org/abs/2411.18162
PDF
-
Dual Prototyping with Domain and Class Prototypes for Affective Brain-Computer Interface in Unseen Target Conditions
2024-11-27 00:56:43Guangli Li, Zhehao Zhou, Tuo Sun, Ping Tan, Li Zhang, Zhen LiangAbstract
EEG signals have emerged as a powerful tool in affective brain-computer interfaces, playing a crucial role in emotion recognition. However, current deep transfer learning-based methods for EEG recognition face challenges due to the reliance of both source and target data in model learning, which significantly affect model performance and generalization. To overcome this limitation, we propose a novel framework (PL-DCP) and introduce the concepts of feature disentanglement and prototype inference. The dual prototyping mechanism incorporates both domain and class prototypes: domain prototypes capture individual variations across subjects, while class prototypes represent the ideal class distributions within their respective domains. Importantly, the proposed PL-DCP framework operates exclusively with source data during training, meaning that target data remains completely unseen throughout the entire process. To address label noise, we employ a pairwise learning strategy that encodes proximity relationships between sample pairs, effectively reducing the influence of mislabeled data. Experimental validation on the SEED and SEED-IV datasets demonstrates that PL-DCP, despite not utilizing target data during training, achieves performance comparable to deep transfer learning methods that require both source and target data. This highlights the potential of PL-DCP as an effective and robust approach for EEG-based emotion recognition.
Abstract (translated)
URL
https://arxiv.org/abs/2412.00082
PDF
-
Push the Limit of Multi-modal Emotion Recognition by Prompting LLMs with Receptive-Field-Aware Attention Weighting
2024-11-26 18:35:24Liyun Zhang, Dian Ding, Yu Lu, Yi-Chao Chen, Guangtao XueAbstract
Understanding the emotions in a dialogue usually requires external knowledge to accurately understand the contents. As the LLMs become more and more powerful, we do not want to settle on the limited ability of the pre-trained language model. However, the LLMs either can only process text modality or are too expensive to process the multimedia in- formation. We aim to utilize both the power of LLMs and the supplementary features from the multimedia modalities. In this paper, we present a framework, Lantern, that can improve the performance of a certain vanilla model by prompting large language models with receptive-field-aware attention weighting. This framework trained a multi-task vanilla model to produce probabilities of emotion classes and dimension scores. These predictions are fed into the LLMs as references to adjust the predicted probabilities of each emotion class with its external knowledge and contextual understanding. We slice the dialogue into different receptive fields, and each sample is included in exactly t receptive fields. Finally, the predictions of LLMs are merged with a receptive-field-aware attention-driven weighting module. In the experiments, vanilla models CORECT and SDT are deployed in Lantern with GPT-4 or Llama-3.1-405B. The experiments in IEMO- CAP with 4-way and 6-way settings demonstrated that the Lantern can significantly improve the performance of current vanilla models by up to 1.23% and 1.80%.
Abstract (translated)
理解对话中的情感通常需要外部知识才能准确理解内容。随着大型语言模型（LLMs）变得越来越强大，我们不希望满足于预训练语言模型的有限能力。然而，这些LLMs要么只能处理文本模式，要么处理多模态信息的成本太高。我们的目标是利用LLMs的能力以及来自多模态补充特征的优势。在这篇论文中，我们提出了一种框架Lantern，它可以通过使用具有感受野感知注意力加权的大语言模型提示来改进特定的基本模型性能。该框架训练一个多任务基本模型生成情感类别概率和维度得分。这些预测被输入到LLMs作为参考，以调整每个情感类别的预测概率，结合其外部知识和上下文理解。我们将对话分割为不同的感受野，并确保每个样本恰好包含在t个感受野中。最后，通过一个感受野感知注意力驱动加权模块合并LLMs的预测结果。实验中，在Lantern框架中部署了CORECT和SDT这两种基本模型，分别配合使用GPT-4或Llama-3.1-405B。在IEMOCAP数据集上进行的四分类和六分类设置实验表明，Lantern能够显著提升现有基本模型的性能，最高可提高1.23%和1.80%。
URL
https://arxiv.org/abs/2411.17674
PDF
-
'Stupid robot, I want to speak to a human!' User Frustration Detection in Task-Oriented Dialog Systems
2024-11-26 13:51:48Mireia Hernandez Caralt, Ivan Sekuli\'c, Filip Carevi\'c, Nghia Khau, Diana Nicoleta Popa, Bruna Guedes, Victor Guimar\~aes, Zeyu Yang, Andre Manso, Meghana Reddy, Paolo Rosso, Roland MathisAbstract
Detecting user frustration in modern-day task-oriented dialog (TOD) systems is imperative for maintaining overall user satisfaction, engagement, and retention. However, most recent research is focused on sentiment and emotion detection in academic settings, thus failing to fully encapsulate implications of real-world user data. To mitigate this gap, in this work, we focus on user frustration in a deployed TOD system, assessing the feasibility of out-of-the-box solutions for user frustration detection. Specifically, we compare the performance of our deployed keyword-based approach, open-source approaches to sentiment analysis, dialog breakdown detection methods, and emerging in-context learning LLM-based detection. Our analysis highlights the limitations of open-source methods for real-world frustration detection, while demonstrating the superior performance of the LLM-based approach, achieving a 16\% relative improvement in F1 score on an internal benchmark. Finally, we analyze advantages and limitations of our methods and provide an insight into user frustration detection task for industry practitioners.
Abstract (translated)
检测现代任务导向对话（TOD）系统中的用户挫败感对于保持整体用户满意度、参与度和留存率至关重要。然而，最近的大部分研究都集中在学术环境下的情感和情绪识别上，未能完全涵盖现实世界用户数据的影响。为了解决这一差距，在这项工作中，我们专注于部署在TOD系统中的用户挫败感问题，并评估现成解决方案用于检测用户挫败感的可行性。具体来说，我们将已部署的关键字方法、开源的情感分析方法、对话中断检测方法以及新兴的情境学习LLM（大型语言模型）为基础的检测方法进行了比较。我们的分析显示了开源方法在现实世界挫败感检测中的局限性，并展示了基于LLM的方法的优越性能，在内部基准测试中实现了16%的F1分数相对提升。最后，我们分析了这些方法的优势和限制，并为行业从业者提供了有关用户挫败感检测任务的见解。
URL
https://arxiv.org/abs/2411.17437
PDF
-
Multimodal Alignment and Fusion: A Survey
2024-11-26 02:10:27Songtao Li, Hao TangAbstract
This survey offers a comprehensive review of recent advancements in multimodal alignment and fusion within machine learning, spurred by the growing diversity of data types such as text, images, audio, and video. Multimodal integration enables improved model accuracy and broader applicability by leveraging complementary information across different modalities, as well as facilitating knowledge transfer in situations with limited data. We systematically categorize and analyze existing alignment and fusion techniques, drawing insights from an extensive review of more than 200 relevant papers. Furthermore, this survey addresses the challenges of multimodal data integration - including alignment issues, noise resilience, and disparities in feature representation - while focusing on applications in domains like social media analysis, medical imaging, and emotion recognition. The insights provided are intended to guide future research towards optimizing multimodal learning systems to enhance their scalability, robustness, and generalizability across various applications.
Abstract (translated)
这项调查提供了对机器学习中多模态对齐和融合近期进展的全面回顾，这些进展受到文本、图像、音频和视频等数据类型日益多样化的影响。多模态集成通过利用不同模态之间互补的信息以及促进在数据有限情况下的知识转移来提高模型的准确性和适用范围。我们系统地分类和分析了现有的对齐和融合技术，并从超过200篇相关论文的广泛回顾中汲取见解。此外，本调查还关注多模态数据集成面临的挑战，包括对齐问题、抗噪能力和特征表示差异等问题，并重点关注其在社交媒体分析、医学影像和情感识别等领域的应用。所提供的见解旨在指导未来的研究优化多模态学习系统，以提高其在各种应用场景中的可扩展性、鲁棒性和泛化能力。
URL
https://arxiv.org/abs/2411.17040
PDF
