Self-Supervised
-
On Separate Normalization in Self-supervised Transformers
2023-09-22 15:30:53Xiaohui Chen, Yinkai Wang, Yuanqi Du, Soha Hassoun, Li-Ping LiuAbstract
Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average 2.7% performance improvement over the image, natural language, and graph domains.
Abstract (translated)
Transformer的自监督训练方法已经在各种领域表现出卓越的性能。以前的Transformer模型,如掩码自动编码器(MAE),通常只使用[CLS]符号和代币的单个标准化层。在本文中,我们提出了一种简单的修改,使用不同的标准化层来处理代币和[CLS]符号,更好地捕捉它们的独特特征并提高后续任务表现。我们的方法旨在减轻使用相同标准化统计对不同代币类型的潜在负面影响,这些可能不太与它们各自的角色最佳匹配。我们的经验表明,通过使用独立的标准化层,[CLS]嵌入可以更好地编码全球上下文信息,并在其非均匀空间中更均匀地分布。当将传统的标准化层替换为两个独立的层时,我们观察到与图像、自然语言和图形 domains相比,平均性能提高了2.7%。
URL
https://arxiv.org/abs/2309.12931
PDF
-
VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks
2023-09-22 15:03:41Heitor R. Guimarães, Arthur Pimentel, Anderson Avila, Tiago H. FalkAbstract
Keyword spotting (KWS) refers to the task of identifying a set of predefined words in audio streams. With the advances seen recently with deep neural networks, it has become a popular technology to activate and control small devices, such as voice assistants. Relying on such models for edge devices, however, can be challenging due to hardware constraints. Moreover, as adversarial attacks have increased against voice-based technologies, developing solutions robust to such attacks has become crucial. In this work, we propose VIC-KD, a robust distillation recipe for model compression and adversarial robustness. Using self-supervised speech representations, we show that imposing geometric priors to the latent representations of both Teacher and Student models leads to more robust target models. Experiments on the Google Speech Commands datasets show that the proposed methodology improves upon current state-of-the-art robust distillation methods, such as ARD and RSLAD, by 12% and 8% in robust accuracy, respectively.
Abstract (translated)
关键字检测(KWS)是指识别音频流中的预先定义词汇的任务。随着深度学习网络的最新进展,它已经成为激活和控制小型设备的流行技术,例如语音助手。然而,依靠此类模型来处理边缘设备可能会由于硬件限制而面临挑战。此外,随着对基于语音技术的对抗攻击的增加,开发对此类攻击具有鲁棒性的解决方案变得越来越重要。在这个研究中,我们提出了VIC-KD,一个模型压缩和对抗鲁棒性的鲁棒分岔方法。通过使用自监督语音表示,我们证明了在教师和学生模型的潜在表示中添加几何先验可以生成更加鲁棒的目标模型。在Google Speech 命令数据集上的实验表明,该方法在鲁棒精度方面相对于当前先进的鲁棒分岔方法如ard和RSLAD分别提高了12%和8%。
URL
https://arxiv.org/abs/2309.12914
PDF
-
A Study on Incorporating Whisper for Robust Speech Assessment
2023-09-22 10:11:05Ryandhimas E. Zezario, Yu-Wen Chen, Yu Tsao, Szu-Wei Fu, Hsin-Min Wang, Chiou-Shann FuhAbstract
This research introduces an enhanced version of the multi-objective speech assessment model, called MOSA-Net+, by leveraging the acoustic features from large pre-trained weakly supervised models, namely Whisper, to create embedding features. The first part of this study investigates the correlation between the embedding features of Whisper and two self-supervised learning (SSL) models with subjective quality and intelligibility scores. The second part evaluates the effectiveness of Whisper in deploying a more robust speech assessment model. Third, the possibility of combining representations from Whisper and SSL models while deploying MOSA-Net+ is analyzed. The experimental results reveal that Whisper's embedding features correlate more strongly with subjective quality and intelligibility than other SSL's embedding features, contributing to more accurate prediction performance achieved by MOSA-Net+. Moreover, combining the embedding features from Whisper and SSL models only leads to marginal improvement. As compared to MOSA-Net and other SSL-based speech assessment models, MOSA-Net+ yields notable improvements in estimating subjective quality and intelligibility scores across all evaluation metrics. We further tested MOSA-Net+ on Track 3 of the VoiceMOS Challenge 2023 and obtained the top-ranked performance.
Abstract (translated)
这项研究介绍了一种增强版本的多目标语音评估模型,称为MOSA-Net+,通过利用大型弱监督预训练模型Whisper的声学特征来创建嵌入特征。本研究第一部分研究了Whisper的嵌入特征与两个基于自我监督学习(SSL)模型的主观质量和语音识别得分之间的相关性。本研究第二部分评估了Whisper在部署更稳健的语音评估模型方面的 effectiveness。第三部分分析了在部署MOSA-Net+的同时,将Whisper和SSL模型的表示相结合的可能性。实验结果显示,Whisper的嵌入特征与主观质量和语音识别得分之间的相关性比SSL模型的其他嵌入特征更强,为MOSA-Net+实现的更准确的预测性能做出了贡献。此外,将Whisper和SSL模型的嵌入特征相结合仅会导致微小改进。与MOSA-Net和其他基于SSL的语音评估模型相比,MOSA-Net+在估计主观质量和语音识别得分方面实现了显著的改进。我们在2023年声音MOS挑战 track 3 上测试了MOSA-Net+,并取得了排名最高的性能。
URL
https://arxiv.org/abs/2309.12766
PDF
-
Reduce, Reuse, Recycle: Is Perturbed Data better than Other Language augmentation for Low Resource Self-Supervised Speech Models
2023-09-22 10:09:09Asad Ullah, Alessandro Ragano, Andrew HinesAbstract
Self-supervised representation learning (SSRL) has improved the performance on downstream phoneme recognition versus supervised models. Training SSRL models requires a large amount of pre-training data and this poses a challenge for low resource languages. A common approach is transferring knowledge from other languages. Instead, we propose to use audio augmentation to pre-train SSRL models in a low resource condition and evaluate phoneme recognition as downstream task. We performed a systematic comparison of augmentation techniques, namely: pitch variation, noise addition, accented target-language speech and other language speech. We found combined augmentations (noise/pitch) was the best augmentation strategy outperforming accent and language knowledge transfer. We compared the performance with various quantities and types of pre-training data. We examined the scaling factor of augmented data to achieve equivalent performance to models pre-trained with target domain speech. Our findings suggest that for resource constrained languages, in-domain synthetic augmentation can outperform knowledge transfer from accented or other language speech.
Abstract (translated)
自监督表示学习(SSRL)已经提高了后续音节识别相对于监督模型的性能。训练SSRL模型需要大量的预训练数据,这对资源有限的语言来说是一个挑战。一种常见方法是从其他语言中转移知识。相反,我们建议利用音频增强来在资源有限的情况下预训练SSRL模型,并将音节识别作为后续任务进行评估。我们对增强技术进行了系统性的比较,包括音调变化、噪音添加、目标语言语音带有口音以及其他语言语音。我们发现合并增强(噪音/音调)是最佳的增强策略,比口音和语言知识转移表现更好。我们与各种数量和类型的预训练数据进行了比较,并研究了增强数据的缩放因子,以获得与目标语言语音预训练模型相当的性能。我们的发现表明,对于资源有限的语言来说,跨域合成增强可以优于带有口音或其他语言语音的知识转移。
URL
https://arxiv.org/abs/2309.12763
PDF
-
Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Where
2023-09-22 09:58:38Zhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen, Wei-Chen ChiuAbstract
While image data starts to enjoy the simple-but-effective self-supervised learning scheme built upon masking and self-reconstruction objective thanks to the introduction of tokenization procedure and vision transformer backbone, convolutional neural networks as another important and widely-adopted architecture for image data, though having contrastive-learning techniques to drive the self-supervised learning, still face the difficulty of leveraging such straightforward and general masking operation to benefit their learning process significantly. In this work, we aim to alleviate the burden of including masking operation into the contrastive-learning framework for convolutional neural networks as an extra augmentation method. In addition to the additive but unwanted edges (between masked and unmasked regions) as well as other adverse effects caused by the masking operations for ConvNets, which have been discussed by prior works, we particularly identify the potential problem where for one view in a contrastive sample-pair the randomly-sampled masking regions could be overly concentrated on important/salient objects thus resulting in misleading contrastiveness to the other view. To this end, we propose to explicitly take the saliency constraint into consideration in which the masked regions are more evenly distributed among the foreground and background for realizing the masking-based augmentation. Moreover, we introduce hard negative samples by masking larger regions of salient patches in an input image. Extensive experiments conducted on various datasets, contrastive learning mechanisms, and downstream tasks well verify the efficacy as well as the superior performance of our proposed method with respect to several state-of-the-art baselines.
Abstract (translated)
尽管图像数据开始享受基于遮蔽和自重构目标的简单但有效的自我监督学习方案,由于引入了 tokenization 过程和视觉Transformer骨架,卷积神经网络也成为了另一种重要且广泛应用的图像数据架构。尽管卷积神经网络有对比学习技术来驱动自我监督学习,但它们仍然面临着利用这种简单而普遍的遮蔽操作来显著改善其学习过程的困难。在这项工作中,我们旨在减轻将遮蔽操作纳入对比学习框架,作为增加的增广方法,对卷积神经网络作为对比学习框架额外的增广方法的负担。除了无害的边缘(在遮蔽和未被遮蔽区域之间)以及由卷积神经网络的遮蔽操作引起的其他不利效应,我们特别发现了一种潜在问题,即在一个对比样本对中,随机选择的遮蔽区域可能过于集中在重要或显著的对象上,从而导致对另一个视图的学习对比度产生误导。为此,我们建议 explicitly 考虑可见性约束,其中遮蔽区域在 foreground 和 background 之间更均匀地分布以实现基于遮蔽的增广。此外,我们通过在输入图像中遮蔽较大的显著斑点来引入硬负样本。我们对多种数据集、对比学习和后续任务进行了广泛的实验,并成功地证明了我们提出的方法和几个前沿基准之间的差距。
URL
https://arxiv.org/abs/2309.12757
PDF
-
Unsupervised Representations Improve Supervised Learning in Speech Emotion Recognition
2023-09-22 08:54:06Amirali Soltani Tehrani, Niloufar Faridani, Ramin ToosiAbstract
Speech Emotion Recognition (SER) plays a pivotal role in enhancing human-computer interaction by enabling a deeper understanding of emotional states across a wide range of applications, contributing to more empathetic and effective communication. This study proposes an innovative approach that integrates self-supervised feature extraction with supervised classification for emotion recognition from small audio segments. In the preprocessing step, to eliminate the need of crafting audio features, we employed a self-supervised feature extractor, based on the Wav2Vec model, to capture acoustic features from audio data. Then, the output featuremaps of the preprocessing step are fed to a custom designed Convolutional Neural Network (CNN)-based model to perform emotion classification. Utilizing the ShEMO dataset as our testing ground, the proposed method surpasses two baseline methods, i.e. support vector machine classifier and transfer learning of a pretrained CNN. comparing the propose method to the state-of-the-art methods in SER task indicates the superiority of the proposed method. Our findings underscore the pivotal role of deep unsupervised feature learning in elevating the landscape of SER, offering enhanced emotional comprehension in the realm of human-computer interactions.
Abstract (translated)
语音情感识别( SER )在增强人机交互方面发挥着关键作用,通过使对多种应用程序中情感状态的理解更深入,有助于更同情心和有效的沟通。本研究提出了一种创新的方法,将自监督特征提取与监督分类集成,从小型音频片段中情感识别。在预处理步骤中,为了消除制作音频特征的需要,我们采用了基于瓦夫2Vec模型的自监督特征提取器,从音频数据中提取声学特征。然后将预处理步骤的输出特征映射喂入一个定制的卷积神经网络(CNN)基模型,进行情感分类。利用 ShEMO 数据集作为我们的测试集,所提出的方法超过了两个基准方法,即支持向量机分类器和预先训练的 CNN 的迁移学习。将所提出的方法与 SER 任务中的先进方法进行比较表明这种方法的优势。我们的研究结果强调了深度无监督特征学习在提高 SER 景观方面的关键作用,提供了在人机交互领域中增强情感理解。
URL
https://arxiv.org/abs/2309.12714
PDF
-
NTT speaker diarization system for CHiME-7: multi-domain, multi-microphone End-to-end and vector clustering diarization
2023-09-22 06:53:34Naohiro Tawara, Marc Delcroix, Atsushi Ando, Atsunori OgawaAbstract
This paper details our speaker diarization system designed for multi-domain, multi-microphone casual conversations. The proposed diarization pipeline uses weighted prediction error (WPE)-based dereverberation as a front end, then applies end-to-end neural diarization with vector clustering (EEND-VC) to each channel separately. It integrates the diarization result obtained from each channel using diarization output voting error reduction plus overlap (DOVER-LAP). To harness the knowledge from the target domain and results integrated across all channels, we apply self-supervised adaptation for each session by retraining the EEND-VC with pseudo-labels derived from DOVER-LAP. The proposed system was incorporated into NTT's submission for the distant automatic speech recognition task in the CHiME-7 challenge. Our system achieved 65 % and 62 % relative improvements on development and eval sets compared to the organizer-provided VC-based baseline diarization system, securing third place in diarization performance.
Abstract (translated)
这篇文章介绍了我们设计的适用于多领域、多麦克风 casual conversations 的 speaker diarization 系统。该 diarization 系统采用基于加权预测误差(WPE)的声学去混响作为前端,然后将全端神经网络声学归一化与向量聚类(EEND-VC)分别应用于每个通道。它通过减少 diarization 输出的投票错误以及融合(DOVER-LAP)来实现每个通道的声学归一化结果,并将它们整合在一起。为了从目标领域中提取知识和将整合在所有通道中的结果进行训练,我们在每个会话中使用自监督适应技术,通过从 DOVER-LAP 中推导出伪标签来重新训练 EEND-VC。该提议系统被 NTT 纳入了 CHiME-7 挑战中远程自动语音识别任务提交的候选列表中。我们的系统相对于组织者提供的基于 VC 基线的声学基线 diarization 系统在开发和应用集上实现了 65 % 和 62 % 的相对改进,确保了声学归一化性能的第三名。
URL
https://arxiv.org/abs/2309.12656
PDF
-
Self-Supervised Contrastive Learning for Robust Audio-Sheet Music Retrieval Systems
2023-09-21 14:54:48Luis Carvalho, Tobias Washüttl, Gerhard WidmerAbstract
Linking sheet music images to audio recordings remains a key problem for the development of efficient cross-modal music retrieval systems. One of the fundamental approaches toward this task is to learn a cross-modal embedding space via deep neural networks that is able to connect short snippets of audio and sheet music. However, the scarcity of annotated data from real musical content affects the capability of such methods to generalize to real retrieval scenarios. In this work, we investigate whether we can mitigate this limitation with self-supervised contrastive learning, by exposing a network to a large amount of real music data as a pre-training step, by contrasting randomly augmented views of snippets of both modalities, namely audio and sheet images. Through a number of experiments on synthetic and real piano data, we show that pre-trained models are able to retrieve snippets with better precision in all scenarios and pre-training configurations. Encouraged by these results, we employ the snippet embeddings in the higher-level task of cross-modal piece identification and conduct more experiments on several retrieval configurations. In this task, we observe that the retrieval quality improves from 30% up to 100% when real music data is present. We then conclude by arguing for the potential of self-supervised contrastive learning for alleviating the annotated data scarcity in multi-modal music retrieval models.
Abstract (translated)
将音乐谱与音频录制链接仍然是开发高效跨modal音乐检索系统的关键问题。对于这个任务,一种基本的方法是通过深度神经网络学习一个跨modal嵌入空间,该空间能够连接音频和音乐片段的短 snippet。然而,从真实音乐内容的标注数据稀缺性的角度来看,这些方法是否能够适用于真实的检索场景具有影响。在这项工作中,我们研究是否能够通过自监督比较学习来减轻这种限制,方法是将网络暴露在大量的真实音乐数据上作为预训练步骤,通过随机增强的两种模式片段的视图进行比较。通过模拟和真实的钢琴数据进行了一系列实验,我们表明预训练模型能够在所有场景和预训练配置下更准确地检索 snippet。因为这些结果的鼓励,我们使用 snippet嵌入在跨modal片段识别的高级任务中,并进行了更多的实验,针对多个检索配置。在这个任务中,我们观察到当存在真实音乐数据时,检索质量从30%增加到100%。因此我们最终得出结论,自监督比较学习的潜力有助于减轻跨modal音乐检索模型中标注数据稀缺性的问题。
URL
https://arxiv.org/abs/2309.12134
PDF
-
Unveiling the Hidden Realm: Self-supervised Skeleton-based Action Recognition in Occluded Environments
2023-09-21 12:51:11Yifei Chen, Kunyu Peng, Alina Roitberg, David Schneider, Jiaming Zhang, Junwei Zheng, Ruiping Liu, Yufan Chen, Kailun Yang, Rainer StiefelhagenAbstract
To integrate action recognition methods into autonomous robotic systems, it is crucial to consider adverse situations involving target occlusions. Such a scenario, despite its practical relevance, is rarely addressed in existing self-supervised skeleton-based action recognition methods. To empower robots with the capacity to address occlusion, we propose a simple and effective method. We first pre-train using occluded skeleton sequences, then use k-means clustering (KMeans) on sequence embeddings to group semantically similar samples. Next, we employ K-nearest-neighbor (KNN) to fill in missing skeleton data based on the closest sample neighbors. Imputing incomplete skeleton sequences to create relatively complete sequences as input provides significant benefits to existing skeleton-based self-supervised models. Meanwhile, building on the state-of-the-art Partial Spatio-Temporal Learning (PSTL), we introduce an Occluded Partial Spatio-Temporal Learning (OPSTL) framework. This enhancement utilizes Adaptive Spatial Masking (ASM) for better use of high-quality, intact skeletons. The effectiveness of our imputation methods is verified on the challenging occluded versions of the NTURGB+D 60 and NTURGB+D 120. The source code will be made publicly available at this https URL.
Abstract (translated)
将行动识别方法整合到自主机器人系统中,必须考虑涉及目标遮挡的不利情况。尽管这种场景的实际 relevance 很低,但在当前基于骨骼的行动识别方法中却很少有人考虑。为了赋予机器人处理遮挡的能力,我们提出了一种简单而有效的方法。我们首先使用遮挡的骨骼序列进行预训练,然后使用 k-means 聚类(KMeans)将序列嵌入向量分组语义相似的样本。接下来,我们使用 KNN 根据最接近的样本邻居填充缺失的骨骼数据。将不完整的骨骼序列输入生成相对完整的序列作为输入,为当前基于骨骼的自监督模型带来重大的好处。同时,基于当前先进的 partial Spatial-Temporal Learning(PSTL)技术,我们提出了一个被改进的遮挡 partial Spatial-Temporal Learning(OPSTL)框架。这种改进利用自适应空间遮蔽(ASM)更好地利用高质量的完整的骨骼。我们的代入方法的有效性在 NturGB+D 60 和 NturGB+D 120 等挑战性的遮挡版本上进行了验证。源代码将在 this https://www.tensorflow.org/zh/api_docs/python/tf/keras/models/Sequential 网站上公开发布。
URL
https://arxiv.org/abs/2309.12029
PDF
-
Elevating Skeleton-Based Action Recognition with Efficient Multi-Modality Self-Supervision
2023-09-21 12:27:43Yiping Wei, Kunyu Peng, Alina Roitberg, Jiaming Zhang, Junwei Zheng, Ruiping Liu, Yufan Chen, Kailun Yang, Rainer StiefelhagenAbstract
Self-supervised representation learning for human action recognition has developed rapidly in recent years. Most of the existing works are based on skeleton data while using a multi-modality setup. These works overlooked the differences in performance among modalities, which led to the propagation of erroneous knowledge between modalities while only three fundamental modalities, i.e., joints, bones, and motions are used, hence no additional modalities are explored. In this work, we first propose an Implicit Knowledge Exchange Module (IKEM) which alleviates the propagation of erroneous knowledge between low-performance modalities. Then, we further propose three new modalities to enrich the complementary information between modalities. Finally, to maintain efficiency when introducing new modalities, we propose a novel teacher-student framework to distill the knowledge from the secondary modalities into the mandatory modalities considering the relationship constrained by anchors, positives, and negatives, named relational cross-modality knowledge distillation. The experimental results demonstrate the effectiveness of our approach, unlocking the efficient use of skeleton-based multi-modality data. Source code will be made publicly available at this https URL.
Abstract (translated)
过去几年中,人类行为识别 self-supervised Representation Learning 快速发展。大多数现有工作都基于骨骼数据,同时使用多模态 setup。这些工作忽略了不同模态的性能差异,导致不同模态之间的错误知识传播,而仅使用三个基本模态(即关节、骨骼和运动),因此没有探索额外的模态。在这项工作中,我们提出了一种隐含知识交换模块(IKEM),减轻低性能模态之间错误知识的传播。我们还提出了三个新的模态,以丰富不同模态之间的互补信息。最后,为了在引入新模态时保持效率,我们提出了一种独特的教师学生框架,将知识从 secondary 模态中提取,并将其转换为必要的模态,称为关系跨模态知识蒸馏。实验结果显示了我们的方法的有效性,解锁了基于骨骼的多模态数据的有效使用。源代码将在 this https URL 上公开发布。
URL
https://arxiv.org/abs/2309.12009
PDF
-
A Study of Forward-Forward Algorithm for Self-Supervised Learning
2023-09-21 10:14:53Jonas Brenig, Radu TimofteAbstract
Self-supervised representation learning has seen remarkable progress in the last few years, with some of the recent methods being able to learn useful image representations without labels. These methods are trained using backpropagation, the de facto standard. Recently, Geoffrey Hinton proposed the forward-forward algorithm as an alternative training method. It utilizes two forward passes and a separate loss function for each layer to train the network without backpropagation. In this study, for the first time, we study the performance of forward-forward vs. backpropagation for self-supervised representation learning and provide insights into the learned representation spaces. Our benchmark employs four standard datasets, namely MNIST, F-MNIST, SVHN and CIFAR-10, and three commonly used self-supervised representation learning techniques, namely rotation, flip and jigsaw. Our main finding is that while the forward-forward algorithm performs comparably to backpropagation during (self-)supervised training, the transfer performance is significantly lagging behind in all the studied settings. This may be caused by a combination of factors, including having a loss function for each layer and the way the supervised training is realized in the forward-forward paradigm. In comparison to backpropagation, the forward-forward algorithm focuses more on the boundaries and drops part of the information unnecessary for making decisions which harms the representation learning goal. Further investigation and research are necessary to stabilize the forward-forward strategy for self-supervised learning, to work beyond the datasets and configurations demonstrated by Geoffrey Hinton.
Abstract (translated)
自监督表示学习在过去几年中取得了显著进展,一些最近的方法能够在没有标签的情况下学习有用的图像表示。这些方法使用回退作为事实上的标准训练方法。最近,Geoffrey Hinton提出了前向-前向算法作为另一种训练方法。它使用两个前向遍历和每个层单独的损失函数来训练网络而无需回退。在本研究中,我们首次研究前向-前向相对于回退在自监督表示学习中的表现,并提供了学到的表示空间 insights。我们的基准使用四个标准数据集,分别是米NIST、F-米NIST、SVHN和CIFAR-10,以及三种常见的自监督表示学习技术,分别是旋转、翻转和拼图。我们的主要发现是,虽然在(自)监督训练中前向-前向算法表现与回退相当,但在所有研究设置中传输性能显著落后。这可能是由多种因素的组合造成的,包括每个层都有一个损失函数以及前向-前向范式中监督训练的实现方式。与回退相比,前向-前向算法更关注边界,并删除不必要的决策信息,这损害了表示学习目标。需要进行进一步研究和研究以稳定自监督学习的前向-前向策略,超越Geoffrey Hinton演示的数据集和配置。
URL
https://arxiv.org/abs/2309.11955
PDF
-
Bridging the Gap: Learning Pace Synchronization for Open-World Semi-Supervised Learning
Abstract
In open-world semi-supervised learning, a machine learning model is tasked with uncovering novel categories from unlabeled data while maintaining performance on seen categories from labeled data. The central challenge is the substantial learning gap between seen and novel categories, as the model learns the former faster due to accurate supervisory information. To address this, we introduce 1) an adaptive margin loss based on estimated class distribution, which encourages a large negative margin for samples in seen classes, to synchronize learning paces, and 2) pseudo-label contrastive clustering, which pulls together samples which are likely from the same class in the output space, to enhance novel class discovery. Our extensive evaluations on multiple datasets demonstrate that existing models still hinder novel class learning, whereas our approach strikingly balances both seen and novel classes, achieving a remarkable 3% average accuracy increase on the ImageNet dataset compared to the prior state-of-the-art. Additionally, we find that fine-tuning the self-supervised pre-trained backbone significantly boosts performance over the default in prior literature. After our paper is accepted, we will release the code.
Abstract (translated)
在开放世界半监督学习中,机器学习模型的任务是发现未标记数据中的新类别,同时保持从标记数据中可见类别的性能。其核心挑战是可见类别和新类别之间的显著学习差距,因为模型更快地学习可见类别这是由于准确的监督信息。为了解决这一问题,我们引入了以下两种方法:1)基于估计类分布的自适应margin损失,这鼓励可见类别样本具有较大的负梯度,以同步学习速度,2)伪标签Contrastive聚类,它将可能在输出空间中来自同一类样本的其他样本聚集在一起,以增强新类别发现。我们对多个数据集进行了广泛的评估,表明现有模型仍然阻碍新类别学习,而我们的算法则戏剧性地平衡了可见类别和新类别,在ImageNet数据集上实现了与之前最佳方法相比显著的3%平均准确性提高。此外,我们发现微调自监督的前训练主干线显著增强了之前文献中的默认表现。在我们论文被接受后,我们将发布代码。
URL
https://arxiv.org/abs/2309.11930
PDF
-
Unlocking the Heart Using Adaptive Locked Agnostic Networks
2023-09-21 09:06:36Sylwia Majchrowska, Anders Hildeman, Philip Teare, Tom DietheAbstract
Supervised training of deep learning models for medical imaging applications requires a significant amount of labeled data. This is posing a challenge as the images are required to be annotated by medical professionals. To address this limitation, we introduce the Adaptive Locked Agnostic Network (ALAN), a concept involving self-supervised visual feature extraction using a large backbone model to produce anatomically robust semantic self-segmentation. In the ALAN methodology, this self-supervised training occurs only once on a large and diverse dataset. Due to the intuitive interpretability of the segmentation, downstream models tailored for specific tasks can be easily designed using white-box models with few parameters. This, in turn, opens up the possibility of communicating the inner workings of a model with domain experts and introducing prior knowledge into it. It also means that the downstream models become less data-hungry compared to fully supervised approaches. These characteristics make ALAN particularly well-suited for resource-scarce scenarios, such as costly clinical trials and rare diseases. In this paper, we apply the ALAN approach to three publicly available echocardiography datasets: EchoNet-Dynamic, CAMUS, and TMED-2. Our findings demonstrate that the self-supervised backbone model robustly identifies anatomical subregions of the heart in an apical four-chamber view. Building upon this, we design two downstream models, one for segmenting a target anatomical region, and a second for echocardiogram view classification.
Abstract (translated)
为医学成像应用训练深度学习模型需要大量标记数据,这提出了挑战,因为图像需要由医疗专业人士标注。为了解决这个问题,我们引入了Adaptive Locked Agnostic Network(ALAN),这是一个涉及使用大型基线模型自我监督视觉特征提取的概念,以产生结构稳定的语义自我分割。在ALAN方法中,这种方法仅在一个大型、多样化的数据集上进行一次自我监督训练。由于分割的直观解释性,可以很容易地使用几个参数较少的白色盒模型设计针对特定任务的目标模型。这反过来增加了与域专家进行沟通并引入先前知识的可能性,同时也意味着目标模型相对于完全监督方法来说需要更多的数据。这些特点使ALAN特别适用于资源匮乏的情况,如昂贵的临床试验和罕见的疾病。在本文中,我们应用ALAN方法访问了三个公开的心脏超声数据集: EchoNet-Dynamic、CAMUS和Tmed-2。我们的发现表明,自我监督基线模型 robustly identifies the anatomical subregions of the heart in an apical four-chamber view。基于这一点,我们设计了两个后续的模型,一个用于分割目标解剖学区域,另一个用于心脏超声视图分类。
URL
https://arxiv.org/abs/2309.11899
PDF
-
Multi-level Asymmetric Contrastive Learning for Medical Image Segmentation Pre-training
2023-09-21 08:22:44Shuang Zeng, Lei Zhu, Xinliang Zhang, Zifeng Tian, Qian Chen, Lujia Jin, Jiayi Wang, Yanye LuAbstract
Contrastive learning, which is a powerful technique for learning image-level representations from unlabeled data, leads a promising direction to dealing with the dilemma between large-scale pre-training and limited labeled data. However, most existing contrastive learning strategies are designed mainly for downstream tasks of natural images, therefore they are sub-optimal and even worse than learning from scratch when directly applied to medical images whose downstream tasks are usually segmentation. In this work, we propose a novel asymmetric contrastive learning framework named JCL for medical image segmentation with self-supervised pre-training. Specifically, (1) A novel asymmetric contrastive learning strategy is proposed to pre-train both encoder and decoder simultaneously in one-stage to provide better initialization for segmentation models. (2) A multi-level contrastive loss is designed to take the correspondence among feature-level, image-level and pixel-level projections, respectively into account to make sure multi-level representations can be learned by the encoder and decoder during pre-training. (3) Experiments on multiple medical image datasets indicate our JCL framework outperforms existing SOTA contrastive learning strategies.
Abstract (translated)
对比学习(Contrastive Learning)是一种从未标记数据中学习图像级表示的强大技术,提供了解决大规模预训练和少量标记数据的困境的有前途的方向。然而,目前大多数对比学习策略主要设计用于自然图像的后续任务,因此它们的优劣程度和对 medical images 的后续任务通常是分割的后续任务,直接应用于这些后续任务通常比从头开始学习更差。在本研究中,我们提出了一种名为 JCL 的新不对称对比学习框架,用于医学图像分割,并采用自监督的预训练。具体来说,(1) 我们提出了一种新不对称对比学习策略,在一步中同时预训练编码器和解码器,为分割模型提供更好的初始化。(2) 我们设计了一个多级对比损失,考虑特征级、图像级和像素级投影之间的对应关系,以确保编码器和解码器在预训练期间可以学习多级表示。(3) 对多个医学图像数据集的实验表明,我们的 JCL 框架优于现有的 SOTA 对比学习策略。
URL
https://arxiv.org/abs/2309.11876
PDF
-
DimCL: Dimensional Contrastive Learning For Improving Self-Supervised Learning
2023-09-21 05:12:55Thanh Nguyen, Trung Pham, Chaoning Zhang, Tung Luu, Thang Vu, Chang D. YooAbstract
Self-supervised learning (SSL) has gained remarkable success, for which contrastive learning (CL) plays a key role. However, the recent development of new non-CL frameworks has achieved comparable or better performance with high improvement potential, prompting researchers to enhance these frameworks further. Assimilating CL into non-CL frameworks has been thought to be beneficial, but empirical evidence indicates no visible improvements. In view of that, this paper proposes a strategy of performing CL along the dimensional direction instead of along the batch direction as done in conventional contrastive learning, named Dimensional Contrastive Learning (DimCL). DimCL aims to enhance the feature diversity, and it can serve as a regularizer to prior SSL frameworks. DimCL has been found to be effective, and the hardness-aware property is identified as a critical reason for its success. Extensive experimental results reveal that assimilating DimCL into SSL frameworks leads to performance improvement by a non-trivial margin on various datasets and backbone architectures.
Abstract (translated)
自监督学习(SSL)取得了显著的成功,而对比学习(CL)则起到了关键作用。然而,最近新非CL框架的发展已经实现了性能和改善潜力的相当或更好的表现,促使研究人员进一步改进这些框架。认为将CL代入非CL框架中是有益的,但经验证据表明并没有明显改进。因此,本文提出了一种策略,即沿着维度方向而不是传统对比学习中的方向进行CL,并称之为维度对比学习(DimCL)。DimCL的目标是增强特征多样性,它可以作为先前SSL框架的Regularizer。DimCL被发现非常有效,并且 Hardness-aware特性被认为是其成功的关键原因。广泛的实验结果表明,将DimCL代入SSL框架中可以在各种数据集和基础架构方面带来显著的性能改进。
URL
https://arxiv.org/abs/2309.11782
PDF
-
Leveraging In-the-Wild Data for Effective Self-Supervised Pretraining in Speaker Recognition
2023-09-21 02:03:26Shuai Wang, Qibing Bai, Qi Liu, Jianwei Yu, Zhengyang Chen, Bing Han, Yanmin Qian, Haizhou LiAbstract
Current speaker recognition systems primarily rely on supervised approaches, constrained by the scale of labeled datasets. To boost the system performance, researchers leverage large pretrained models such as WavLM to transfer learned high-level features to the downstream speaker recognition task. However, this approach introduces extra parameters as the pretrained model remains in the inference stage. Another group of researchers directly apply self-supervised methods such as DINO to speaker embedding learning, yet they have not explored its potential on large-scale in-the-wild datasets. In this paper, we present the effectiveness of DINO training on the large-scale WenetSpeech dataset and its transferability in enhancing the supervised system performance on the CNCeleb dataset. Additionally, we introduce a confidence-based data filtering algorithm to remove unreliable data from the pretraining dataset, leading to better performance with less training data. The associated pretrained models, confidence files, pretraining and finetuning scripts will be made available in the Wespeaker toolkit.
Abstract (translated)
当前的人名识别系统主要依赖于监督方法,受到标注数据规模的限制。为了提升系统性能,研究人员利用像WavLM这样的大型预训练模型将学习的高级特征转移到后续的人名识别任务中。然而,这种方法引入了额外的参数,因为预训练模型仍然处于推理阶段。另外一组研究人员直接应用自监督方法如DiNO来进行人名嵌入学习,但尚未在大规模野生数据集上探索其潜力。在本文中,我们介绍了DiNO训练在大规模WenetSpeech数据集上的有效性,以及它在提升CNCeleb数据集上监督系统性能的可移植性。此外,我们引入了一种基于信心的数据过滤算法,从预训练数据集中删除不可靠的数据,导致更少的训练数据却更好的表现。相关的预训练模型、信心文件、预训练和微调脚本将在我们的We Speaker工具包中提供。
URL
https://arxiv.org/abs/2309.11730
PDF
-
MoDA: Leveraging Motion Priors from Videos for Advancing Unsupervised Domain Adaptation in Semantic Segmentation
2023-09-21 01:31:54Fei Pan, Xu Yin, Seokju Lee, Sungeui Yoon, In So KweonAbstract
Unsupervised domain adaptation (UDA) is an effective approach to handle the lack of annotations in the target domain for the semantic segmentation task. In this work, we consider a more practical UDA setting where the target domain contains sequential frames of the unlabeled videos which are easy to collect in practice. A recent study suggests self-supervised learning of the object motion from unlabeled videos with geometric constraints. We design a motion-guided domain adaptive semantic segmentation framework (MoDA), that utilizes self-supervised object motion to learn effective representations in the target domain. MoDA differs from previous methods that use temporal consistency regularization for the target domain frames. Instead, MoDA deals separately with the domain alignment on the foreground and background categories using different strategies. Specifically, MoDA contains foreground object discovery and foreground semantic mining to align the foreground domain gaps by taking the instance-level guidance from the object motion. Additionally, MoDA includes background adversarial training which contains a background category-specific discriminator to handle the background domain gaps. Experimental results on multiple benchmarks highlight the effectiveness of MoDA against existing approaches in the domain adaptive image segmentation and domain adaptive video segmentation. Moreover, MoDA is versatile and can be used in conjunction with existing state-of-the-art approaches to further improve performance.
Abstract (translated)
无监督域适应(UDA)是一种有效的方法,用于处理目标域中缺乏标注数据的任务,即语义分割任务。在本文中,我们考虑一个更加实用的UDA场景,其中目标域包含未标注视频的连续帧,在实践中很容易收集。一项最近的研究建议从未标注视频进行自监督的物体运动学习。我们设计了一个基于运动指导的目标域自适应语义分割框架(MoDA),该框架利用自监督的物体运动学习在目标域中学习有效的表示。MoDA与之前的方法不同,使用目标域帧的时序一致性 regularization,而不同于以往的方法,MoDA采用不同的策略来处理前方和背景类别之间的域对齐。具体而言,MoDA包括前方物体发现和前方语义挖掘,通过从物体运动获取实例级指导,将前方域差距对齐。此外,MoDA还包括背景对抗训练,其中包含背景类别特定的分类器,以处理背景域差距。多个基准测试的实验结果强调了MoDA在域自适应图像分割和域自适应视频分割中的效果。此外,MoDA是灵活的,可以与现有的高级方法相结合,以进一步提高性能。
URL
https://arxiv.org/abs/2309.11711
PDF
-
Understanding Pose and Appearance Disentanglement in 3D Human Pose Estimation
2023-09-20 22:22:21Krishna Kanth Nakka, Mathieu SalzmannAbstract
As 3D human pose estimation can now be achieved with very high accuracy in the supervised learning scenario, tackling the case where 3D pose annotations are not available has received increasing attention. In particular, several methods have proposed to learn image representations in a self-supervised fashion so as to disentangle the appearance information from the pose one. The methods then only need a small amount of supervised data to train a pose regressor using the pose-related latent vector as input, as it should be free of appearance information. In this paper, we carry out in-depth analysis to understand to what degree the state-of-the-art disentangled representation learning methods truly separate the appearance information from the pose one. First, we study disentanglement from the perspective of the self-supervised network, via diverse image synthesis experiments. Second, we investigate disentanglement with respect to the 3D pose regressor following an adversarial attack perspective. Specifically, we design an adversarial strategy focusing on generating natural appearance changes of the subject, and against which we could expect a disentangled network to be robust. Altogether, our analyses show that disentanglement in the three state-of-the-art disentangled representation learning frameworks if far from complete, and that their pose codes contain significant appearance information. We believe that our approach provides a valuable testbed to evaluate the degree of disentanglement of pose from appearance in self-supervised 3D human pose estimation.
Abstract (translated)
现如今,在监督学习场景中,3D人类姿态估计可以实现极高的精度。因此,解决3D姿态标注不足的问题已经引起了越来越多的关注。特别是,有一些方法提出了通过自监督学习来学习图像表示,以便将外观信息从姿态信息中分离。这些方法只需要少量的监督数据来训练姿态回归器,使用姿态相关的隐向量作为输入,因为外观信息应该被排除在外。在本文中,我们进行了深入分析,以了解art-of-the-art的分离表示学习方法在何种程度上真正将外观信息从姿态信息中分离。首先,我们从自监督网络的角度出发,通过多种图像合成实验研究了分离的实现方式。其次,我们研究了从攻击视角下的3D姿态回归器的分离实现。具体来说,我们设计了一种新的攻击策略,重点是生成物体的自然外观变化,我们可以期望分离网络具有鲁棒性。总之,我们的分析表明,三个art-of-the-art的分离表示学习框架的分离程度虽然还没有完全完成,但它们的姿态代码中包含重要的外观信息。我们相信,我们的方法提供了评估自监督3D人类姿态估计中姿态与外观分离程度的宝贵测试平台。
URL
https://arxiv.org/abs/2309.11667
PDF
-
Enhancing motion trajectory segmentation of rigid bodies using a novel screw-based trajectory-shape representation
2023-09-20 15:40:22Arno Verduyn, Maxim Vochten, Joris De SchutterAbstract
Trajectory segmentation refers to dividing a trajectory into meaningful consecutive sub-trajectories. This paper focuses on trajectory segmentation for 3D rigid-body motions. Most segmentation approaches in the literature represent the body's trajectory as a point trajectory, considering only its translation and neglecting its rotation. We propose a novel trajectory representation for rigid-body motions that incorporates both translation and rotation, and additionally exhibits several invariant properties. This representation consists of a geometric progress rate and a third-order trajectory-shape descriptor. Concepts from screw theory were used to make this representation time-invariant and also invariant to the choice of body reference point. This new representation is validated for a self-supervised segmentation approach, both in simulation and using real recordings of human-demonstrated pouring motions. The results show a more robust detection of consecutive submotions with distinct features and a more consistent segmentation compared to conventional representations. We believe that other existing segmentation methods may benefit from using this trajectory representation to improve their invariance.
Abstract (translated)
轨迹分割是指将轨迹划分为有意义的连续子轨迹。本文专注于对3D固体运动的轨迹分割。在文献中,大多数分割方法将物体的运动轨迹视为点运动轨迹,仅考虑它的 Translation,而忽略了它的 rotation。我们提出了一种 novel 的运动轨迹表示,该表示包括 Translation 和 rotation,并同时表现出几个不变的性质。该表示由几何进展率和第三级轨迹形状描述器组成。 screw 理论的概念被用来使该表示时间不变,同时也不变于选择物体参考点。该新表示被用于 self-supervised 分割方法的验证,无论是在模拟中还是使用人类示范的倒入动作的实际记录中。结果表明,相较于传统的表示,它能够更有效地检测具有独特特征的连续子运动,并且分割结果更一致。我们认为,其他现有的分割方法可能会从使用这个轨迹表示中受益,以改善它们的不变性。
URL
https://arxiv.org/abs/2309.11413
PDF
-
Self-supervised learning unveils change in urban housing from street-level images
2023-09-20 14:35:23Steven Stalder, Michele Volpi, Nicolas Büttner, Stephen Law, Kenneth Harttgen, Esra SuelAbstract
Cities around the world face a critical shortage of affordable and decent housing. Despite its critical importance for policy, our ability to effectively monitor and track progress in urban housing is limited. Deep learning-based computer vision methods applied to street-level images have been successful in the measurement of socioeconomic and environmental inequalities but did not fully utilize temporal images to track urban change as time-varying labels are often unavailable. We used self-supervised methods to measure change in London using 15 million street images taken between 2008 and 2021. Our novel adaptation of Barlow Twins, Street2Vec, embeds urban structure while being invariant to seasonal and daily changes without manual annotations. It outperformed generic embeddings, successfully identified point-level change in London's housing supply from street-level images, and distinguished between major and minor change. This capability can provide timely information for urban planning and policy decisions toward more liveable, equitable, and sustainable cities.
Abstract (translated)
世界各地的城市面临缺乏 affordable 和体面住房的危机。尽管这对于政策来说非常重要,但我们有效地监测和跟踪城市住房进展的能力是有限的。将深度学习计算机视觉方法应用于街道图像成功测量了社会经济和环境不平等,但并未充分利用时间图像来跟踪城市变化,因为时间变化的标签通常难以获得。我们使用自监督方法使用2008年至2021年间拍摄的1.5亿街道图像来测量伦敦的变化。我们开发了 Barlow Twins 的创新性改编,即 Street2Vec,能够在不进行手动标注的情况下,嵌入城市结构,同时适应季节性和日常变化。它比通用嵌入表现更好,成功从街道图像识别伦敦住房供应的点级别变化,并区分了主要和次要变化。这种能力可以为城市规划和政策决策提供及时信息,以创建更加宜居、平等和可持续的城市。
URL
https://arxiv.org/abs/2309.11354
PDF
