Transfer_Learning
-
The Early Bird Identifies the Worm: You Can't Beat a Head Start in Long-Term Body Re-ID
2025-07-23 16:04:45Thomas M. Metz, Matthew Q. Hill, Alice J. O'TooleAbstract
Person identification in unconstrained viewing environments presents significant challenges due to variations in distance, viewpoint, imaging conditions, and clothing. We introduce $\textbf{E}$va $\textbf{C}$lothes-Change from $\textbf{H}$idden $\textbf{O}$bjects - $\textbf{B}$ody $\textbf{ID}$entification (ECHO-BID), a class of long-term re-id models built on object-pretrained EVA-02 Large backbones. We compare ECHO-BID to 9 other models that vary systematically in backbone architecture, model size, scale of object classification pretraining, and transfer learning protocol. Models were evaluated on benchmark datasets across constrained, unconstrained, and occluded settings. ECHO-BID, with transfer learning on the most challenging clothes-change data, achieved state-of-the-art results on long-term re-id -- substantially outperforming other methods. ECHO-BID also surpassed other methods by a wide margin in occluded viewing scenarios. A combination of increased model size and Masked Image Modeling during pretraining underlie ECHO-BID's strong performance on long-term re-id. Notably, a smaller, but more challenging transfer learning dataset, generalized better across datasets than a larger, less challenging one. However, the larger dataset with an additional fine-tuning step proved best on the most difficult data. Selecting the correct pretrained backbone architecture and transfer learning protocols can drive substantial gains in long-term re-id performance.
Abstract (translated)
在不受限制的观看环境中进行人员识别面临重大挑战，因为距离、视角、成像条件和着装变化都会引起变异。我们介绍了ECHO-BID（Eva Clothes-Change from Hidden Objects - Body IDentification），这是一个基于对象预训练的EVA-02 Large骨干网络构建的长期重识别模型系列。我们在基准数据集上对ECHO-BID和其他9种在骨干架构、模型大小、物体分类预训练规模和迁移学习协议方面系统变化的模型进行了评估，这些数据集涵盖了受限环境、非受限环境以及被遮挡场景。 通过在最具挑战性的着装变化数据上的转移学习，ECHO-BID在长期重识别任务上取得了最先进的结果，大幅超越了其他方法。此外，在观察角度受到遮挡的情况下，ECHO-BID也显著超过了其他方法。增加模型规模和在预训练期间使用Masked Image Modeling是ECHO-BID在长期重识别中表现出色的关键因素。 值得注意的是，尽管较小的、更具挑战性的转移学习数据集比较大的、较不具挑战性的数据集更好地泛化到不同的数据集中，但是更大的数据集加上额外的微调步骤对最困难的数据点表现最佳。选择适当的预训练骨干架构和迁移学习协议可以显著提高长期重识别性能。 总结来说，ECHO-BID通过使用改进的模型规模、特定的预训练技术以及有效的转移学习策略，在解决非受限环境下的人员长期识别问题上取得了突破性的进展。
URL
https://arxiv.org/abs/2507.17640
PDF
-
Robust Five-Class and binary Diabetic Retinopathy Classification Using Transfer Learning and Data Augmentation
2025-07-23 01:52:27Faisal Ahmed, Mohammad Alfrad Nobel BhuiyanAbstract
Diabetic retinopathy (DR) is a leading cause of vision loss worldwide, and early diagnosis through automated retinal image analysis can significantly reduce the risk of blindness. This paper presents a robust deep learning framework for both binary and five-class DR classification, leveraging transfer learning and extensive data augmentation to address the challenges of class imbalance and limited training data. We evaluate a range of pretrained convolutional neural network architectures, including variants of ResNet and EfficientNet, on the APTOS 2019 dataset. For binary classification, our proposed model achieves a state-of-the-art accuracy of 98.9%, with a precision of 98.6%, recall of 99.3%, F1-score of 98.9%, and an AUC of 99.4%. In the more challenging five-class severity classification task, our model obtains a competitive accuracy of 84.6% and an AUC of 94.1%, outperforming several existing approaches. Our findings also demonstrate that EfficientNet-B0 and ResNet34 offer optimal trade-offs between accuracy and computational efficiency across both tasks. These results underscore the effectiveness of combining class-balanced augmentation with transfer learning for high-performance DR diagnosis. The proposed framework provides a scalable and accurate solution for DR screening, with potential for deployment in real-world clinical environments.
Abstract (translated)
糖尿病视网膜病变（DR）是全球视力丧失的主要原因之一，通过自动化视网膜图像分析进行早期诊断可以显著降低失明的风险。本文提出了一种稳健的深度学习框架，用于二分类和五级分级的DR分类，利用迁移学习和广泛的数据增强来解决类别不平衡和训练数据有限的问题。我们在APTOS 2019数据集上评估了多种预训练卷积神经网络架构，包括ResNet及其变体、EfficientNet等。对于二分类任务，我们的模型实现了业内领先的准确率（98.9%），精确度为98.6%，召回率为99.3%，F1分数为98.9%，AUC值为99.4%。在更具挑战性的五级分级严重程度分类任务中，我们的模型获得了竞争力的准确性（84.6%）和AUC（94.1%），超越了多种现有方法。此外，研究还表明EfficientNet-B0和ResNet34在这两项任务上都能提供最优的精度与计算效率之间的平衡。这些结果强调了结合类别均衡增强技术与迁移学习对于实现高性能DR诊断的有效性。提出的框架为DR筛查提供了可扩展且准确的解决方案，并具备在实际临床环境中部署的应用潜力。
URL
https://arxiv.org/abs/2507.17121
PDF
-
A Unifying Scheme for Extractive Content Selection Tasks
2025-07-22 18:02:54Shmuel Amar, Ori Shapira, Aviv Slobodkin, Ido DaganAbstract
A broad range of NLP tasks involve selecting relevant text spans from given source texts. Despite this shared objective, such \textit{content selection} tasks have traditionally been studied in isolation, each with its own modeling approaches, datasets, and evaluation metrics. In this work, we propose \textit{instruction-guided content selection (IGCS)} as a beneficial unified framework for such settings, where the task definition and any instance-specific request are encapsulated as instructions to a language model. To promote this framework, we introduce \igcsbench{}, the first unified benchmark covering diverse content selection tasks. Further, we create a large generic synthetic dataset that can be leveraged for diverse content selection tasks, and show that transfer learning with these datasets often boosts performance, whether dedicated training for the targeted task is available or not. Finally, we address generic inference time issues that arise in LLM-based modeling of content selection, assess a generic evaluation metric, and overall propose the utility of our resources and methods for future content selection models. Models and datasets available at this https URL.
Abstract (translated)
许多自然语言处理（NLP）任务都涉及到从给定的源文本中选择相关的文本片段。尽管这些任务有着共同的目标，即“内容选择”，但它们通常被孤立地研究，每个任务都有自己特定的建模方法、数据集和评估指标。在本工作中，我们提出了**指令引导的内容选择（IGCS）**作为此类设置的一个有益的统一框架，在该框架中，任务定义以及任何实例特定的要求都被封装为提供给语言模型的指令。 为了推广这一框架，我们引入了**\igcsbench**，这是第一个涵盖多种内容选择任务的统一基准。此外，我们创建了一个大型通用合成数据集，它可以用于各种内容选择任务，并展示了使用这些数据集进行迁移学习通常可以提高性能，无论是否为特定任务提供了专门训练的数据。 最后，我们解决了基于大语言模型（LLM）的内容选择建模在推理时间上的一般性问题，评估了一般的评价指标，并总体上提出了我们的资源和方法对未来内容选择模型的实用性。相关模型和数据集可在[此链接](https://this https URL)获取。
URL
https://arxiv.org/abs/2507.16922
PDF
-
Improving U-Net Confidence on TEM Image Data with L2-Regularization, Transfer Learning, and Deep Fine-Tuning
2025-07-22 17:27:33Aiden Ochoa, Xinyuan Xu, Xing WangAbstract
With ever-increasing data volumes, it is essential to develop automated approaches for identifying nanoscale defects in transmission electron microscopy (TEM) images. However, compared to features in conventional photographs, nanoscale defects in TEM images exhibit far greater variation due to the complex contrast mechanisms and intricate defect structures. These challenges often result in much less labeled data and higher rates of annotation errors, posing significant obstacles to improving machine learning model performance for TEM image analysis. To address these limitations, we examined transfer learning by leveraging large, pre-trained models used for natural images. We demonstrated that by using the pre-trained encoder and L2-regularization, semantically complex features are ignored in favor of simpler, more reliable cues, substantially improving the model performance. However, this improvement cannot be captured by conventional evaluation metrics such as F1-score, which can be skewed by human annotation errors treated as ground truth. Instead, we introduced novel evaluation metrics that are independent of the annotation accuracy. Using grain boundary detection in UO2 TEM images as a case study, we found that our approach led to a 57% improvement in defect detection rate, which is a robust and holistic measure of model performance on the TEM dataset used in this work. Finally, we showed that model self-confidence is only achieved through transfer learning and fine-tuning of very deep layers.
Abstract (translated)
随着数据量的不断增加，开发自动识别透射电子显微镜（TEM）图像中纳米级缺陷的方法变得至关重要。然而，与传统照片中的特征相比，由于复杂的对比机制和复杂的缺陷结构，TEM图像中的纳米级缺陷表现出更大的变异性。这些挑战通常导致标记的数据大大减少，并且注释错误率更高，这对提高机器学习模型在TEM图像分析方面的性能构成了重大障碍。 为了解决这些问题，我们研究了迁移学习方法，通过利用大型预训练模型来处理自然图像。我们证明，在使用预训练编码器和L2正则化的情况下，复杂的语义特征会被忽略，而更简单且可靠的线索则被保留下来，从而显著提高了模型的性能。然而，这种改进无法通过传统的评估指标（如F1分数）捕捉到，因为这些传统指标可能会受到被视为真实标签的人类注释错误的影响。因此，我们引入了独立于标注准确性的新型评价指标。 以UO2 TEM图像中的晶界检测为例，我们的方法将缺陷检出率提高了57%，这是一个在用于本工作的TEM数据集上衡量模型性能的稳健且全面的标准。最后，我们证明只有通过迁移学习和对非常深层进行微调才能获得模型自信心。
URL
https://arxiv.org/abs/2507.16779
PDF
-
An empirical study for the early detection of Mpox from skin lesion images using pretrained CNN models leveraging XAI technique
2025-07-21 17:30:08Mohammad Asifur Rahim, Muhammad Nazmul Arefin, Md. Mizanur Rahman, Md Ali Hossain, Ahmed MoustafaAbstract
Context: Mpox is a zoonotic disease caused by the Mpox virus, which shares similarities with other skin conditions, making accurate early diagnosis challenging. Artificial intelligence (AI), especially Deep Learning (DL), has a strong tool for medical image analysis; however, pre-trained models like CNNs and XAI techniques for mpox detection is underexplored. Objective: This study aims to evaluate the effectiveness of pre-trained CNN models (VGG16, VGG19, InceptionV3, MobileNetV2) for the early detection of monkeypox using binary and multi-class datasets. It also seeks to enhance model interpretability using Grad-CAM an XAI technique. Method: Two datasets, MSLD and MSLD v2.0, were used for training and validation. Transfer learning techniques were applied to fine-tune pre-trained CNN models by freezing initial layers and adding custom layers for adapting the final features for mpox detection task and avoid overfitting. Models performance were evaluated using metrics such as accuracy, precision, recall, F1-score and ROC. Grad-CAM was utilized for visualizing critical features. Results: InceptionV3 demonstrated the best performance on the binary dataset with an accuracy of 95%, while MobileNetV2 outperformed on the multi-class dataset with an accuracy of 93%. Grad-CAM successfully highlighted key image regions. Despite high accuracy, some models showed overfitting tendencies, as videnced by discrepancies between training and validation losses. Conclusion: This study underscores the potential of pre-trained CNN models in monkeypox detection and the value of XAI techniques. Future work should address dataset limitations, incorporate multimodal data, and explore additional interpretability techniques to improve diagnostic reliability and model transparency
Abstract (translated)
**翻译** 背景：猴痘是一种由猴痘病毒引起的动物源性疾病，其临床表现与其他皮肤疾病相似，使得早期准确诊断变得具有挑战性。人工智能（AI），特别是深度学习（DL）在医学图像分析方面表现出强大的能力；然而，预训练模型如卷积神经网络（CNNs）和用于猴痘检测的可解释的人工智能（XAI）技术目前研究较少。 **目的：** 本研究旨在评估VGG16、VGG19、InceptionV3和MobileNetV2等预训练CNN模型在二分类和多分类数据集上早期诊断猴痘的有效性。同时，利用梯度加权类激活映射（Grad-CAM）这一XAI技术提高模型的可解释性。 **方法：** 研究使用了MSLD和MSLD v2.0两个数据集进行训练和验证。通过应用迁移学习技术，冻结预训练CNN模型的初始层，并添加自定义层以适应最终特征用于猴痘检测任务，从而避免过拟合现象。评估模型性能时采用了准确性、精确度、召回率、F1分数及ROC等指标；同时利用Grad-CAM进行关键图像区域可视化。 **结果：** 在二分类数据集上，InceptionV3表现最佳，准确率达到95%；而在多分类数据集上，MobileNetV2表现出色，其准确率为93%。通过Grad-CAM成功地识别出了重要的图像特征。尽管大多数模型展现了较高的准确性，但某些模型仍然显示出过拟合的倾向，这从训练损失与验证损失之间的差异可以明显看出。 **结论：** 本研究强调了预训练CNN模型在猴痘诊断中的潜力以及XAI技术的价值。未来的工作应当解决数据集的局限性，引入多模态数据，并探索额外的可解释性方法以提高诊断可靠性和模型透明度。
URL
https://arxiv.org/abs/2507.15915
PDF
-
A novel language model for predicting serious adverse event results in clinical trials from their prospective registrations
2025-07-21 07:56:06Qixuan Hu, Xumou Zhang, Jinman Kim, Florence Bourgeois, Adam G. DunnAbstract
Objectives: With accurate estimates of expected safety results, clinical trials could be designed to avoid terminations and limit exposing participants to unnecessary risks. We evaluated methods for predicting serious adverse event (SAE) results in clinical trials using information only from their registrations prior to the trial. Material and Methods: We analysed 22,107 two-arm parallel interventional clinical trials from this http URL with structured summary results. Two prediction models were developed: a classifier predicting will experimental arm have higher SAE rates (area under the receiver operating characteristic curve; AUC) than control arm, and a regression model to predict the proportion of SAEs in control arms (root mean squared error; RMSE). A transfer learning approach using pretrained language models (e.g., ClinicalT5, BioBERT) was used for feature extraction, combined with downstream model for prediction. To maintain semantic representation in long trial texts exceeding localised language model input limits, a sliding window method was developed for embedding extraction. Results: The best model (ClinicalT5+Transformer+MLP) had 77.6% AUC predicting which trial arm has a higher proportion of patients with SAEs. When predicting proportion of participants experiencing SAE in the control arm, the same model achieved RMSE of 18.6%. The sliding window approach consistently outperformed methods without it. Across 12 classifiers, the average absolute AUC increase was 2.00%; across 12 regressors, the average absolute RMSE reduction was 1.58%. Discussion: Summary results data available at this http URL remains underutilised. The potential to estimate results of trials before they start is an opportunity to improve trial design and flag discrepancies between expected and reported safety results.
Abstract (translated)
目标：通过准确估算预期的安全结果，临床试验可以被设计得更加完善，避免终止，并限制参与者暴露于不必要的风险中。我们评估了使用临床试验注册信息（在试验开始之前）来预测严重不良事件（SAE）结果的方法。 材料与方法：我们分析了来自这个网址的22,107项两臂平行干预性临床试验的数据，这些数据包含结构化的总结结果。开发了两个预测模型：一个分类器用于预测实验组是否会有比对照组更高的严重不良事件率（受试者操作特性曲线下的面积AUC），以及一个回归模型来预测对照组中严重不良事件的比例（均方根误差RMSE）。采用预训练语言模型（如ClinicalT5、BioBERT）的迁移学习方法用于特征提取，结合下游模型进行预测。为了保持长篇试验文本中的语义表示，并超过本地化语言模型输入限制，开发了一种滑动窗口方法来进行嵌入抽取。 结果：最佳模型（ClinicalT5+Transformer+MLP）在预测哪一组有更高的严重不良事件患者比例时达到了77.6%的AUC。当预测对照组中经历严重不良事件的比例时，同一模型实现了18.6%的RMSE。滑动窗口方法在所有情况下都优于不使用该方法的方法。在12个分类器中，平均绝对AUC增加为2.00%；在12个回归器中，平均绝对RMSE减少了1.58%。 讨论：在这个网址上可用的总结结果数据仍然未得到充分利用。能够预测试验开始前的结果是一个改善试验设计的机会，并且可以标记预期与报告的安全性结果之间的差异。
URL
https://arxiv.org/abs/2507.22919
PDF
-
A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script
2025-07-20 22:35:08Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Henok Biadglign Ademtew, Hizkel Mitiku Alemayehu, Negasi Haile Abadi, Tadesse Destaw Belay, Seid Muhie YimamAbstract
Homophone normalization, where characters that have the same sound in a writing script are mapped to one character, is a pre-processing step applied in Amharic Natural Language Processing (NLP) literature. While this may improve performance reported by automatic metrics, it also results in models that are not able to understand different forms of writing in a single language. Further, there might be impacts in transfer learning, where models trained on normalized data do not generalize well to other languages. In this paper, we experiment with monolingual training and cross-lingual transfer to understand the impacts of normalization on languages that use the Ge'ez script. We then propose a post-inference intervention in which normalization is applied to model predictions instead of training data. With our simple scheme of post-inference normalization, we show that we can achieve an increase in BLEU score of up to 1.03 while preserving language features in training. Our work contributes to the broader discussion on technology-facilitated language change and calls for more language-aware interventions.
Abstract (translated)
同音字规范化，即将书写体系中发音相同但形式不同的字符映射为同一个字符，是阿姆哈拉语自然语言处理（NLP）文献中的一个预处理步骤。虽然这可能会提高自动评测指标报告的性能，但它也会导致模型无法理解单一语言中的不同写作风格。此外，在迁移学习方面，基于规范化数据训练出来的模型在其他语言上泛化能力较差。本文中，我们对单语种训练和跨语言转移进行了实验，以探讨规范化对使用 Geez 脚本的语言的影响。随后，我们提出了一种推理后干预措施，即在模型预测阶段进行规范化而不是在训练数据中进行。通过我们的简单的推理后规范化方案，我们在保持训练数据语言特征的同时展示了最高达 1.03 的 BLEU 分数提升。我们的研究有助于更广泛的关于技术促进的语言变化的讨论，并呼吁采取更多注重语言特性的干预措施。
URL
https://arxiv.org/abs/2507.15142
PDF
-
Quantizing Text-attributed Graphs for Semantic-Structural Integration
2025-07-20 09:18:02Jianyuan Bo, Hao Wu, Yuan FangAbstract
Text-attributed graphs (TAGs) have emerged as a powerful representation for modeling complex relationships across diverse domains. With the rise of large language models (LLMs), there is growing interest in leveraging their capabilities for graph learning. However, current approaches face significant challenges in embedding structural information into LLM-compatible formats, requiring either computationally expensive alignment mechanisms or manual graph verbalization techniques that often lose critical structural details. Moreover, these methods typically require labeled data from source domains for effective transfer learning, significantly constraining their adaptability. We propose STAG, a novel self-supervised framework that directly quantizes graph structural information into discrete tokens using a frozen codebook. Unlike traditional quantization approaches, our method employs soft assignment and KL divergence guided quantization to address the unique challenges of graph data, which lacks natural tokenization structures. Our framework enables both LLM-based and traditional learning approaches, supporting true zero-shot transfer learning without requiring labeled data even in the source domain. Extensive experiments demonstrate state-of-the-art performance across multiple node classification benchmarks while maintaining compatibility with different LLM architectures, offering an elegant solution to bridging graph learning with LLMs.
Abstract (translated)
基于属性的图（TAG）作为一种强大的表示方法，已被用于跨多个领域的复杂关系建模。随着大规模语言模型（LLMs）的发展，人们对利用其能力进行图学习的兴趣日益增加。然而，目前的方法在将结构信息嵌入到与LLM兼容的格式中面临重大挑战，这要么需要计算成本高昂的对齐机制，要么需要手动图转文本的技术，而这些技术往往会导致关键结构细节的丢失。此外，这些方法通常需要来自源域的标注数据以实现有效的迁移学习，从而大大限制了其适应性。 我们提出了STAG，一种新颖的自我监督框架，该框架直接利用冻结代码簿将图的结构信息量化为离散标记。与传统的量化方法不同，我们的方法采用软分配和KL散度引导的量化来解决图数据特有的挑战，即缺乏自然的分词结构。我们的框架同时支持基于LLM的方法和传统学习方法，并且能够在不依赖源域标注数据的情况下实现真正的零样本迁移学习。广泛的实验表明，STAG在多个节点分类基准测试中表现出色，同时与不同的LLM架构兼容，为图学习与大规模语言模型之间的桥梁提供了一种优雅的解决方案。
URL
https://arxiv.org/abs/2507.19526
PDF
-
Rethinking Suicidal Ideation Detection: A Trustworthy Annotation Framework and Cross-Lingual Model Evaluation
2025-07-19 16:54:36Amina Dzafic, Merve Kavut, Ulya BayramAbstract
Suicidal ideation detection is critical for real-time suicide prevention, yet its progress faces two under-explored challenges: limited language coverage and unreliable annotation practices. Most available datasets are in English, but even among these, high-quality, human-annotated data remains scarce. As a result, many studies rely on available pre-labeled datasets without examining their annotation process or label reliability. The lack of datasets in other languages further limits the global realization of suicide prevention via artificial intelligence (AI). In this study, we address one of these gaps by constructing a novel Turkish suicidal ideation corpus derived from social media posts and introducing a resource-efficient annotation framework involving three human annotators and two large language models (LLMs). We then address the remaining gaps by performing a bidirectional evaluation of label reliability and model consistency across this dataset and three popular English suicidal ideation detection datasets, using transfer learning through eight pre-trained sentiment and emotion classifiers. These transformers help assess annotation consistency and benchmark model performance against manually labeled data. Our findings underscore the need for more rigorous, language-inclusive approaches to annotation and evaluation in mental health natural language processing (NLP) while demonstrating the questionable performance of popular models with zero-shot transfer learning. We advocate for transparency in model training and dataset construction in mental health NLP, prioritizing data and model reliability.
Abstract (translated)
自杀念头的检测对于实时预防自杀至关重要，然而其进展面临着两个被忽视的挑战：语言覆盖范围有限以及标注实践不可靠。现有的可用数据集大多数是英文，即使在这些数据集中，高质量的人工标注数据仍然稀缺。因此，许多研究依赖于已有的预标注数据集，并未对其标注过程或标签可靠性进行审查。缺乏其他语言的数据集进一步限制了通过人工智能（AI）实现全球自杀预防的进展。 在这项研究中，我们填补了一个空白，构建了一个新颖的土耳其语自杀念头语料库，该语料库来源于社交媒体帖子，并引入了一种资源高效的标注框架，涉及三位人工标注员和两个大型语言模型（LLM）。然后，通过使用八种预训练的情感和情绪分类器进行迁移学习，在此数据集以及三个流行的英语自杀念头检测数据集中执行双向标签可靠性和模型一致性的评估。这些转换器帮助评估注释的一致性，并将模型性能与人工标记的数据相比较。 我们的研究结果强调了在精神健康自然语言处理（NLP）中需要采取更为严谨、多语种包容的方法来进行标注和评估，同时揭示了零样本迁移学习方法中的流行模型存在令人质疑的表现。我们倡导在精神健康NLP领域提高模型训练和数据集构建的透明度，并优先考虑数据和模型的可靠性。
URL
https://arxiv.org/abs/2507.14693
PDF
-
Depthwise-Dilated Convolutional Adapters for Medical Object Tracking and Segmentation Using the Segment Anything Model 2
2025-07-19 13:19:55Guoping Xu, Christopher Kabat, You ZhangAbstract
Recent advances in medical image segmentation have been driven by deep learning; however, most existing methods remain limited by modality-specific designs and exhibit poor adaptability to dynamic medical imaging scenarios. The Segment Anything Model 2 (SAM2) and its related variants, which introduce a streaming memory mechanism for real-time video segmentation, present new opportunities for prompt-based, generalizable solutions. Nevertheless, adapting these models to medical video scenarios typically requires large-scale datasets for retraining or transfer learning, leading to high computational costs and the risk of catastrophic forgetting. To address these challenges, we propose DD-SAM2, an efficient adaptation framework for SAM2 that incorporates a Depthwise-Dilated Adapter (DD-Adapter) to enhance multi-scale feature extraction with minimal parameter overhead. This design enables effective fine-tuning of SAM2 on medical videos with limited training data. Unlike existing adapter-based methods focused solely on static images, DD-SAM2 fully exploits SAM2's streaming memory for medical video object tracking and segmentation. Comprehensive evaluations on TrackRad2025 (tumor segmentation) and EchoNet-Dynamic (left ventricle tracking) datasets demonstrate superior performance, achieving Dice scores of 0.93 and 0.97, respectively. To the best of our knowledge, this work provides an initial attempt at systematically exploring adapter-based SAM2 fine-tuning for medical video segmentation and tracking. Code, datasets, and models will be publicly available at this https URL.
Abstract (translated)
最近的医学图像分割进展主要得益于深度学习技术；然而，大多数现有的方法仍然局限于特定模态的设计，并且在动态医疗成像场景中表现出较差的适应性。引入了流式内存机制以实现实时视频分割的Segment Anything Model 2（SAM2）及其相关变体，为基于提示、可推广性的解决方案带来了新的机遇。然而，将这些模型应用于医学视频场景通常需要大规模的数据集来进行重新训练或迁移学习，这会导致高昂的计算成本和灾难性遗忘的风险。 为了应对这一挑战，我们提出了DD-SAM2，这是一个针对SAM2的有效适应框架，它结合了深度可分离膨胀适配器（DD-Adapter），以增强多尺度特征提取，并且参数开销极小。这种设计使得在训练数据有限的情况下对医学视频进行有效的微调成为可能。与现有的仅关注静态图像的基于适配器的方法不同，DD-SAM2充分利用了SAM2的流式内存机制来进行医学视频中的对象跟踪和分割。 我们在TrackRad2025（肿瘤分割）和EchoNet-Dynamic（左心室追踪）数据集上进行了全面评估。结果表明，在这两个任务中，我们的模型分别达到了Dice分数为0.93和0.97的优异性能。 据我们所知，这项工作首次系统地探索了基于适配器的SAM2微调在医学视频分割与跟踪中的应用。代码、数据集以及模型将公开发布在这个链接：[此URL]。
URL
https://arxiv.org/abs/2507.14613
PDF
-
Towards a Proactive Autoscaling Framework for Data Stream Processing at the Edge using GRU and Transfer Learning
2025-07-19 12:47:50Eugene Armah, Linda Amoako BannningAbstract
Processing data at high speeds is becoming increasingly critical as digital economies generate enormous data. The current paradigms for timely data processing are edge computing and data stream processing (DSP). Edge computing places resources closer to where data is generated, while stream processing analyzes the unbounded high-speed data in motion. However, edge stream processing faces rapid workload fluctuations, complicating resource provisioning. Inadequate resource allocation leads to bottlenecks, whereas excess allocation results in wastage. Existing reactive methods, such as threshold-based policies and queuing theory scale only after performance degrades, potentially violating SLAs. Although reinforcement learning (RL) offers a proactive approach through agents that learn optimal runtime adaptation policies, it requires extensive simulation. Furthermore, predictive machine learning models face online distribution and concept drift that minimize their accuracy. We propose a three-step solution to the proactive edge stream processing autoscaling problem. Firstly, a GRU neural network forecasts the upstream load using real-world and synthetic DSP datasets. Secondly, a transfer learning framework integrates the predictive model into an online stream processing system using the DTW algorithm and joint distribution adaptation to handle the disparities between offline and online domains. Finally, a horizontal autoscaling module dynamically adjusts the degree of operator parallelism, based on predicted load while considering edge resource constraints. The lightweight GRU model for load predictions recorded up to 1.3\% SMAPE value on a real-world data set. It outperformed CNN, ARIMA, and Prophet on the SMAPE and RMSE evaluation metrics, with lower training time than the computationally intensive RL models.
Abstract (translated)
高速处理数据变得越来越重要，因为数字经济产生了大量数据。目前及时处理这些数据的方法主要包括边缘计算和数据流处理（DSP）。边缘计算将资源部署在靠近数据生成的地方，而数据流处理则分析未定义边界且速度极高的流动数据。然而，边缘流处理面临的工作负载波动迅速，给资源分配带来了复杂性。不适当的资源配置会导致瓶颈，而过度配置则会造成浪费。现有的反应式方法（例如基于阈值的策略和排队理论）仅在性能下降后才会进行扩展，这可能会违反服务水平协议（SLA）。虽然强化学习（RL）通过代理学习最优运行时自适应策略提供了前瞻性的方式，但这种方法需要大量的模拟工作。此外，在线分布和概念漂移会降低预测性机器学习模型的准确性。 我们提出了一种三步解决方案来解决前瞻性的边缘流处理自动扩展问题。首先，使用GRU神经网络基于现实世界及合成DSP数据集对未来负载进行预测。其次，采用转移学习框架将预测模型集成到在线流处理系统中，并利用DTW算法和联合分布适应性技术处理离线与在线领域之间的差异。最后，一个水平的自动扩展模块会根据预测的工作量动态调整操作员并行度，同时考虑到边缘资源限制。 轻量级的GRU模型在现实世界的数据集上实现了高达1.3% SMAPE值（对称平均百分比误差）的表现，并且在SMAPE和RMSE评估指标上优于CNN、ARIMA及Prophet模型。该模型还具有比计算密集型RL模型更短的训练时间。
URL
https://arxiv.org/abs/2507.14597
PDF
-
IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark
2025-07-19 02:53:01Zhe Cao, Jin Zhang, Ruiheng ZhangAbstract
Real-world infrared imagery presents unique challenges for vision-language models due to the scarcity of aligned text data and domain-specific characteristics. Although existing methods have advanced the field, their reliance on synthetic infrared images generated through style transfer from visible images, which limits their ability to capture the unique characteristics of the infrared modality. To address this, we propose IRGPT, the first multi-modal large language model for real-world infrared images, built upon a large-scale InfraRed-Text Dataset (IR-TD) comprising over 260K authentic image-text pairs. The proposed IR-TD dataset contains real infrared images paired with meticulously handcrafted texts, where the initial drafts originated from two complementary processes: (1) LLM-generated descriptions of visible images, and (2) rule-based descriptions of annotations. Furthermore, we introduce a bi-cross-modal curriculum transfer learning strategy that systematically transfers knowledge from visible to infrared domains by considering the difficulty scores of both infrared-visible and infrared-text. Evaluated on a benchmark of 9 tasks (e.g., recognition, grounding), IRGPT achieves state-of-the-art performance even compared with larger-scale models.
Abstract (translated)
现实世界中的红外图像对视觉语言模型提出了独特的挑战，由于缺乏与之匹配的文本数据以及特定领域的特性。尽管现有方法已经推动了这一领域的发展，但它们依赖于通过可见光图像进行风格迁移生成的人工合成红外图像，这限制了这些模型捕捉红外模态独特特征的能力。 为了解决这个问题，我们提出了IRGPT——第一个用于真实世界红外图像的多模态大型语言模型，并且建立在大规模的“红外-文本数据集”（IR-TD）之上。该数据集包含超过260,000对真实的红外图像和精心手工制作的文字描述。所提出的IR-TD数据集包括实际的红外图像与文本配对，其初始草稿来源于两种互补的过程：(1) 大型语言模型生成可见光图像的描述；(2) 基于规则的注释描述。 此外，我们还引入了一种双向跨模态课程迁移学习策略，该策略通过考虑红外和可见光域之间以及红外和文本之间的难度得分，系统地将知识从可见光领域转移到红外领域。在涵盖9项任务（例如识别、定位等）的标准基准测试上进行评估时，IRGPT即使与更大规模的模型相比也表现出最先进的性能。
URL
https://arxiv.org/abs/2507.14449
PDF
-
Language Models as Ontology Encoders
2025-07-18 19:26:16Hui Yang, Jiaoyan Chen, Yuan He, Yongsheng Gao, Ian HorrocksAbstract
OWL (Web Ontology Language) ontologies which are able to formally represent complex knowledge and support semantic reasoning have been widely adopted across various domains such as healthcare and bioinformatics. Recently, ontology embeddings have gained wide attention due to its potential to infer plausible new knowledge and approximate complex reasoning. However, existing methods face notable limitations: geometric model-based embeddings typically overlook valuable textual information, resulting in suboptimal performance, while the approaches that incorporate text, which are often based on language models, fail to preserve the logical structure. In this work, we propose a new ontology embedding method OnT, which tunes a Pretrained Language Model (PLM) via geometric modeling in a hyperbolic space for effectively incorporating textual labels and simultaneously preserving class hierarchies and other logical relationships of Description Logic EL. Extensive experiments on four real-world ontologies show that OnT consistently outperforms the baselines including the state-of-the-art across both tasks of prediction and inference of axioms. OnT also demonstrates strong potential in real-world applications, indicated by its robust transfer learning abilities and effectiveness in real cases of constructing a new ontology from SNOMED CT. Data and code are available at this https URL.
Abstract (translated)
OWL（Web本体语言）本体能够正式表示复杂知识并支持语义推理，在医疗保健和生物信息学等多个领域得到了广泛应用。最近，由于其潜在的推断合理的新知识和近似复杂推理的能力，本体嵌入技术引起了广泛关注。然而，现有的方法面临显著限制：基于几何模型的嵌入通常忽略了有价值的文本信息，导致性能不佳；而那些结合文本的方法往往基于语言模型，却无法保留逻辑结构。在本文中，我们提出了一种新的本体嵌入方法OnT，该方法通过在双曲空间中的几何建模调优预训练的语言模型（PLM），从而有效整合文本标签的同时保持描述逻辑EL的类层次和其他逻辑关系。四项真实世界本体上的广泛实验表明，在预测和公理推理两项任务中，OnT始终优于包括最先进在内的基线方法。此外，OnT还展示了在现实应用中的巨大潜力，这体现在其强大的迁移学习能力和从SNOMED CT构建新本体的实际案例的有效性上。数据和代码可在该URL获取：[此链接]（请将[此链接]替换为实际的URL）。
URL
https://arxiv.org/abs/2507.14334
PDF
-
Physics-informed transfer learning for SHM via feature selection
2025-07-18 14:28:52J. Poole, P. Gardner, A. J. Hughes, N. Dervilis, R. S. Mills, T. A. Dardeno, K. WordenAbstract
Data used for training structural health monitoring (SHM) systems are expensive and often impractical to obtain, particularly labelled data. Population-based SHM presents a potential solution to this issue by considering the available data across a population of structures. However, differences between structures will mean the training and testing distributions will differ; thus, conventional machine learning methods cannot be expected to generalise between structures. To address this issue, transfer learning (TL), can be used to leverage information across related domains. An important consideration is that the lack of labels in the target domain limits data-based metrics to quantifying the discrepancy between the marginal distributions. Thus, a prerequisite for the application of typical unsupervised TL methods is to identify suitable source structures (domains), and a set of features, for which the conditional distributions are related to the target structure. Generally, the selection of domains and features is reliant on domain expertise; however, for complex mechanisms, such as the influence of damage on the dynamic response of a structure, this task is not trivial. In this paper, knowledge of physics is leveraged to select more similar features, the modal assurance criterion (MAC) is used to quantify the correspondence between the modes of healthy structures. The MAC is shown to have high correspondence with a supervised metric that measures joint-distribution similarity, which is the primary indicator of whether a classifier will generalise between domains. The MAC is proposed as a measure for selecting a set of features that behave consistently across domains when subjected to damage, i.e. features with invariance in the conditional distributions. This approach is demonstrated on numerical and experimental case studies to verify its effectiveness in various applications.
Abstract (translated)
用于训练结构健康监测（SHM）系统的数据既昂贵又难以获取，尤其是带有标签的数据。基于群体的SHM为解决这一问题提供了潜在方案，通过考虑一组建筑物可用的数据来实现。然而，不同建筑之间的差异意味着训练和测试分布将有所不同；因此，传统的机器学习方法无法期望在不同的建筑之间泛化。为了应对这个问题，可以使用迁移学习（TL）来利用相关领域的信息。重要的一点是，在目标领域中缺乏标签限制了数据驱动的度量标准只能量化边际分布间的差异。因此，典型无监督TL方法的应用前提是要识别合适的源结构（领域），以及一组条件分布与目标结构相关的特征。通常，域和特征的选择依赖于专业知识；然而，对于复杂的机制，例如损伤对建筑动态响应的影响，这一任务并不简单。 本文利用物理学知识来选择更具相似性的特征，并使用模态保证准则（MAC）量化健康建筑物模式之间的对应关系。研究表明，MAC与衡量联合分布相似度的监督度量具有高度一致性，这是判断分类器是否能在不同领域之间泛化的主要指标。提出将MAC作为一种度量标准，用于选取在受到损伤时跨域行为一致的一组特征，即条件分布不变性的特征。通过数值和实验案例研究证明了该方法在各种应用中的有效性。
URL
https://arxiv.org/abs/2507.19519
PDF
-
Improving Diagnostic Accuracy of Pigmented Skin Lesions With CNNs: an Application on the DermaMNIST Dataset
2025-07-17 10:00:07Nerma Kadric, Amila Akagic, Medina KapoAbstract
Pigmented skin lesions represent localized areas of increased melanin and can indicate serious conditions like melanoma, a major contributor to skin cancer mortality. The MedMNIST v2 dataset, inspired by MNIST, was recently introduced to advance research in biomedical imaging and includes DermaMNIST, a dataset for classifying pigmented lesions based on the HAM10000 dataset. This study assesses ResNet-50 and EfficientNetV2L models for multi-class classification using DermaMNIST, employing transfer learning and various layer configurations. One configuration achieves results that match or surpass existing methods. This study suggests that convolutional neural networks (CNNs) can drive progress in biomedical image analysis, significantly enhancing diagnostic accuracy.
Abstract (translated)
色素皮肤病变代表局部黑色素增多的区域，可能是包括黑素瘤在内的严重疾病的标志。黑素瘤是导致皮肤癌死亡的主要因素之一。最近，受MNIST数据集启发而创建的MedMNIST v2数据集被引入以推进生物医学影像研究，并包含基于HAM10000数据集构建的用于色素病变分类的数据子集DermaMNIST。 本研究评估了ResNet-50和EfficientNetV2L模型在使用DermaMNIST进行多类分类任务中的表现，采用了迁移学习以及多种层配置。其中一种配置达到了与现有方法相匹配或超越的结果。这项研究表明，卷积神经网络（CNN）可以推动生物医学图像分析的进步，并显著提高诊断准确性。
URL
https://arxiv.org/abs/2507.12961
PDF
-
Improving physics-informed neural network extrapolation via transfer learning and adaptive activation functions
2025-07-16 22:19:53Athanasios Papastathopoulos-Katsaros, Alexandra Stavrianidi, Zhandong LiuAbstract
Physics-Informed Neural Networks (PINNs) are deep learning models that incorporate the governing physical laws of a system into the learning process, making them well-suited for solving complex scientific and engineering problems. Recently, PINNs have gained widespread attention as a powerful framework for combining physical principles with data-driven modeling to improve prediction accuracy. Despite their successes, however, PINNs often exhibit poor extrapolation performance outside the training domain and are highly sensitive to the choice of activation functions (AFs). In this paper, we introduce a transfer learning (TL) method to improve the extrapolation capability of PINNs. Our approach applies transfer learning (TL) within an extended training domain, using only a small number of carefully selected collocation points. Additionally, we propose an adaptive AF that takes the form of a linear combination of standard AFs, which improves both the robustness and accuracy of the model. Through a series of experiments, we demonstrate that our method achieves an average of 40% reduction in relative L2 error and an average of 50% reduction in mean absolute error in the extrapolation domain, all without a significant increase in computational cost. The code is available at this https URL .
Abstract (translated)
物理信息神经网络（Physics-Informed Neural Networks，PINNs）是一种深度学习模型，它将系统的支配物理定律融入到学习过程中，因此非常适合解决复杂的科学和工程问题。近年来，由于能够结合物理原理与数据驱动建模以提高预测准确性，PINNs受到了广泛的关注。尽管取得了成功，但PINNs在训练域之外的外推性能较差，并且对激活函数（AFs）的选择非常敏感。在这篇论文中，我们引入了一种迁移学习（TL）方法来改进PINNs的外推能力。我们的方法在扩展的训练域内应用迁移学习，仅使用少量精心挑选的配置点(collocation points)。此外，我们还提出了一种自适应激活函数，它由标准激活函数以线性组合的形式构成，从而提高了模型的鲁棒性和准确性。通过一系列实验，我们展示了该方法在外推领域中平均减少了40%的相对L2误差和50%的平均绝对误差，而无需显著增加计算成本。代码可在[此处](https://this https URL)获取。
URL
https://arxiv.org/abs/2507.12659
PDF
-
Best Practices for Large-Scale, Pixel-Wise Crop Mapping and Transfer Learning Workflows
2025-07-16 19:18:25Judy Long, Tao Liu, Sean Alexander Woznicki, Miljana Markovi\'c, Oskar Marko, Molly SearsAbstract
Crop mapping involves identifying and classifying crop types using spatial data, primarily derived from remote sensing imagery. This study presents the first comprehensive review of large-scale, pixel-wise crop mapping workflows, encompassing both conventional supervised methods and emerging transfer learning approaches. To identify the optimal supervised crop mapping workflows, we conducted systematic experiments, comparing six widely adopted satellite image-based preprocessing methods, alongside eleven supervised pixel-wise classification models. Additionally, we assessed the synergistic impact of varied training sample sizes and variable combinations. Moreover, we identified optimal transfer learning techniques for different magnitudes of domain shift. The evaluation of best methods was conducted across five diverse agricultural sites. Landsat 8 served as the primary satellite data source. Labels come from CDL trusted pixels and field surveys. Our findings reveal three key insights. First, fine-scale interval preprocessing paired with Transformer models consistently delivered optimal performance for both supervised and transferable workflows. RF offered rapid training and competitive performance in conventional supervised learning and direct transfer to similar domains. Second, transfer learning techniques enhanced workflow adaptability, with UDA being effective for homogeneous crop classes while fine-tuning remains robust across diverse scenarios. Finally, workflow choice depends heavily on the availability of labeled samples. With a sufficient sample size, supervised training typically delivers more accurate and generalizable results. Below a certain threshold, transfer learning that matches the level of domain shift is a viable alternative to achieve crop mapping. Repository: Best-Practices-for-Large-Scale-Pixel-Wise-Crop-Mapping-and-Transfer-Learning-Workflows
Abstract (translated)
作物制图涉及使用空间数据（主要来自遥感图像）识别和分类农作物类型。本研究首次全面回顾了大规模像素级作物制图工作流程，涵盖了传统的监督方法和新兴的迁移学习方法。为了确定最佳的监督作物制图工作流程，我们进行了系统实验，比较了六种常用的卫星影像预处理方法以及十一款监督下的像素级分类模型。此外，还评估了不同训练样本大小及变量组合所带来的协同效应。同时，我们也识别出适用于不同程度领域偏移的最佳迁移学习技术。最佳方法的评估在五个不同的农业地点进行，并使用Landsat 8作为主要卫星数据源。标签来自CDL可信像素和实地调查。 我们的研究揭示了三个关键见解： 1. 精细化间隔预处理与Transformer模型相结合，始终为监督学习和可迁移的工作流程提供最优性能。 2. RF（随机森林）提供了快速的训练速度，并且在传统的监督学习以及直接转移到类似领域时具有竞争力的表现。 3. 迁移学习技术增强了工作流的适应性：UDA（无标签数据增强）对于同质化的农作物类别有效，而微调则适用于各种场景。 最后一点是，工作流程的选择高度依赖于标记样本的可用性。在有足够的样本量的情况下，监督训练通常会产生更准确且泛化能力更强的结果。然而，在低于某一阈值时，与领域偏移水平相匹配的迁移学习成为实现作物制图的一种可行替代方案。 该研究的相关代码和资源已托管在GitHub仓库中：Best-Practices-for-Large-Scale-Pixel-Wise-Crop-Mapping-and-Transfer-Learning-Workflows。
URL
https://arxiv.org/abs/2507.12590
PDF
-
Hybrid Ensemble Approaches: Optimal Deep Feature Fusion and Hyperparameter-Tuned Classifier Ensembling for Enhanced Brain Tumor Classification
2025-07-16 12:22:11Zahid Ullah, Dragan Pamucar, Jihie KimAbstract
Magnetic Resonance Imaging (MRI) is widely recognized as the most reliable tool for detecting tumors due to its capability to produce detailed images that reveal their presence. However, the accuracy of diagnosis can be compromised when human specialists evaluate these images. Factors such as fatigue, limited expertise, and insufficient image detail can lead to errors. For example, small tumors might go unnoticed, or overlap with healthy brain regions could result in misidentification. To address these challenges and enhance diagnostic precision, this study proposes a novel double ensembling framework, consisting of ensembled pre-trained deep learning (DL) models for feature extraction and ensembled fine-tuned hyperparameter machine learning (ML) models to efficiently classify brain tumors. Specifically, our method includes extensive preprocessing and augmentation, transfer learning concepts by utilizing various pre-trained deep convolutional neural networks and vision transformer networks to extract deep features from brain MRI, and fine-tune hyperparameters of ML classifiers. Our experiments utilized three different publicly available Kaggle MRI brain tumor datasets to evaluate the pre-trained DL feature extractor models, ML classifiers, and the effectiveness of an ensemble of deep features along with an ensemble of ML classifiers for brain tumor classification. Our results indicate that the proposed feature fusion and classifier fusion improve upon the state of the art, with hyperparameter fine-tuning providing a significant enhancement over the ensemble method. Additionally, we present an ablation study to illustrate how each component contributes to accurate brain tumor classification.
Abstract (translated)
磁共振成像（MRI）因其能够生成详细图像以揭示肿瘤的存在，而被广泛认为是检测肿瘤最可靠的工具。然而，当人类专家评估这些图像时，诊断的准确性可能会受到诸如疲劳、专业知识有限以及图像细节不足等因素的影响，导致误诊。例如，小肿瘤可能未被发现，或者与健康脑区重叠可能导致错误识别。 为了解决这些问题并提高诊断精度，本研究提出了一种新型双集合框架，包括用于特征提取的预训练深度学习（DL）模型的集合和用于高效分类脑肿瘤的微调超参数机器学习（ML）模型的集合。具体而言，我们的方法涵盖了广泛的预处理和增强、利用各种预训练的深度卷积神经网络和视觉变换器网络来从脑MRI中提取深层特征，并对ML分类器进行超参数微调。 在实验中，我们使用了三个不同的公开可用Kaggle MRI脑肿瘤数据集来评估预训练DL特征提取模型、ML分类器以及深层特征集合与ML分类器集合对于脑肿瘤分类的有效性。我们的结果显示，提出的特征融合和分类器融合改进了现有技术水平，并且超参数微调提供了比集合方法更显著的提升。此外，我们还展示了消融研究，以说明每个组件如何有助于准确地进行脑肿瘤分类。
URL
https://arxiv.org/abs/2507.12177
PDF
-
Physics-Informed Transfer Learning for Data-Driven Sound Source Reconstruction in Near-Field Acoustic Holography
2025-07-15 08:03:05Xinmeng Luan, Mirco Pezzoli, Fabio Antonacci, Augusto SartiAbstract
We propose a transfer learning framework for sound source reconstruction in Near-field Acoustic Holography (NAH), which adapts a well-trained data-driven model from one type of sound source to another using a physics-informed procedure. The framework comprises two stages: (1) supervised pre-training of a complex-valued convolutional neural network (CV-CNN) on a large dataset, and (2) purely physics-informed fine-tuning on a single data sample based on the Kirchhoff-Helmholtz integral. This method follows the principles of transfer learning by enabling generalization across different datasets through physics-informed adaptation. The effectiveness of the approach is validated by transferring a pre-trained model from a rectangular plate dataset to a violin top plate dataset, where it shows improved reconstruction accuracy compared to the pre-trained model and delivers performance comparable to that of Compressive-Equivalent Source Method (C-ESM). Furthermore, for successful modes, the fine-tuned model outperforms both the pre-trained model and C-ESM in accuracy.
Abstract (translated)
我们提出了一种用于近场声全息（NAH）中声音源重建的迁移学习框架，该框架通过物理信息引导的过程将一个类型的声音源上训练好的数据驱动模型适应到另一个类型的声音源。此框架包含两个阶段：(1) 在大型数据集上对复数卷积神经网络（CV-CNN）进行监督预训练；(2) 基于Kirchhoff-Helmholtz积分，在单个数据样本上仅使用物理信息进行微调。这种方法遵循迁移学习的原则，通过基于物理知识的适应性实现了跨不同数据集的泛化能力。该方法的有效性通过将一个预先在矩形板数据集上训练好的模型迁移到小提琴面板数据集中得到了验证，在这种情况下，与预训练模型相比，重建精度提高了，并且其性能可媲美压缩等效源法（C-ESM）。此外，在成功模式下，微调后的模型不仅比预训练的模型更准确，也超过了C-ESM的准确性。
URL
https://arxiv.org/abs/2507.11070
PDF
-
Supporting SEN'{C}OTEN Language Documentation Efforts with Automatic Speech Recognition
2025-07-14 21:44:35Mengzhe Geng, Patrick Littell, Aidan Pine, PEN\'A\'C, Marc Tessier, Roland KuhnAbstract
The SENÄOTEN language, spoken on the Saanich peninsula of southern Vancouver Island, is in the midst of vigorous language revitalization efforts to turn the tide of language loss as a result of colonial language policies. To support these on-the-ground efforts, the community is turning to digital technology. Automatic Speech Recognition (ASR) technology holds great promise for accelerating language documentation and the creation of educational resources. However, developing ASR systems for SENÄOTEN is challenging due to limited data and significant vocabulary variation from its polysynthetic structure and stress-driven metathesis. To address these challenges, we propose an ASR-driven documentation pipeline that leverages augmented speech data from a text-to-speech (TTS) system and cross-lingual transfer learning with Speech Foundation Models (SFMs). An n-gram language model is also incorporated via shallow fusion or n-best restoring to maximize the use of available data. Experiments on the SENÄOTEN dataset show a word error rate (WER) of 19.34% and a character error rate (CER) of 5.09% on the test set with a 57.02% out-of-vocabulary (OOV) rate. After filtering minor cedilla-related errors, WER improves to 14.32% (26.48% on unseen words) and CER to 3.45%, demonstrating the potential of our ASR-driven pipeline to support SENÄOTEN language documentation.
Abstract (translated)
SENĆOŦEN语言，一种在温哥华岛南部的萨尼奇半岛上使用的语言，正面临着由于殖民时期的语言政策导致的语言流失问题。为了扭转这一趋势，该社区正在积极进行语言复兴工作，并且开始借助数字技术来支持这些努力。自动语音识别（ASR）技术在这方面具有很大的潜力，可以加速语言记录和教育资源的创建。 然而，为SENĆOŦEN开发ASR系统面临诸多挑战，包括数据有限以及由于其多合成结构和重音驱动的元音互换导致词汇变化显著的问题。为了应对这些挑战，我们提出了一个基于ASR的语言文档流水线方法，这种方法利用了来自文本到语音（TTS）系统的增强语音数据，并结合跨语言迁移学习与语音基础模型（SFM）。此外，通过浅层融合或n-best恢复技术，我们将n-gram语言模型融入其中以最大限度地利用现有数据。 在SENĆOŦEN的数据集上进行的实验显示，在测试集中，词错误率为19.34%，字符错误率为5.09%。当考虑到26.48%未登录词汇（OOV）的情况下，单词错误率降至14.32%，字符错误率降至3.45%，这表明了我们提出的基于ASR的流水线方法在支持SENĆOŦEN语言记录方面具有巨大的潜力。
URL
https://arxiv.org/abs/2507.10827
PDF
