Transfer_Learning
-
Offensive Language Detection on Social Media Using XLNet
2025-06-26 22:37:35Reem Alothman, Hafida Benhidour, Said KerracheAbstract
The widespread use of text-based communication on social media-through chats, comments, and microblogs-has improved user interaction but has also led to an increase in offensive content, including hate speech, racism, and other forms of abuse. Due to the enormous volume of user-generated content, manual moderation is impractical, which creates a need for automated systems that can detect offensive language. Deep learning models, particularly those using transfer learning, have demonstrated significant success in understanding natural language through large-scale pretraining. In this study, we propose an automatic offensive language detection model based on XLNet, a generalized autoregressive pretraining method, and compare its performance with BERT (Bidirectional Encoder Representations from Transformers), which is a widely used baseline in natural language processing (NLP). Both models are evaluated using the Offensive Language Identification Dataset (OLID), a benchmark Twitter dataset that includes hierarchical annotations. Our experimental results show that XLNet outperforms BERT in detecting offensive content and in categorizing the types of offenses, while BERT performs slightly better in identifying the targets of the offenses. Additionally, we find that oversampling and undersampling strategies are effective in addressing class imbalance and improving classification performance. These findings highlight the potential of transfer learning and XLNet-based architectures to create robust systems for detecting offensive language on social media platforms.
Abstract (translated)
社交媒体上基于文本的通信——通过聊天、评论和微博等形式——虽然增强了用户间的互动，但也导致了包括仇恨言论、种族主义和其他形式滥用在内的攻击性内容增多。由于用户生成的内容数量庞大，人工审核变得不切实际，因此需要能够自动检测攻击性语言的系统。深度学习模型，特别是那些使用迁移学习的方法，在通过大规模预训练来理解自然语言方面表现出显著的成功。在这项研究中，我们提出了一种基于XLNet（一种广义自回归预训练方法）的自动检测攻击性语言的模型，并将其性能与广泛应用于自然语言处理（NLP）领域的基准BERT模型进行了比较。这两种模型均使用了带有层次化注释的 Offensive Language Identification Dataset (OLID) 进行评估，这是一个基准推特数据集。 实验结果显示，XLNet在检测和分类攻击内容方面优于BERT，而BERT在识别攻击目标时略胜一筹。此外，我们发现过采样和欠采样策略对于解决类别不平衡问题并提高分类性能有效。这些发现强调了迁移学习以及基于XLNet架构的潜力，在创建能够探测社交媒体平台上攻击性语言的强大系统中具有重要意义。
URL
https://arxiv.org/abs/2506.21795
PDF
-
Benchmarking Deep Learning and Vision Foundation Models for Atypical vs. Normal Mitosis Classification with Cross-Dataset Evaluation
2025-06-26 16:30:42Sweta Banerjee, Viktoria Weiss, Taryn A. Donovan, Rutger A. Fick, Thomas Conrad, Jonas Ammeling, Nils Porsche, Robert Klopfleisch, Christopher Kaltenecker, Katharina Breininger, Marc Aubreville, Christof A. BertramAbstract
Atypical mitoses mark a deviation in the cell division process that can be an independent prognostically relevant marker for tumor malignancy. However, their identification remains challenging due to low prevalence, at times subtle morphological differences from normal mitoses, low inter-rater agreement among pathologists, and class imbalance in datasets. Building on the Atypical Mitosis dataset for Breast Cancer (AMi-Br), this study presents a comprehensive benchmark comparing deep learning approaches for automated atypical mitotic figure (AMF) classification, including baseline models, foundation models with linear probing, and foundation models fine-tuned with low-rank adaptation (LoRA). For rigorous evaluation, we further introduce two new hold-out AMF datasets - AtNorM-Br, a dataset of mitoses from the The TCGA breast cancer cohort, and AtNorM-MD, a multi-domain dataset of mitoses from the MIDOG++ training set. We found average balanced accuracy values of up to 0.8135, 0.7696, and 0.7705 on the in-domain AMi-Br and the out-of-domain AtNorm-Br and AtNorM-MD datasets, respectively, with the results being particularly good for LoRA-based adaptation of the Virchow-line of foundation models. Our work shows that atypical mitosis classification, while being a challenging problem, can be effectively addressed through the use of recent advances in transfer learning and model fine-tuning techniques. We make available all code and data used in this paper in this github repository: this https URL.
Abstract (translated)
非典型有丝分裂标志着细胞分裂过程中的偏差，这可能是肿瘤恶性的一个独立的预后相关标志。然而，由于其在样本中出现频率低、与正常有丝分裂形态学上的差异细微、病理学家之间的一致性较低以及数据集中的类别不平衡等问题，它们的识别仍然具有挑战性。基于乳腺癌非典型有丝分裂数据集（AMi-Br），本研究提出了一个全面的基准测试，比较用于自动分类非典型有丝分裂图（AMF）的深度学习方法，包括基线模型、通过线性探测的基础模型和采用低秩适应（LoRA）进行微调的基础模型。为了严格评估这些方法的效果，我们进一步引入了两个新的未使用数据集——AtNorM-Br，这是一个来自TCGA乳腺癌队列的有丝分裂数据集；以及AtNorM-MD，这是MIDOG++训练集中多领域有丝分裂的一个数据集。我们在内部域AMi-Br和外部域AtNorm-Br及AtNorM-MD数据集上得到了最高平均平衡精度值分别为0.8135、0.7696和0.7705，其中基于LoRA的Virchow基础模型线系适应的结果尤为出色。我们的工作表明，尽管非典型有丝分裂分类是一个具有挑战性的问题，但通过利用最近在迁移学习和模型微调技术方面的进展可以有效地解决这一问题。我们在这个GitHub仓库中提供了用于本研究的所有代码和数据：[提供链接]（请将URL替换为实际的链接地址）。
URL
https://arxiv.org/abs/2506.21444
PDF
-
Comparative Analysis of Deep Learning Models for Crop Disease Detection: A Transfer Learning Approach
2025-06-25 11:04:33Saundarya Subramaniam, Shalini Majumdar, Shantanu Nadar, Kaustubh KulkarniAbstract
This research presents the development of an Artificial Intelligence (AI) - driven crop disease detection system designed to assist farmers in rural areas with limited resources. We aim to compare different deep learning models for a comparative analysis, focusing on their efficacy in transfer learning. By leveraging deep learning models, including EfficientNet, ResNet101, MobileNetV2, and our custom CNN, which achieved a validation accuracy of 95.76%, the system effectively classifies plant diseases. This research demonstrates the potential of transfer learning in reshaping agricultural practices, improving crop health management, and supporting sustainable farming in rural environments.
Abstract (translated)
这项研究开发了一个由人工智能（AI）驱动的农作物病害检测系统，旨在帮助资源有限的农村地区农民。我们旨在比较不同的深度学习模型进行对比分析，重点关注它们在迁移学习中的有效性。通过利用包括EfficientNet、ResNet101、MobileNetV2和我们的自定义CNN在内的深度学习模型（后者实现了95.76%的有效验证准确率），该系统能够有效地对植物病害进行分类。本研究展示了迁移学习在重塑农业实践、改善作物健康管理以及支持农村环境可持续耕作方面的潜力。
URL
https://arxiv.org/abs/2506.20323
PDF
-
How Can Multimodal Remote Sensing Datasets Transform Classification via SpatialNet-ViT?
2025-06-25 10:50:33Gautam Siddharth Kashyap, Manaswi Kulahara, Nipun Joshi, Usman NaseemAbstract
Remote sensing datasets offer significant promise for tackling key classification tasks such as land-use categorization, object presence detection, and rural/urban classification. However, many existing studies tend to focus on narrow tasks or datasets, which limits their ability to generalize across various remote sensing classification challenges. To overcome this, we propose a novel model, SpatialNet-ViT, leveraging the power of Vision Transformers (ViTs) and Multi-Task Learning (MTL). This integrated approach combines spatial awareness with contextual understanding, improving both classification accuracy and scalability. Additionally, techniques like data augmentation, transfer learning, and multi-task learning are employed to enhance model robustness and its ability to generalize across diverse datasets
Abstract (translated)
遥感数据集在处理诸如土地利用分类、目标存在检测和城乡分类等关键任务方面具有巨大潜力。然而，许多现有的研究往往集中于狭窄的任务或特定的数据集上，这限制了它们应对不同遥感分类挑战的能力。为了解决这一问题，我们提出了一种新的模型SpatialNet-ViT，该模型结合了视觉变换器（Vision Transformers, ViT）和多任务学习（Multi-Task Learning, MTL）的优势。这种集成方法将空间感知与上下文理解相结合，提高了分类准确性和可扩展性。此外，通过应用数据增强、迁移学习和多任务学习等技术，增强了模型的鲁棒性和跨多种数据集泛化的能力。
URL
https://arxiv.org/abs/2506.22501
PDF
-
FundaQ-8: A Clinically-Inspired Scoring Framework for Automated Fundus Image Quality Assessment
2025-06-25 10:28:53Lee Qi Zun, Oscar Wong Jin Hao, Nor Anita Binti Che Omar, Zalifa Zakiah Binti Asnir, Mohamad Sabri bin Sinal Zainal, Goh Man FyeAbstract
Automated fundus image quality assessment (FIQA) remains a challenge due to variations in image acquisition and subjective expert evaluations. We introduce FundaQ-8, a novel expert-validated framework for systematically assessing fundus image quality using eight critical parameters, including field coverage, anatomical visibility, illumination, and image artifacts. Using FundaQ-8 as a structured scoring reference, we develop a ResNet18-based regression model to predict continuous quality scores in the 0 to 1 range. The model is trained on 1800 fundus images from real-world clinical sources and Kaggle datasets, using transfer learning, mean squared error optimization, and standardized preprocessing. Validation against the EyeQ dataset and statistical analyses confirm the framework's reliability and clinical interpretability. Incorporating FundaQ-8 into deep learning models for diabetic retinopathy grading also improves diagnostic robustness, highlighting the value of quality-aware training in real-world screening applications.
Abstract (translated)
自动化视网膜图像质量评估（FIQA）由于成像获取的差异和主观专家评价的存在，仍然是一项挑战。我们引入了FundaQ-8，这是一个经过专家验证的新框架，用于系统地评估视网膜图像的质量，采用了八个关键参数，包括视野覆盖范围、解剖结构可见度、照明情况以及图像伪影等。 利用FundaQ-8作为结构化的评分参考，我们开发了一个基于ResNet18的回归模型来预测0到1范围内连续的质量分数。该模型在来自实际临床来源和Kaggle数据集的1800张视网膜图像上进行训练，采用了迁移学习、均方误差优化以及标准化预处理技术。 通过对EyeQ数据集进行验证并结合统计分析，证实了此框架具备可靠性和临床解释性。将FundaQ-8融入用于糖尿病视网膜病变分级的深度学习模型中，也进一步提高了诊断的稳健性，这突显出在实际筛查应用中质量感知训练的价值。
URL
https://arxiv.org/abs/2506.20303
PDF
-
Enhancing Sports Strategy with Video Analytics and Data Mining: Automated Video-Based Analytics Framework for Tennis Doubles
Abstract
We present a comprehensive video-based analytics framework for tennis doubles that addresses the lack of automated analysis tools for this strategically complex sport. Our approach introduces a standardised annotation methodology encompassing player positioning, shot types, court formations, and match outcomes, coupled with a specialised annotation tool designed to meet the unique requirements of tennis video labelling. The framework integrates advanced machine learning techniques including GroundingDINO for precise player localisation through natural language grounding and YOLO-Pose for robust pose estimation. This combination significantly reduces manual annotation effort whilst improving data consistency and quality. We evaluate our approach on doubles tennis match data and demonstrate that CNN-based models with transfer learning substantially outperform pose-based methods for predicting shot types, player positioning, and formations. The CNN models effectively capture complex visual and contextual features essential for doubles tennis analysis. Our integrated system bridges advanced analytical capabilities with the strategic complexities of tennis doubles, providing a foundation for automated tactical analysis, performance evaluation, and strategic modelling in professional tennis.
Abstract (translated)
我们提出了一种全面的基于视频的网球双打分析框架，旨在解决该战略复杂的运动缺乏自动化分析工具的问题。我们的方法引入了标准化注释方法，涵盖了球员位置、击球类型、场地布局和比赛结果，并结合了一个专门设计以满足网球视频标注独特需求的注释工具。 该框架集成了先进的机器学习技术，包括通过自然语言定位实现精确球员定位的GroundingDINO以及用于稳健姿态估计的YOLO-Pose。这种组合显著减少了手动注释的工作量，同时提高了数据的一致性和质量。我们在双打网球比赛的数据上评估了我们的方法，并展示了基于卷积神经网络（CNN）并通过迁移学习的方法在预测击球类型、球员位置和场地布局方面远胜过姿态基线方法。 这些CNN模型能够有效捕捉对双打网球分析至关重要的复杂视觉和上下文特征。我们集成的系统将高级分析能力与网球双打的战略复杂性相结合，为职业网球中的战术分析、性能评估和战略建模提供了基础。
URL
https://arxiv.org/abs/2507.02906
PDF
-
From High-SNR Radar Signal to ECG: A Transfer Learning Model with Cardio-Focusing Algorithm for Scenarios with Limited Data
2025-06-24 06:38:34Yuanyuan Zhang, Haocheng Zhao, Sijie Xiong, Rui Yang, Eng Gee Lim, Yutao YueAbstract
Electrocardiogram (ECG), as a crucial find-grained cardiac feature, has been successfully recovered from radar signals in the literature, but the performance heavily relies on the high-quality radar signal and numerous radar-ECG pairs for training, restricting the applications in new scenarios due to data scarcity. Therefore, this work will focus on radar-based ECG recovery in new scenarios with limited data and propose a cardio-focusing and -tracking (CFT) algorithm to precisely track the cardiac location to ensure an efficient acquisition of high-quality radar signals. Furthermore, a transfer learning model (RFcardi) is proposed to extract cardio-related information from the radar signal without ECG ground truth based on the intrinsic sparsity of cardiac features, and only a few synchronous radar-ECG pairs are required to fine-tune the pre-trained model for the ECG recovery. The experimental results reveal that the proposed CFT can dynamically identify the cardiac location, and the RFcardi model can effectively generate faithful ECG recoveries after using a small number of radar-ECG pairs for training. The code and dataset are available after the publication.
Abstract (translated)
心电图（ECG）作为重要的心脏特征之一，在文献中已经成功地从雷达信号中恢复出来，但其性能严重依赖于高质量的雷达信号和大量的雷达-ECG配对数据进行训练。由于新场景下数据稀缺，限制了应用范围。因此，本工作将专注于在数据有限的新应用场景中的基于雷达的心电图恢复，并提出了一种心脏聚焦与跟踪（Cardio-Focusing and -Tracking, CFT）算法，以精准地追踪心脏位置，确保高效获取高质量的雷达信号。 此外，我们还提出了一个迁移学习模型（RFcardi），该模型可以在没有心电图真实值的情况下从雷达信号中提取心脏相关的信息，并利用心脏特征内在的稀疏性进行操作。仅需少量同步的雷达-ECG配对数据即可微调预训练模型以实现心电图恢复。 实验结果表明，所提出的CFT算法能够动态识别心脏位置，而RFcardi模型能够在使用少量雷达-ECG配对数据训练后有效生成准确的心电图恢复。发布之后代码和数据集将对外提供。
URL
https://arxiv.org/abs/2506.19358
PDF
-
Sensing Cardiac Health Across Scenarios and Devices: A Multi-Modal Foundation Model Pretrained on Heterogeneous Data from 1.7 Million Individuals
2025-06-23 20:58:12Xiao Gu, Wei Tang, Jinpei Han, Veer Sangha, Fenglin Liu, Shreyank N Gowda, Antonio H. Ribeiro, Patrick Schwab, Kim Branson, Lei Clifton, Antonio Luiz P. Ribeiro, Zhangdaihong Liu, David A. CliftonAbstract
Cardiac biosignals, such as electrocardiograms (ECG) and photoplethysmograms (PPG), are of paramount importance for the diagnosis, prevention, and management of cardiovascular diseases, and have been extensively used in a variety of clinical tasks. Conventional deep learning approaches for analyzing these signals typically rely on homogeneous datasets and static bespoke models, limiting their robustness and generalizability across diverse clinical settings and acquisition protocols. In this study, we present a cardiac sensing foundation model (CSFM) that leverages advanced transformer architectures and a generative, masked pretraining strategy to learn unified representations from vast, heterogeneous health records. Our model is pretrained on an innovative multi-modal integration of data from multiple large-scale datasets (including MIMIC-III-WDB, MIMIC-IV-ECG, and CODE), comprising cardiac signals and the corresponding clinical or machine-generated text reports from approximately 1.7 million individuals. We demonstrate that the embeddings derived from our CSFM not only serve as effective feature extractors across diverse cardiac sensing scenarios, but also enable seamless transfer learning across varying input configurations and sensor modalities. Extensive evaluations across diagnostic tasks, demographic information recognition, vital sign measurement, clinical outcome prediction, and ECG question answering reveal that CSFM consistently outperforms traditional one-modal-one-task approaches. Notably, CSFM exhibits robust performance across multiple ECG lead configurations from standard 12-lead systems to single-lead setups, and in scenarios where only ECG, only PPG, or a combination thereof is available. These findings highlight the potential of CSFM as a versatile and scalable solution, for comprehensive cardiac monitoring.
Abstract (translated)
心脏生物信号，如心电图（ECG）和光电容积描记图（PPG），对于诊断、预防和管理心血管疾病至关重要，并已在各种临床任务中广泛应用。传统的深度学习方法在分析这些信号时通常依赖于同质的数据集和静态定制模型，这限制了它们在不同临床环境和采集协议下的稳健性和泛化能力。 在这项研究中，我们提出了一个心脏感知基础模型（CSFM），它利用先进的转换器架构和生成式、屏蔽预训练策略从庞大的异构健康记录中学习统一的表示。我们的模型是通过将来自多个大规模数据集（包括MIMIC-III-WDB、MIMIC-IV-ECG 和 CODE）的数据进行创新的多模态整合来预先训练的，这些数据集包含了大约170万个体的心脏信号及其相应的临床或机器生成的文本报告。我们证明了从我们的CSFM 中导出的嵌入不仅可以作为在各种心脏感知场景中的有效特征提取器，还能够实现在不同输入配置和传感器模式下的无缝迁移学习。 通过诊断任务、人口统计信息识别、生命体征测量、临床结果预测以及心电图问题回答等多方面的广泛评估显示，CSFM 在传统的单一模态单一任务方法上表现出了持续的优越性。值得注意的是，在标准12导联系统到单导联设置等各种ECG 导联配置中，CSFM 表现出稳健的表现，并且在仅提供 ECG 信号、PPG 信号或两者组合的情况下也能保持良好的性能。 这些发现突显了 CSFM 作为一种灵活和可扩展的解决方案的潜力，以实现全面的心脏监测。
URL
https://arxiv.org/abs/2507.01045
PDF
-
Focus Your Attention: Towards Data-Intuitive Lightweight Vision Transformers
2025-06-23 16:00:57Suyash Gaurav, Muhammad Farhan Humayun, Jukka Heikkonen, Jatin ChaudharyAbstract
The evolution of Vision Transformers has led to their widespread adaptation to different domains. Despite large-scale success, there remain significant challenges including their reliance on extensive computational and memory resources for pre-training on huge datasets as well as difficulties in task-specific transfer learning. These limitations coupled with energy inefficiencies mainly arise due to the computation-intensive self-attention mechanism. To address these issues, we propose a novel Super-Pixel Based Patch Pooling (SPPP) technique that generates context-aware, semantically rich, patch embeddings to effectively reduce the architectural complexity and improve efficiency. Additionally, we introduce the Light Latent Attention (LLA) module in our pipeline by integrating latent tokens into the attention mechanism allowing cross-attention operations to significantly reduce the time and space complexity of the attention module. By leveraging the data-intuitive patch embeddings coupled with dynamic positional encodings, our approach adaptively modulates the cross-attention process to focus on informative regions while maintaining the global semantic structure. This targeted attention improves training efficiency and accelerates convergence. Notably, the SPPP module is lightweight and can be easily integrated into existing transformer architectures. Extensive experiments demonstrate that our proposed architecture provides significant improvements in terms of computational efficiency while achieving comparable results with the state-of-the-art approaches, highlighting its potential for energy-efficient transformers suitable for edge deployment. (The code is available on our GitHub repository: this https URL).
Abstract (translated)
视觉变换器（Vision Transformers）的演进已经促使其在不同领域的广泛应用。尽管取得了大规模的成功，但它们仍然面临着一些重要挑战，包括依赖于大量计算和内存资源进行预训练的巨大数据集，以及任务特定迁移学习的困难。这些限制主要源于计算密集型的自我注意机制，并且伴随着能源效率低下的问题。 为了解决这些问题，我们提出了一种基于超级像素的补丁池化（Super-Pixel Based Patch Pooling, SPPP）技术，该技术生成了上下文感知、语义丰富的补丁嵌入，从而有效地减少架构复杂性并提高效率。此外，我们在我们的管道中引入了一个轻量级潜注意（Light Latent Attention, LLA）模块，通过将潜在令牌整合到注意力机制中，允许跨注意力操作显著降低时间与空间的复杂度。 通过利用直观的数据补丁嵌入和动态位置编码，我们所提出的方法能够自适应地调节跨注意力过程，聚焦于信息丰富的区域的同时保持全局语义结构。这种目标化注意提高了训练效率并加速了收敛速度。值得注意的是，SPPP模块轻量级且易于集成到现有的变换器架构中。 广泛的实验表明，我们的提出的架构在计算效率方面提供了显著的改进，并能与最先进的方法实现相媲美的结果，突显了其用于边缘部署的能量高效型变压器的潜力。代码可在我们的GitHub存储库（此链接）上获得。
URL
https://arxiv.org/abs/2506.18791
PDF
-
Benchmarking histopathology foundation models in a multi-center dataset for skin cancer subtyping
2025-06-23 14:12:16Pablo Meseguer, Roc\'io del Amor, Valery NaranjoAbstract
Pretraining on large-scale, in-domain datasets grants histopathology foundation models (FM) the ability to learn task-agnostic data representations, enhancing transfer learning on downstream tasks. In computational pathology, automated whole slide image analysis requires multiple instance learning (MIL) frameworks due to the gigapixel scale of the slides. The diversity among histopathology FMs has highlighted the need to design real-world challenges for evaluating their effectiveness. To bridge this gap, our work presents a novel benchmark for evaluating histopathology FMs as patch-level feature extractors within a MIL classification framework. For that purpose, we leverage the AI4SkIN dataset, a multi-center cohort encompassing slides with challenging cutaneous spindle cell neoplasm subtypes. We also define the Foundation Model - Silhouette Index (FM-SI), a novel metric to measure model consistency against distribution shifts. Our experimentation shows that extracting less biased features enhances classification performance, especially in similarity-based MIL classifiers.
Abstract (translated)
大规模、领域内数据集的预训练赋予了组织病理学基础模型（Foundation Models，简称 FM）学习任务无关的数据表示能力，从而增强了其在下游任务中的迁移学习效果。在计算病理学中，由于全片扫描图像达到数十亿像素的规模，自动化分析需要采用多实例学习（Multiple Instance Learning, MIL）框架。组织病理学基础模型之间的多样性凸显了设计现实世界挑战以评估它们有效性的必要性。 为弥合这一差距，我们的工作提出了一项新的基准测试，用于在基于MIL分类框架中作为补丁级特征提取器的组织病理学基础模型的效果进行评价。为此，我们利用了AI4SkIN数据集，该数据集中涵盖了多中心队列的皮肤梭形细胞肿瘤亚型全片扫描图像，这些图像极具挑战性。 此外，我们还定义了一个新的评估标准——Foundation Model - Silhouette Index（FM-SI），用于衡量模型在面对分布偏移时的一致性。实验结果表明，在基于相似性的MIL分类器中，提取出较少偏置特征能显著提升分类性能。
URL
https://arxiv.org/abs/2506.18668
PDF
-
When Fine-Tuning Fails: Lessons from MS MARCO Passage Ranking
2025-06-23 11:46:05Manu Pande, Shahil Kumar, Anay Yatin DamleAbstract
This paper investigates the counterintuitive phenomenon where fine-tuning pre-trained transformer models degrades performance on the MS MARCO passage ranking task. Through comprehensive experiments involving five model variants-including full parameter fine-tuning and parameter efficient LoRA adaptations-we demonstrate that all fine-tuning approaches underperform the base sentence-transformers/all- MiniLM-L6-v2 model (MRR@10: 0.3026). Our analysis reveals that fine-tuning disrupts the optimal embedding space structure learned during the base model's extensive pre-training on 1 billion sentence pairs, including 9.1 million MS MARCO samples. UMAP visualizations show progressive embedding space flattening, while training dynamics analysis and computational efficiency metrics further support our findings. These results challenge conventional wisdom about transfer learning effectiveness on saturated benchmarks and suggest architectural innovations may be necessary for meaningful improvements.
Abstract (translated)
本文研究了一种违反直觉的现象，即微调预训练的变压器模型会降低在MS MARCO段落排名任务上的性能。通过包括全参数微调和高效LoRA适应在内的五种模型变体的全面实验，我们证明了所有微调方法的表现均劣于基础句子转换器/all-MiniLM-L6-v2模型（MRR@10：0.3026）。我们的分析表明，微调破坏了基础模型在大规模预训练中学习到的最佳嵌入空间结构，该过程基于10亿个句子对的预训练数据集，其中包括910万份MS MARCO样本。UMAP可视化显示了嵌入空间逐渐变平的趋势，而训练动态分析和计算效率指标进一步支持了我们的发现。这些结果挑战了关于饱和基准测试中迁移学习有效性的传统观点，并暗示为了实现有意义的进步，可能需要架构创新。
URL
https://arxiv.org/abs/2506.18535
PDF
-
Generalizing Vision-Language Models to Novel Domains: A Comprehensive Survey
2025-06-23 10:56:37Xinyao Li, Jingjing Li, Fengling Li, Lei Zhu, Yang Yang, Heng Tao ShenAbstract
Recently, vision-language pretraining has emerged as a transformative technique that integrates the strengths of both visual and textual modalities, resulting in powerful vision-language models (VLMs). Leveraging web-scale pretraining data, these models exhibit strong zero-shot capabilities. However, their performance often deteriorates when confronted with domain-specific or specialized generalization tasks. To address this, a growing body of research focuses on transferring or generalizing the rich knowledge embedded in VLMs to various downstream applications. This survey aims to comprehensively summarize the generalization settings, methodologies, benchmarking and results in VLM literatures. Delving into the typical VLM structures, current literatures are categorized into prompt-based, parameter-based and feature-based methods according to the transferred modules. The differences and characteristics in each category are furthered summarized and discussed by revisiting the typical transfer learning (TL) settings, providing novel interpretations for TL in the era of VLMs. Popular benchmarks for VLM generalization are further introduced with thorough performance comparisons among the reviewed methods. Following the advances in large-scale generalizable pretraining, this survey also discusses the relations and differences between VLMs and up-to-date multimodal large language models (MLLM), e.g., DeepSeek-VL. By systematically reviewing the surging literatures in vision-language research from a novel and practical generalization prospective, this survey contributes to a clear landscape of current and future multimodal researches.
Abstract (translated)
最近，视觉-语言预训练作为一种变革性技术应运而生，它结合了视觉和文本模态的优势，产生了强大的视觉-语言模型（VLM）。通过利用大规模的网络预训练数据，这些模型展示了强大的零样本学习能力。然而，在面对特定领域或专业化的泛化任务时，它们的表现往往会下降。为了解决这一问题，越来越多的研究专注于将嵌入在VLM中的丰富知识转移到各种下游应用中。本综述旨在全面总结视觉-语言文献中的泛化设置、方法论、基准测试和结果。深入探讨了典型的VLM结构，并根据转移模块的不同，当前的文献被分类为基于提示（prompt-based）、基于参数（parameter-based）和基于特征（feature-based）的方法。通过回顾典型的迁移学习（TL）设定，对每一类别的差异和特点进行了进一步总结与讨论，提供了在VLM时代TL的新颖解释。本综述还介绍了用于VLM泛化的流行基准，并对比了所审查方法的性能表现。随着大规模可泛化预训练的进步，该综述还讨论了VLM与最新多模态大型语言模型（MLLM）如DeepSeek-VL之间的关系和差异。通过从新颖且实用的角度系统地回顾视觉-语言研究中的激增文献，本综述有助于清晰描绘当前和未来的多模态研究图景。
URL
https://arxiv.org/abs/2506.18504
PDF
-
These are Not All the Features You are Looking For: A Fundamental Bottleneck In Supervised Pretraining
2025-06-23 01:04:29Xingyu Alice Yang, Jianyu Zhang, L\'eon BottouAbstract
Transfer learning is a cornerstone of modern machine learning, promising a way to adapt models pretrained on a broad mix of data to new tasks with minimal new data. However, a significant challenge remains in ensuring that transferred features are sufficient to handle unseen datasets, amplified by the difficulty of quantifying whether two tasks are "related". To address these challenges, we evaluate model transfer from a pretraining mixture to each of its component tasks, assessing whether pretrained features can match the performance of task-specific direct training. We identify a fundamental limitation in deep learning models -- an "information saturation bottleneck" -- where networks fail to learn new features once they encode similar competing features during training. When restricted to learning only a subset of key features during pretraining, models will permanently lose critical features for transfer and perform inconsistently on data distributions, even components of the training mixture. Empirical evidence from published studies suggests that this phenomenon is pervasive in deep learning architectures -- factors such as data distribution or ordering affect the features that current representation learning methods can learn over time. This study suggests that relying solely on large-scale networks may not be as effective as focusing on task-specific training, when available. We propose richer feature representations as a potential solution to better generalize across new datasets and, specifically, present existing methods alongside a novel approach, the initial steps towards addressing this challenge.
Abstract (translated)
迁移学习是现代机器学习的基石，它承诺了一种方法，可以将预先训练好的模型在大量数据上进行微调，以便适应新的任务，并且只需要少量的新数据。然而，一个重要的挑战仍然存在：如何确保迁移到新任务中的特征足以应对未见过的数据集，尤其是在量化两个任务是否“相关”这一难题面前。 为了解决这些问题，我们评估了从预训练混合物到其各个组成部分任务的模型迁移情况，以确定预先训练好的特征能否达到针对特定任务直接训练的表现水平。我们的研究揭示了一个深度学习模型的基本局限性——一种“信息饱和瓶颈”，在这种情况下，网络一旦在训练过程中编码了相似的竞争特征就无法再学习新的特征。 当我们限制模型仅在预训练阶段学习关键特征的一个子集时，发现这些模型会永久失去对于迁移至关重要的某些特征，并且会在数据分布上表现不稳定，甚至包括它们原来的数据混合部分。来自已发表研究的实证证据表明，这种现象广泛存在于深度学习架构中——数据分布或顺序等因素影响了当前表示学习方法随时间所能学到的特征。 这项研究表明，在有特定任务训练的情况下，仅仅依赖大规模网络可能不如专注于针对具体任务的训练有效。我们提出更丰富的特征表示作为解决这一挑战的一种潜在方案，以更好地在新数据集中泛化，并且特别地介绍了现有方法以及一种新的方法，这标志着应对这一挑战的第一步。
URL
https://arxiv.org/abs/2506.18221
PDF
-
Deep Supervised LSTM for 3D morphology estimation from Multi-View RGB Images of Wheat Spikes
2025-06-22 15:02:18Olivia Zumsteg, Nico Graf, Aaron Haeusler, Norbert Kirchgessner, Nicola Storni, Lukas Roth, Andreas HundAbstract
Estimating three-dimensional morphological traits from two-dimensional RGB images presents inherent challenges due to the loss of depth information, projection distortions, and occlusions under field conditions. In this work, we explore multiple approaches for non-destructive volume estimation of wheat spikes, using RGB image sequences and structured-light 3D scans as ground truth references. Due to the complex geometry of the spikes, we propose a neural network approach for volume estimation in 2D images, employing a transfer learning pipeline that combines DINOv2, a self-supervised Vision Transformer, with a unidirectional Long Short-Term Memory (LSTM) network. By using deep supervision, the model is able to learn more robust intermediate representations, which enhances its generalisation ability across varying evaluation sequences. We benchmark our model against two conventional baselines: a 2D area-based projection and a geometric reconstruction using axis-aligned cross-sections. Our deep supervised model achieves a mean absolute percentage error (MAPE) of 6.46% on six-view indoor images, outperforming the area (9.36%) and geometric (13.98%) baselines. Fine-tuning the model on field-based single-image data enables domain adaptation, yielding a MAPE of 10.82%. We demonstrate that object shape significantly impacts volume prediction accuracy, with irregular geometries such as wheat spikes posing greater challenges for geometric methods compared to our deep learning approach.
Abstract (translated)
从二维RGB图像估算三维形态特征由于深度信息丢失、投影失真以及现场条件下的遮挡问题，存在固有挑战。在这项工作中，我们探索了多种方法用于小麦穗的无损体积估计，使用RGB图像序列和结构光3D扫描作为基准参考数据。鉴于穗子复杂的几何形状，我们提出了一种基于二维图像的卷积神经网络体积估算方法，采用一个结合自监督视觉变换器DINOv2与单向长短期记忆（LSTM）网络的迁移学习管道。通过深度监督训练模型，它能够学习更加稳健的中间表示形式，从而提高其在不同评估序列中的泛化能力。 我们使用两种传统基准进行对比测试：基于二维面积的投影方法和轴对齐横截面的几何重建法。我们的深度监督模型在六视图室内图像数据集上取得了6.46%的平均绝对百分比误差（MAPE），优于面积（9.36%）和几何（13.98%）基准方法。通过针对基于单张现场图片的数据进行微调，该模型实现了领域适应性，在这种情况下其预测结果的MAPE值为10.82%。 实验表明物体形状对体积预测准确性具有重大影响，对于像小麦穗这样几何不规则的对象来说，传统的几何方法面临着更大的挑战，而我们的深度学习方法表现更佳。
URL
https://arxiv.org/abs/2506.18060
PDF
-
Classification of Tents in Street Bazaars Using CNN
2025-06-22 08:52:28Azamat Ibragimov, Ruslan Isaev, Remudin Reshid Mekuria, Gulnaz Gimaletdinova, Dim ShaiakhmetovAbstract
This research paper proposes an improved deep learning model for classifying tents in street bazaars, comparing a custom Convolutional Neural Network (CNN) with EfficientNetB0. This is a critical task for market organization with a tent classification, but manual methods in the past have been inefficient. Street bazaars represent a vital economic hub in many regions, yet their unstructured nature poses significant challenges for the automated classification of market infrastructure, such as tents. In Kyrgyzstan, more than a quarter of the country's GDP is derived from bazaars. While CNNs have been widely applied to object recognition, their application to bazaar-specific tasks remains underexplored. Here, we build upon our original approach by training on an extended set of 126 original photographs that were augmented to generate additional images. This dataset is publicly available for download on Kaggle. A variety of performance metrics, such as accuracy, precision, recall, F1 score, and mean average precision (mAP), were used to assess the models comparatively, providing a more extensive analysis of classification performance. The results show that the CNN custom model achieved 92.8% accuracy, and EfficientNetB0 showed 98.4% accuracy results, confirming the effectiveness of transfer learning in the bazaar image classification. Also, when analyzing the confusion matrix, the analysis reveals the weaknesses and strengths of each model. These findings suggest that using a pre-trained model such as EfficientNetB0 significantly improves classification accuracy and generalization.
Abstract (translated)
这篇研究论文提出了一种改进的深度学习模型，用于分类街头集市中的帐篷，并将自定义卷积神经网络（CNN）与EfficientNetB0进行了比较。这一任务对于以帐篷分类为关键环节的市场组织至关重要，但过去的手动方法效率低下。街头集市在许多地区是重要的经济中心，但由于其非结构化的特性，自动分类市场的基础设施（如帐篷）存在重大挑战。在吉尔吉斯斯坦，超过四分之一的国内生产总值来自集市。尽管卷积神经网络广泛应用于对象识别领域，但它们在特定于集市的任务上的应用仍然较少被探索。 在这里，我们扩展了原来的方法，在包含126张原始照片的数据集上进行训练，并对这些照片进行了数据增强以生成更多图像，这个数据集可以在Kaggle上公开下载。为了对比模型性能，使用了多种评估指标，包括准确率、精确度、召回率、F1分数和平均精度（mAP），这为分类性能提供了更全面的分析。 实验结果显示，自定义CNN模型实现了92.8%的准确率，而EfficientNetB0则达到了98.4%的准确率。这些结果证实了迁移学习在集市图像分类中的有效性。此外，在分析混淆矩阵时，揭示了每个模型的优势和劣势所在。研究表明，使用像EfficientNetB0这样的预训练模型可以显著提高分类精度并增强泛化能力。
URL
https://arxiv.org/abs/2506.17946
PDF
-
Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages
2025-06-21 13:33:07Matthias Sch\"offel, Esteban Garces Arias, Marinus Wiedner, Paula Ruppert, Meimingwei Li, Christian Heumann, Matthias A{\ss}enmacherAbstract
Part-of-speech (POS) tagging remains a foundational component in natural language processing pipelines, particularly critical for historical text analysis at the intersection of computational linguistics and digital humanities. Despite significant advancements in modern large language models (LLMs) for ancient languages, their application to Medieval Romance languages presents distinctive challenges stemming from diachronic linguistic evolution, spelling variations, and labeled data scarcity. This study systematically investigates the central determinants of POS tagging performance across diverse corpora of Medieval Occitan, Medieval Spanish, and Medieval French texts, spanning biblical, hagiographical, medical, and dietary domains. Through rigorous experimentation, we evaluate how fine-tuning approaches, prompt engineering, model architectures, decoding strategies, and cross-lingual transfer learning techniques affect tagging accuracy. Our results reveal both notable limitations in LLMs' ability to process historical language variations and non-standardized spelling, as well as promising specialized techniques that effectively address the unique challenges presented by low-resource historical languages.
Abstract (translated)
词性标注（POS）仍然是自然语言处理管道中的一个基础组成部分，特别是在计算语言学和数字人文科学交汇处的历史文本分析中尤为重要。尽管现代大型语言模型（LLM）在古代语言应用上取得了显著进展，但这些模型在处理中世纪罗曼语系语言时面临着独特的挑战，这些问题源自历时性的语言演变、拼写变化以及标注数据的稀缺性。本研究系统地调查了词性标注性能在不同中世纪奥克语、西班牙语和法语文本语料库中的关键决定因素，这些文本涵盖了圣经、圣徒传记、医学和饮食等领域。通过严格的实验，我们评估了微调方法、提示工程、模型架构、解码策略以及跨语言迁移学习技术如何影响词性标注的准确性。我们的研究结果揭示了LLM处理历史语言变化及非标准化拼写时存在的显著局限，并且还发现了一些有前景的专业技术，这些技术能够有效应对低资源历史语言所面临的独特挑战。
URL
https://arxiv.org/abs/2506.17715
PDF
-
Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction
2025-06-20 22:48:07Julio Silva-Rodr\'iguez, Ismail Ben Ayed, Jose DolzAbstract
Medical vision-language models (VLMs) have demonstrated unprecedented transfer capabilities and are being increasingly adopted for data-efficient image classification. Despite its growing popularity, its reliability aspect remains largely unexplored. This work explores the split conformal prediction (SCP) framework to provide trustworthiness guarantees when transferring such models based on a small labeled calibration set. Despite its potential, the generalist nature of the VLMs' pre-training could negatively affect the properties of the predicted conformal sets for specific tasks. While common practice in transfer learning for discriminative purposes involves an adaptation stage, we observe that deploying such a solution for conformal purposes is suboptimal since adapting the model using the available calibration data breaks the rigid exchangeability assumptions for test data in SCP. To address this issue, we propose transductive split conformal adaptation (SCA-T), a novel pipeline for transfer learning on conformal scenarios, which performs an unsupervised transductive adaptation jointly on calibration and test data. We present comprehensive experiments utilizing medical VLMs across various image modalities, transfer tasks, and non-conformity scores. Our framework offers consistent gains in efficiency and conditional coverage compared to SCP, maintaining the same empirical guarantees.
Abstract (translated)
医学视觉-语言模型（VLMs）已经展示了前所未有的转移能力，并且越来越多地被用于基于少量标注数据进行高效的图像分类。尽管其流行度日益增加，但其可靠性方面仍鲜有研究。这项工作探索了分裂置信预测（SCP）框架，在将这些模型应用于特定任务时，利用一个小的标记校准集提供可信度保证。尽管具有潜在优势，VLMs预训练的一般特性可能会对特定任务的预测置信集合属性产生负面影响。虽然在转移学习中为了区分目的通常涉及一个适应阶段，但我们观察到将其用于置信性目的效果不佳，因为在使用可用校准数据调整模型时会破坏SCP中关于测试数据的刚性可交换性假设。 为了解决这个问题，我们提出了一种新型的管道——转导分裂置信预测适配（SCA-T），专门针对置信场景下的转移学习。该方法在标记和未标记数据上执行无监督的转导适应过程。我们在各种医学VLMs、不同的图像模态、转移任务以及非一致性评分上进行了全面实验，验证了我们框架的有效性。 我们的框架相比于SCP，在效率和条件覆盖率方面表现出一致的优势，并保持相同的实证保证。
URL
https://arxiv.org/abs/2506.17503
PDF
-
Energy-Based Transfer for Reinforcement Learning
2025-06-19 20:25:52Zeyun Deng, Jasorsi Ghosh, Fiona Xie, Yuzhe Lu, Katia Sycara, Joseph CampbellAbstract
Reinforcement learning algorithms often suffer from poor sample efficiency, making them challenging to apply in multi-task or continual learning settings. Efficiency can be improved by transferring knowledge from a previously trained teacher policy to guide exploration in new but related tasks. However, if the new task sufficiently differs from the teacher's training task, the transferred guidance may be sub-optimal and bias exploration toward low-reward behaviors. We propose an energy-based transfer learning method that uses out-of-distribution detection to selectively issue guidance, enabling the teacher to intervene only in states within its training distribution. We theoretically show that energy scores reflect the teacher's state-visitation density and empirically demonstrate improved sample efficiency and performance across both single-task and multi-task settings.
Abstract (translated)
强化学习算法常常面临样本效率低下的问题，这使得它们在多任务或多阶段学习环境中难以应用。通过从先前训练好的教师策略中转移知识来指导新但相关任务中的探索可以提高效率。然而，如果新的任务与教师训练的任务有显著差异，转移的引导可能会次优，并导致向低收益行为偏移的探索。 我们提出了一种基于能量的迁移学习方法，该方法使用分布外检测（out-of-distribution detection）来选择性地发出指导信号，使教师仅在处于其训练数据分布内的状态下进行干预。理论上，这种能量得分反映了教师访问状态的概率密度，并且实验上表明这种方法能够提升单任务和多任务设置中的样本效率及性能。
URL
https://arxiv.org/abs/2506.16590
PDF
-
DiFuse-Net: RGB and Dual-Pixel Depth Estimation using Window Bi-directional Parallax Attention and Cross-modal Transfer Learning
2025-06-17 16:49:27Kunal Swami, Debtanu Gupta, Amrit Kumar Muduli, Chirag Jaiswal, Pankaj Kumar BajpaiAbstract
Depth estimation is crucial for intelligent systems, enabling applications from autonomous navigation to augmented reality. While traditional stereo and active depth sensors have limitations in cost, power, and robustness, dual-pixel (DP) technology, ubiquitous in modern cameras, offers a compelling alternative. This paper introduces DiFuse-Net, a novel modality decoupled network design for disentangled RGB and DP based depth estimation. DiFuse-Net features a window bi-directional parallax attention mechanism (WBiPAM) specifically designed to capture the subtle DP disparity cues unique to smartphone cameras with small aperture. A separate encoder extracts contextual information from the RGB image, and these features are fused to enhance depth prediction. We also propose a Cross-modal Transfer Learning (CmTL) mechanism to utilize large-scale RGB-D datasets in the literature to cope with the limitations of obtaining large-scale RGB-DP-D dataset. Our evaluation and comparison of the proposed method demonstrates its superiority over the DP and stereo-based baseline methods. Additionally, we contribute a new, high-quality, real-world RGB-DP-D training dataset, named Dual-Camera Dual-Pixel (DCDP) dataset, created using our novel symmetric stereo camera hardware setup, stereo calibration and rectification protocol, and AI stereo disparity estimation method.
Abstract (translated)
深度估计对于智能系统至关重要，可支持从自主导航到增强现实等多种应用。虽然传统的立体视觉和主动测距传感器在成本、功耗和鲁棒性方面存在局限性，但如今广泛应用于现代相机的双像素（DP）技术提供了一种颇具吸引力的替代方案。本文介绍了DiFuse-Net，这是一种新型的模态解耦网络设计，用于基于RGB和DP的深度估计。DiFuse-Net具备一种专门针对手机摄像头的小孔径特性而设计的窗口双向视差注意力机制（WBiPAM），能够捕捉双像素特有的细微视差线索。该网络还包括一个单独的编码器从RGB图像中提取上下文信息，并将这些特征融合以增强深度预测。我们还提出了一种跨模态迁移学习（CmTL）机制，利用文献中的大规模RGB-D数据集来克服获取大规模RGB-DP-D数据集的局限性。我们的评估和对比试验表明，所提方法优于基于DP和立体视觉的基线方法。此外，我们贡献了一个新的高质量的真实世界RGB-DP-D训练数据集——双摄像头双像素（DCDP）数据集，该数据集使用我们创新性的对称立体相机硬件设置、立体校准和直方图矫正协议以及AI立体视差估计方法创建而成。
URL
https://arxiv.org/abs/2506.14709
PDF
-
Adjustment for Confounding using Pre-Trained Representations
2025-06-17 09:11:17Rickmer Schulte, David R\"ugamer, Thomas NaglerAbstract
There is growing interest in extending average treatment effect (ATE) estimation to incorporate non-tabular data, such as images and text, which may act as sources of confounding. Neglecting these effects risks biased results and flawed scientific conclusions. However, incorporating non-tabular data necessitates sophisticated feature extractors, often in combination with ideas of transfer learning. In this work, we investigate how latent features from pre-trained neural networks can be leveraged to adjust for sources of confounding. We formalize conditions under which these latent features enable valid adjustment and statistical inference in ATE estimation, demonstrating results along the example of double machine learning. We discuss critical challenges inherent to latent feature learning and downstream parameter estimation arising from the high dimensionality and non-identifiability of representations. Common structural assumptions for obtaining fast convergence rates with additive or sparse linear models are shown to be unrealistic for latent features. We argue, however, that neural networks are largely insensitive to these issues. In particular, we show that neural networks can achieve fast convergence rates by adapting to intrinsic notions of sparsity and dimension of the learning problem.
Abstract (translated)
人们对将平均处理效应（ATE）估计扩展到包含非表格数据（如图像和文本），这些数据可能作为混杂因素来源的兴趣日益增加。忽视这些影响可能会导致结果偏倚及科学结论错误。然而，纳入非表格数据需要复杂的特征提取器，并且通常与迁移学习的概念相结合。在本工作中，我们研究了如何利用预训练神经网络的潜在特征来调整混杂因素的影响。我们形式化了这些潜在特征能够使ATE估计中的有效调整和统计推断成立的条件，并通过双重机器学习的例子展示了其结果。我们讨论了源自高维度和表示非唯一性带来的下游参数估计中潜在特征学习所固有的关键挑战。获取具有加性和稀疏线性模型的快速收敛率所需的常见结构假设对于潜在特征来说是不现实的。然而，我们认为神经网络在很大程度上对这些问题不敏感。特别是，我们展示了通过适应于学习问题内在的稀疏度和维度概念，神经网络可以实现快速收敛速度。
URL
https://arxiv.org/abs/2506.14329
PDF
