Transfer_Learning
-
Less is More: Undertraining Experts Improves Model Upcycling
2025-06-17 02:42:10Stefan Horoi, Guy Wolf, Eugene Belilovsky, Gintare Karolina DziugaiteAbstract
Modern deep learning is increasingly characterized by the use of open-weight foundation models that can be fine-tuned on specialized datasets. This has led to a proliferation of expert models and adapters, often shared via platforms like HuggingFace and AdapterHub. To leverage these resources, numerous model upcycling methods have emerged, enabling the reuse of fine-tuned models in multi-task systems. A natural pipeline has thus formed to harness the benefits of transfer learning and amortize sunk training costs: models are pre-trained on general data, fine-tuned on specific tasks, and then upcycled into more general-purpose systems. A prevailing assumption is that improvements at one stage of this pipeline propagate downstream, leading to gains at subsequent steps. In this work, we challenge that assumption by examining how expert fine-tuning affects model upcycling. We show that long fine-tuning of experts that optimizes for their individual performance leads to degraded merging performance, both for fully fine-tuned and LoRA-adapted models, and to worse downstream results when LoRA adapters are upcycled into MoE layers. We trace this degradation to the memorization of a small set of difficult examples that dominate late fine-tuning steps and are subsequently forgotten during merging. Finally, we demonstrate that a task-dependent aggressive early stopping strategy can significantly improve upcycling performance.
Abstract (translated)
现代深度学习的一个显著特征是使用开放权重基础模型，这些模型可以在特定数据集上进行微调。这导致了专家模型和适配器的大量涌现，并且常常通过像HuggingFace和AdapterHub这样的平台共享。为了利用这些资源，已经出现了许多模型再利用方法，使得经过微调的模型可以被重用于多任务系统中。因此，形成了一种自然的工作流程，以实现迁移学习的好处并分摊沉没训练成本：首先是基于通用数据对模型进行预训练，然后在特定任务上对其进行微调，并最终将其重新用作更通用系统的组成部分。一种普遍的观点是，在这条管道中的每一个阶段的改进都会传递到下游环节，从而带来后续步骤上的收益。 然而，在这项工作中，我们通过研究专家级微调如何影响模型再利用来挑战这一假设。我们展示了长期对专家进行微调以优化其个体性能会导致合并时表现下降，并且当使用LoRA适配器时还会导致更差的下游结果。我们将这种退化归因于记忆少数几个困难样本的现象，这些样本在后期微调中占主导地位，但在合并阶段又被遗忘。最后，我们证明了一种依赖任务的激进早期停止策略可以显著提升再利用性能。 简而言之，过度优化特定任务会导致模型泛化能力下降，从而影响到后续的多任务或转移学习效果。我们的研究提出了一种新的微调和适配器重用策略，以提高这些模型在广泛场景中的适应性和有效性。
URL
https://arxiv.org/abs/2506.14126
PDF
-
Advancing Image-Based Grapevine Variety Classification with a New Benchmark and Evaluation of Masked Autoencoders
2025-06-16 10:25:23Gabriel A. Carneiro, Thierry J. Aubry, Ant\'onio Cunha, Petia Radeva, Joaquim SousaAbstract
Grapevine varieties are essential for the economies of many wine-producing countries, influencing the production of wine, juice, and the consumption of fruits and leaves. Traditional identification methods, such as ampelography and molecular analysis, have limitations: ampelography depends on expert knowledge and is inherently subjective, while molecular methods are costly and time-intensive. To address these limitations, recent studies have applied deep learning (DL) models to classify grapevine varieties using image data. However, due to the small dataset sizes, these methods often depend on transfer learning from datasets from other domains, e.g., ImageNet1K (IN1K), which can lead to performance degradation due to domain shift and supervision collapse. In this context, self-supervised learning (SSL) methods can be a good tool to avoid this performance degradation, since they can learn directly from data, without external labels. This study presents an evaluation of Masked Autoencoders (MAEs) for identifying grapevine varieties based on field-acquired images. The main contributions of this study include two benchmarks comprising 43 grapevine varieties collected across different seasons, an analysis of MAE's application in the agricultural context, and a performance comparison of trained models across seasons. Our results show that a ViT-B/16 model pre-trained with MAE and the unlabeled dataset achieved an F1 score of 0.7956, outperforming all other models. Additionally, we observed that pre-trained models benefit from long pre-training, perform well under low-data training regime, and that simple data augmentation methods are more effective than complex ones. The study also found that the mask ratio in MAE impacts performance only marginally.
Abstract (translated)
葡萄品种对于许多葡萄酒生产国的经济至关重要，影响着葡萄酒、果汁的生产和水果及叶片的消费。传统的识别方法，如植物学分类（ampelography）和分子分析，存在局限性：植物学分类依赖于专家知识且具有主观性，而分子方法则成本高昂且耗时较长。为解决这些限制，最近的研究开始应用深度学习（DL）模型，利用图像数据对葡萄品种进行分类。然而，由于小规模的数据集，这些方法通常需要从其他领域（例如ImageNet1K (IN1K)）的大型数据集中借用迁移学习，这可能会导致因域转移和监督崩溃而导致性能下降。在这种背景下，自监督学习（SSL）方法可以成为避免这种性能下降的好工具，因为它们可以直接从数据中进行学习，而无需外部标签。 本研究评估了Masked Autoencoders (MAEs) 在基于实地采集图像识别葡萄品种方面的应用。该研究的主要贡献包括两个基准测试集，涵盖了43种不同季节的葡萄品种，对MAE在农业领域的应用进行了分析，并对比了不同季节训练模型的表现情况。我们的结果显示，ViT-B/16模型经过MAE和未标记数据预训练后，在F1得分上达到了0.7956，优于所有其他模型表现。此外，我们还观察到，预训练模型长期预训练效果更佳，在低数据量的训练环境中性能更好，并且简单的数据增强方法比复杂的更为有效。该研究还发现，MAE中的遮罩比率对性能的影响微乎其微。 总的来说，这项工作提供了一种新的自监督学习框架来改善葡萄品种识别任务的有效性，特别关注于小规模数据集和复杂农业环境中的应用挑战。
URL
https://arxiv.org/abs/2506.13335
PDF
-
Evolution of ReID: From Early Methods to LLM Integration
2025-06-16 02:03:46Amran Bhuiyan, Mizanur Rahman, Md Tahmid Rahman Laskar, Aijun An, Jimmy Xiangji HuangAbstract
Person re-identification (ReID) has evolved from handcrafted feature-based methods to deep learning approaches and, more recently, to models incorporating large language models (LLMs). Early methods struggled with variations in lighting, pose, and viewpoint, but deep learning addressed these issues by learning robust visual features. Building on this, LLMs now enable ReID systems to integrate semantic and contextual information through natural language. This survey traces that full evolution and offers one of the first comprehensive reviews of ReID approaches that leverage LLMs, where textual descriptions are used as privileged information to improve visual matching. A key contribution is the use of dynamic, identity-specific prompts generated by GPT-4o, which enhance the alignment between images and text in vision-language ReID systems. Experimental results show that these descriptions improve accuracy, especially in complex or ambiguous cases. To support further research, we release a large set of GPT-4o-generated descriptions for standard ReID datasets. By bridging computer vision and natural language processing, this survey offers a unified perspective on the field's development and outlines key future directions such as better prompt design, cross-modal transfer learning, and real-world adaptability.
Abstract (translated)
人员重新识别（ReID）技术已经从基于手工制作特征的方法演进到深度学习方法，再到最近结合大型语言模型（LLMs）的模型。早期方法在处理光照、姿态和视角变化时遇到了困难，但深度学习通过学习鲁棒的视觉特征解决了这些问题。在此基础上，LLMs使得ReID系统能够通过自然语言整合语义和上下文信息。本次调查追溯了这一完整的发展历程，并对利用LLMs增强人员重新识别的方法进行了迄今为止最全面的回顾之一，其中文本描述被用作特权信息来提升视觉匹配性能。一个关键贡献是使用GPT-4o生成的身份特定动态提示，这增强了图像与文字之间的对齐，在视觉语言ReID系统中尤为有效。实验结果显示，这些描述在复杂或模棱两可的情况下特别提升了准确度。为了支持进一步的研究，我们发布了一大批针对标准ReID数据集的由GPT-4o生成的描述。通过将计算机视觉和自然语言处理领域结合在一起，这项调查提供了一个统一视角来审视该领域的进展，并概述了未来的关键发展方向，如更好的提示设计、跨模态迁移学习以及现实世界的适应性。
URL
https://arxiv.org/abs/2506.13039
PDF
-
Geometric Embedding Alignment via Curvature Matching in Transfer Learning
2025-06-16 00:54:22Sung Moon Ko, Jaewan Lee, Sumin Lee, Soorin Yim, Kyunghoon Bae, Sehui HanAbstract
Geometrical interpretations of deep learning models offer insightful perspectives into their underlying mathematical structures. In this work, we introduce a novel approach that leverages differential geometry, particularly concepts from Riemannian geometry, to integrate multiple models into a unified transfer learning framework. By aligning the Ricci curvature of latent space of individual models, we construct an interrelated architecture, namely Geometric Embedding Alignment via cuRvature matching in transfer learning (GEAR), which ensures comprehensive geometric representation across datapoints. This framework enables the effective aggregation of knowledge from diverse sources, thereby improving performance on target tasks. We evaluate our model on 23 molecular task pairs sourced from various domains and demonstrate significant performance gains over existing benchmark model under both random (14.4%) and scaffold (8.3%) data splits.
Abstract (translated)
深度学习模型的几何解释为理解其数学结构提供了深刻的视角。在这项工作中，我们提出了一种新颖的方法，利用微分几何（特别是黎曼几何的概念）将多个模型整合到一个统一的迁移学习框架中。通过使单个模型潜在空间的里奇曲率对齐，我们构建了一个相互关联的架构，即通过曲率匹配进行转移学习的几何嵌入对齐（GEAR），确保了数据点之间的全面几何表示。该框架能够有效地汇总来自不同来源的知识，从而提高目标任务上的性能。我们在23个分子任务配对（源自不同的领域）上评估了我们的模型，并在随机和支架数据拆分下均表现出显著的性能提升（分别提高了14.4%和8.3%）。
URL
https://arxiv.org/abs/2506.13015
PDF
-
Konooz: Multi-domain Multi-dialect Corpus for Named Entity Recognition
2025-06-14 19:58:55Nagham Hamad, Mohammed Khalilia, Mustafa JarrarAbstract
We introduce Konooz, a novel multi-dimensional corpus covering 16 Arabic dialects across 10 domains, resulting in 160 distinct corpora. The corpus comprises about 777k tokens, carefully collected and manually annotated with 21 entity types using both nested and flat annotation schemes - using the Wojood guidelines. While Konooz is useful for various NLP tasks like domain adaptation and transfer learning, this paper primarily focuses on benchmarking existing Arabic Named Entity Recognition (NER) models, especially cross-domain and cross-dialect model performance. Our benchmarking of four Arabic NER models using Konooz reveals a significant drop in performance of up to 38% when compared to the in-distribution data. Furthermore, we present an in-depth analysis of domain and dialect divergence and the impact of resource scarcity. We also measured the overlap between domains and dialects using the Maximum Mean Discrepancy (MMD) metric, and illustrated why certain NER models perform better on specific dialects and domains. Konooz is open-source and publicly available at this https URL
Abstract (translated)
我们介绍了Konooz，这是一个新颖的多维度语料库，涵盖了10个领域的16种阿拉伯方言，总计形成了160个独特的语料库。该语料库包含大约777,000个标记，并且这些标记被仔细收集并使用Wojood指南进行了手动标注，包括了21种实体类型和嵌套及扁平化注释方案。虽然Konooz对于各种NLP任务（如领域适应和迁移学习）非常有用，但本文主要关注于利用现有阿拉伯命名实体识别（NER）模型进行基准测试，特别是跨领域的跨方言模型性能。使用Konooz对四种阿拉伯NER模型的基准测试表明，在与分布内数据相比时，这些模型的表现下降了最多达38%。 此外，我们还深入分析了领域和方言差异以及资源稀缺的影响，并利用最大平均不一致度（MMD）指标测量了不同领域的重叠情况。我们展示了为什么某些NER模型在特定的方言和领域中表现更好，并解释了其背后的原因。Konooz是一个开源项目，并且可以在[此处](https://this-URL.com)公开获取。
URL
https://arxiv.org/abs/2506.12615
PDF
-
Hierarchical Deep Feature Fusion and Ensemble Learning for Enhanced Brain Tumor MRI Classification
2025-06-14 05:53:54Zahid Ullah, Jihie KimAbstract
Accurate brain tumor classification is crucial in medical imaging to ensure reliable diagnosis and effective treatment planning. This study introduces a novel double ensembling framework that synergistically combines pre-trained deep learning (DL) models for feature extraction with optimized machine learning (ML) classifiers for robust classification. The framework incorporates comprehensive preprocessing and data augmentation of brain magnetic resonance images (MRI), followed by deep feature extraction using transfer learning with pre-trained Vision Transformer (ViT) networks. The novelty lies in the dual-level ensembling strategy: feature-level ensembling, which integrates deep features from the top-performing ViT models, and classifier-level ensembling, which aggregates predictions from hyperparameter-optimized ML classifiers. Experiments on two public Kaggle MRI brain tumor datasets demonstrate that this approach significantly surpasses state-of-the-art methods, underscoring the importance of feature and classifier fusion. The proposed methodology also highlights the critical roles of hyperparameter optimization (HPO) and advanced preprocessing techniques in improving diagnostic accuracy and reliability, advancing the integration of DL and ML for clinically relevant medical image analysis.
Abstract (translated)
精确的大脑肿瘤分类在医学成像中至关重要，以确保可靠的诊断和有效的治疗计划。本研究提出了一种新颖的双重集成框架，该框架协同结合预训练深度学习（DL）模型进行特征提取与优化后的机器学习（ML）分类器进行稳健分类。此框架包括全面处理大脑磁共振图像（MRI）的预处理和数据增强步骤，并使用迁移学习从预训练视觉变换器（ViT）网络中提取深层特征。创新之处在于双层集成策略：在特征级融合来自顶级性能ViT模型的深度特征，以及在分类器级别通过超参数优化后的ML分类器进行预测聚合。 实验基于两个公共Kaggle MRI脑肿瘤数据集显示了该方法显著超越现有最先进的方法，强调了特征和分类器融合的重要性。所提出的方法还突显了超参数优化（HPO）和高级预处理技术在提高诊断准确性和可靠性中的关键作用，推进了深度学习和机器学习在临床相关医学图像分析中的一体化应用。
URL
https://arxiv.org/abs/2506.12363
PDF
-
PromptTSS: A Prompting-Based Approach for Interactive Multi-Granularity Time Series Segmentation
2025-06-12 06:45:29Ching Chang, Ming-Chih Lo, Wen-Chih Peng, Tien-Fu ChenAbstract
Multivariate time series data, collected across various fields such as manufacturing and wearable technology, exhibit states at multiple levels of granularity, from coarse-grained system behaviors to fine-grained, detailed events. Effectively segmenting and integrating states across these different granularities is crucial for tasks like predictive maintenance and performance optimization. However, existing time series segmentation methods face two key challenges: (1) the inability to handle multiple levels of granularity within a unified model, and (2) limited adaptability to new, evolving patterns in dynamic environments. To address these challenges, we propose PromptTSS, a novel framework for time series segmentation with multi-granularity states. PromptTSS uses a unified model with a prompting mechanism that leverages label and boundary information to guide segmentation, capturing both coarse- and fine-grained patterns while adapting dynamically to unseen patterns. Experiments show PromptTSS improves accuracy by 24.49% in multi-granularity segmentation, 17.88% in single-granularity segmentation, and up to 599.24% in transfer learning, demonstrating its adaptability to hierarchical states and evolving time series dynamics.
Abstract (translated)
多变量时间序列数据在制造和可穿戴技术等各个领域收集，这些数据展示了从粗粒度的系统行为到细粒度的详细事件等多种级别的状态。有效地对不同层次精细程度的状态进行分割和整合对于任务如预测性维护和性能优化至关重要。然而，现有的时间序列分割方法面临两个关键挑战：（1）无法在统一模型中处理多个级别精细度的问题；（2）适应动态环境中新出现、不断变化的模式的能力有限。 为了解决这些挑战，我们提出了PromptTSS框架，这是一种用于具有多级状态的时间序列分割的新颖方法。PromptTSS使用了一种统一模型，并结合了提示机制来利用标签和边界信息引导时间序列的分割过程。这样不仅可以捕捉到粗粒度和细粒度的所有模式，而且还可以动态适应未见过的数据模式。 实验结果显示，与现有技术相比，PromptTSS在多级分割中提高了24.49%的准确性，在单级分割中提高了17.88%，并且在迁移学习中的表现最高提升了599.24%。这些结果表明该框架能够适应分层状态和不断变化的时间序列动态，并具有强大的可扩展性。
URL
https://arxiv.org/abs/2506.11170
PDF
-
Uncertainty-Aware Deep Learning for Automated Skin Cancer Classification: A Comprehensive Evaluation
2025-06-12 02:29:16Hamzeh Asgharnezhad, Pegah Tabarisaadi, Abbas Khosravi, Roohallah Alizadehsani, U. Rajendra AcharyaAbstract
Accurate and reliable skin cancer diagnosis is critical for early treatment and improved patient outcomes. Deep learning (DL) models have shown promise in automating skin cancer classification, but their performance can be limited by data scarcity and a lack of uncertainty awareness. In this study, we present a comprehensive evaluation of DL-based skin lesion classification using transfer learning and uncertainty quantification (UQ) on the HAM10000 dataset. In the first phase, we benchmarked several pre-trained feature extractors-including Contrastive Language-Image Pretraining (CLIP) variants, Residual Network-50 (ResNet50), Densely Connected Convolutional Network (DenseNet121), Visual Geometry Group network (VGG16), and EfficientNet-V2-Large-combined with a range of traditional classifiers such as Support Vector Machine (SVM), eXtreme Gradient Boosting (XGBoost), and logistic regression. Our results show that CLIP-based vision transformers, particularly LAION CLIP ViT-H/14 with SVM, deliver the highest classification performance. In the second phase, we incorporated UQ using Monte Carlo Dropout (MCD), Ensemble, and Ensemble Monte Carlo Dropout (EMCD) to assess not only prediction accuracy but also the reliability of model outputs. We evaluated these models using uncertainty-aware metrics such as uncertainty accuracy(UAcc), uncertainty sensitivity(USen), uncertainty specificity(USpe), and uncertainty precision(UPre). The results demonstrate that ensemble methods offer a good trade-off between accuracy and uncertainty handling, while EMCD is more sensitive to uncertain predictions. This study highlights the importance of integrating UQ into DL-based medical diagnosis to enhance both performance and trustworthiness in real-world clinical applications.
Abstract (translated)
准确且可靠的皮肤癌诊断对于早期治疗和改善患者预后至关重要。深度学习（DL）模型在自动化皮肤癌分类方面显示出前景，但其性能可能因数据稀缺及不确定性意识不足而受限。本研究中，我们使用迁移学习和不确定性量化（UQ）对HAM10000数据集进行了基于深度学习的皮肤病变分类的全面评估。 第一阶段，我们基准测试了几种预训练特征提取器，包括对比语言图像预训练（CLIP）变体、残差网络-50 (ResNet50)、密集连接卷积网络(DenseNet121)、视觉几何组网络(VGG16)，以及EfficientNet-V2-Large，并将其与传统的分类器如支持向量机(SVM)、极端梯度提升(XGBoost)和逻辑回归相结合。结果显示，基于CLIP的视觉变换器（特别是LAION CLIP ViT-H/14 与 SVM结合）提供了最高的分类性能。 第二阶段，我们使用蒙特卡洛丢弃法(MCD)、集成学习以及集成蒙特卡洛丢弃法(EMCD)，将不确定性量化应用于模型中，以评估不仅预测准确性还包括模型输出的可靠性。我们使用了不确定度准确率(UAcc)、不确定度敏感性(USen)、不确定度特异性(USpe)和不确定度精确度(UPre)等基于不确定性的指标来评价这些模型。结果显示，集成方法在精度与处理不确定性之间提供了良好的权衡，而EMCD对于处理不确定预测更为灵敏。 本研究强调了将不确定性量化整合到基于深度学习的医学诊断中以提高实际临床应用中的性能和可信度的重要性。
URL
https://arxiv.org/abs/2506.10302
PDF
-
From Token to Rhythm: A Multi-Scale Approach for ECG-Language Pretraining
2025-06-11 07:22:17Fuying Wang, Jiacheng Xu, Lequan YuAbstract
Electrocardiograms (ECGs) play a vital role in monitoring cardiac health and diagnosing heart diseases. However, traditional deep learning approaches for ECG analysis rely heavily on large-scale manual annotations, which are both time-consuming and resource-intensive to obtain. To overcome this limitation, self-supervised learning (SSL) has emerged as a promising alternative, enabling the extraction of robust ECG representations that can be efficiently transferred to various downstream tasks. While previous studies have explored SSL for ECG pretraining and multi-modal ECG-language alignment, they often fail to capture the multi-scale nature of ECG signals. As a result, these methods struggle to learn generalized representations due to their inability to model the hierarchical structure of ECG data. To address this gap, we introduce MELP, a novel Multi-scale ECG-Language Pretraining (MELP) model that fully leverages hierarchical supervision from ECG-text pairs. MELP first pretrains a cardiology-specific language model to enhance its understanding of clinical text. It then applies three levels of cross-modal supervision-at the token, beat, and rhythm levels-to align ECG signals with textual reports, capturing structured information across different time scales. We evaluate MELP on three public ECG datasets across multiple tasks, including zero-shot ECG classification, linear probing, and transfer learning. Experimental results demonstrate that MELP outperforms existing SSL methods, underscoring its effectiveness and adaptability across diverse clinical applications. Our code is available at this https URL.
Abstract (translated)
心电图（ECG）在监测心脏健康和诊断心脏病方面发挥着重要作用。然而，传统的深度学习方法在分析心电图时依赖于大规模的手动注释，这既耗时又耗费资源。为了解决这一限制，自监督学习（SSL）作为一种有前景的替代方案已经出现，它能够提取出鲁棒的心电图表示，并高效地应用于各种下游任务中。尽管以往的研究探索了心电图预训练和多模态心电信号-语言对齐方面的SSL方法，但它们往往未能捕捉到心电图信号的多层次特性。因此，由于无法建模心电数据中的层次结构，这些方法难以学习到泛化的表示形式。 为了解决这一不足，我们引入了一种新的模型MELP（多尺度心电图-语言预训练），该模型充分利用了来自心电图文本对的分层监督信号。首先，MELP预先训练了一个心脏病学特定的语言模型以增强其理解和分析临床文本的能力。然后，在标记、节律和节奏三个层次上应用跨模态监督，将心电信号与文字报告相匹配，从而捕获不同时间尺度上的结构化信息。 我们在三个公开的心电图数据集的多个任务上对MELP进行了评估，包括零样本心电图分类、线性探测以及迁移学习。实验结果显示，MELP优于现有的SSL方法，在各种临床应用中表现出强大的有效性和适应能力。我们的代码可在该链接提供。 这段文本描述了一种新的自监督学习模型——多尺度心电图-语言预训练（MELP），用于改进对心电信号的分析和理解，并在多个任务上证明了其优越性。
URL
https://arxiv.org/abs/2506.21803
PDF
-
An Effective End-to-End Solution for Multimodal Action Recognition
2025-06-11 02:54:02Songping Wang, Xiantao Hu, Yueming Lyu, Caifeng ShanAbstract
Recently, multimodal tasks have strongly advanced the field of action recognition with their rich multimodal information. However, due to the scarcity of tri-modal data, research on tri-modal action recognition tasks faces many challenges. To this end, we have proposed a comprehensive multimodal action recognition solution that effectively utilizes multimodal information. First, the existing data are transformed and expanded by optimizing data enhancement techniques to enlarge the training scale. At the same time, more RGB datasets are used to pre-train the backbone network, which is better adapted to the new task by means of transfer learning. Secondly, multimodal spatial features are extracted with the help of 2D CNNs and combined with the Temporal Shift Module (TSM) to achieve multimodal spatial-temporal feature extraction comparable to 3D CNNs and improve the computational efficiency. In addition, common prediction enhancement methods, such as Stochastic Weight Averaging (SWA), Ensemble and Test-Time augmentation (TTA), are used to integrate the knowledge of models from different training periods of the same architecture and different architectures, so as to predict the actions from different perspectives and fully exploit the target information. Ultimately, we achieved the Top-1 accuracy of 99% and the Top-5 accuracy of 100% on the competition leaderboard, demonstrating the superiority of our solution.
Abstract (translated)
最近，多模态任务通过其丰富的信息促进了动作识别领域的进步。然而，由于三模态数据的稀缺性，对三模态动作识别任务的研究面临着许多挑战。为此，我们提出了一种全面的多模态动作识别解决方案，能够有效利用多模态信息。首先，通过对现有数据进行优化的数据增强技术来转换和扩展这些数据，以扩大训练规模。同时，使用更多的RGB数据集预训练骨干网络，并通过迁移学习将其更好地适应新任务。其次，在2D CNNs的帮助下提取多模态空间特征，并与时间偏移模块（TSM）结合，实现与3D CNNs相媲美的多模态时空特征提取，从而提高计算效率。此外，还采用了诸如随机权重平均(SWA)、集成和测试时增强(TTA)等常见的预测增强方法，以整合相同架构的不同训练时期以及不同架构的模型的知识，从不同的角度进行动作预测，并充分挖掘目标信息。最终，在竞赛排行榜上我们达到了99%的第一准确率和100%的前五准确率，展示了我们的解决方案的优势。
URL
https://arxiv.org/abs/2506.09345
PDF
-
An Explainable Deep Learning Framework for Brain Stroke and Tumor Progression via MRI Interpretation
2025-06-10 18:19:56Rajan Das Gupta, Md Imrul Hasan Showmick, Mushfiqur Rahman Abir, Shanjida Akter, Md. Yeasin Rahat, Md. Jakir HossenAbstract
Early and accurate detection of brain abnormalities, such as tumors and strokes, is essential for timely intervention and improved patient outcomes. In this study, we present a deep learning-based system capable of identifying both brain tumors and strokes from MRI images, along with their respective stages. We have executed two groundbreaking strategies involving convolutional neural networks, MobileNet V2 and ResNet-50-optimized through transfer learning to classify MRI scans into five diagnostic categories. Our dataset, aggregated and augmented from various publicly available MRI sources, was carefully curated to ensure class balance and image diversity. To enhance model generalization and prevent overfitting, we applied dropout layers and extensive data augmentation. The models achieved strong performance, with training accuracy reaching 93\% and validation accuracy up to 88\%. While ResNet-50 demonstrated slightly better results, Mobile Net V2 remains a promising option for real-time diagnosis in low resource settings due to its lightweight architecture. This research offers a practical AI-driven solution for early brain abnormality detection, with potential for clinical deployment and future enhancement through larger datasets and multi modal inputs.
Abstract (translated)
早期且准确地检测大脑异常，如肿瘤和中风，对于及时干预及改善患者预后至关重要。在本研究中，我们介绍了一种基于深度学习的系统，能够从MRI图像中识别脑瘤和中风，并确定它们各自的阶段。我们执行了两项突破性策略：采用经过迁移学习优化的卷积神经网络（CNN）MobileNet V2 和 ResNet-50 来将 MRI 扫描分类为五个诊断类别。我们的数据集是从各种公开来源汇集并扩增而来的MRI图像，为了确保类平衡和图像多样性，我们精心策划了该数据集。 为了增强模型的泛化能力和防止过拟合，我们在训练过程中应用了dropout层和广泛的图像增强技术。模型在性能上表现出色，训练准确率达到了93%，验证准确率达到88%。虽然ResNet-50展示了稍好一些的结果，但轻量级架构的Mobile Net V2 在低资源环境下的实时诊断中仍然具有很大的潜力。 这项研究为早期大脑异常检测提供了实用的人工智能解决方案，并且通过更大规模的数据集和多模态输入进一步优化后，在临床部署方面具有巨大的应用前景。
URL
https://arxiv.org/abs/2506.09161
PDF
-
Do MIL Models Transfer?
2025-06-10 17:50:04Daniel Shao, Richard J. Chen, Andrew H. Song, Joel Runevic, Ming Y. Lu, Tong Ding, Faisal MahmoodAbstract
Multiple Instance Learning (MIL) is a cornerstone approach in computational pathology (CPath) for generating clinically meaningful slide-level embeddings from gigapixel tissue images. However, MIL often struggles with small, weakly supervised clinical datasets. In contrast to fields such as NLP and conventional computer vision, where transfer learning is widely used to address data scarcity, the transferability of MIL models remains poorly understood. In this study, we systematically evaluate the transfer learning capabilities of pretrained MIL models by assessing 11 models across 21 pretraining tasks for morphological and molecular subtype prediction. Our results show that pretrained MIL models, even when trained on different organs than the target task, consistently outperform models trained from scratch. Moreover, pretraining on pancancer datasets enables strong generalization across organs and tasks, outperforming slide foundation models while using substantially less pretraining data. These findings highlight the robust adaptability of MIL models and demonstrate the benefits of leveraging transfer learning to boost performance in CPath. Lastly, we provide a resource which standardizes the implementation of MIL models and collection of pretrained model weights on popular CPath tasks, available at this https URL
Abstract (translated)
多示例学习（MIL）是计算病理学（CPath）中的一个基石方法，用于从数十亿像素的组织图像中生成具有临床意义的幻灯片级别的嵌入。然而，在小规模、弱监督的临床数据集上，MIL常常面临挑战。与自然语言处理（NLP）和传统计算机视觉等领域不同，这些领域广泛使用迁移学习来解决数据稀缺问题，关于多示例学习模型的可迁移性知之甚少。 在本研究中，我们通过评估11个预训练模型在21项预训练任务中的表现来系统地评价了预训练MIL模型的迁移学习能力。这些任务涵盖形态和分子亚型预测。我们的结果显示，即使是在与目标任务不同的器官上进行训练，预训练过的MIL模型也始终优于从头开始训练的模型。此外，在泛癌症数据集上的预训练能够实现跨不同器官和任务的强大泛化性能，并且在使用显著较少的预训练数据的情况下超过了幻灯片基础模型。 这些发现强调了MIL模型的稳健适应性，展示了利用迁移学习提高CPath中性能的好处。最后，我们提供了一个资源，用于标准化流行CPath任务中的多示例学习模型实现和收集预训练模型权重，可在提供的网址上获取：[此链接]（请确保将实际URL替换为具体的可用链接）。
URL
https://arxiv.org/abs/2506.09022
PDF
-
Data-Efficient Challenges in Visual Inductive Priors: A Retrospective
2025-06-10 09:21:48Robert-Jan Bruintjes, Attila Lengyel, Osman Semih Kayhan, Davide Zambrano, Nergis T\"omen, Hadi Jamali-Rad, Jan van GemertAbstract
Deep Learning requires large amounts of data to train models that work well. In data-deficient settings, performance can be degraded. We investigate which Deep Learning methods benefit training models in a data-deficient setting, by organizing the "VIPriors: Visual Inductive Priors for Data-Efficient Deep Learning" workshop series, featuring four editions of data-impaired challenges. These challenges address the problem of training deep learning models for computer vision tasks with limited data. Participants are limited to training models from scratch using a low number of training samples and are not allowed to use any form of transfer learning. We aim to stimulate the development of novel approaches that incorporate prior knowledge to improve the data efficiency of deep learning models. Successful challenge entries make use of large model ensembles that mix Transformers and CNNs, as well as heavy data augmentation. Novel prior knowledge-based methods contribute to success in some entries.
Abstract (translated)
深度学习需要大量的数据来训练表现良好的模型。在数据稀缺的情况下，性能会受到影响。我们通过组织“VIPriors：用于高效深度学习的视觉归纳先验”研讨会系列（共四届），探讨了哪些深度学习方法有助于在数据不足的情况下训练模型。这些挑战旨在解决使用有限的数据训练计算机视觉任务中深度学习模型的问题。参赛者被限制只能从零开始使用少量的训练样本进行建模，并且不允许任何形式的迁移学习。我们的目标是激发创新的方法，这些方法可以利用先验知识来提高深度学习模型在数据效率方面的能力。成功的挑战入围作品采用了大型模型集成（结合了Transformer和CNN），以及大量的数据增强技术。基于新颖先验知识的方法也在一些参赛作品的成功中起到了关键作用。
URL
https://arxiv.org/abs/2506.08612
PDF
-
Robust Evolutionary Multi-Objective Network Architecture Search for Reinforcement Learning
2025-06-10 07:52:35Nihal Acharya Adde, Alexandra Gianzina, Hanno Gottschalk, Andreas EbertAbstract
This paper introduces Evolutionary Multi-Objective Network Architecture Search (EMNAS) for the first time to optimize neural network architectures in large-scale Reinforcement Learning (RL) for Autonomous Driving (AD). EMNAS uses genetic algorithms to automate network design, tailored to enhance rewards and reduce model size without compromising performance. Additionally, parallelization techniques are employed to accelerate the search, and teacher-student methodologies are implemented to ensure scalable optimization. This research underscores the potential of transfer learning as a robust framework for optimizing performance across iterative learning processes by effectively leveraging knowledge from earlier generations to enhance learning efficiency and stability in subsequent generations. Experimental results demonstrate that tailored EMNAS outperforms manually designed models, achieving higher rewards with fewer parameters. The findings of these strategies contribute positively to EMNAS for RL in autonomous driving, advancing the field toward better-performing networks suitable for real-world scenarios.
Abstract (translated)
本文首次介绍了进化多目标网络架构搜索（EMNAS），用于优化大规模强化学习（RL）中自动驾驶（AD）的神经网络架构。EMNAS利用遗传算法自动化网络设计，旨在提升奖励并减小模型规模而不影响性能。此外，还采用了并行化技术以加速搜索过程，并实施了师生方法确保可扩展性优化。这项研究强调了迁移学习作为迭代学习过程中优化性能的强大框架潜力，通过有效利用早期代的知识来提高后续代的学习效率和稳定性。实验结果表明，定制的EMNAS优于手动设计的模型，在参数更少的情况下获得更高的奖励。这些策略的研究成果对增强EMNAS在自动驾驶强化学习中的应用贡献积极，并推动该领域向更适合现实场景的高性能网络发展。
URL
https://arxiv.org/abs/2506.08533
PDF
-
CrosswalkNet: An Optimized Deep Learning Framework for Pedestrian Crosswalk Detection in Aerial Images with High-Performance Computing
2025-06-09 15:56:24Zubin Bhuyan, Yuanchang Xie, AngkeaReach Rith, Xintong Yan, Nasko Apostolov, Jimi Oke, Chengbo AiAbstract
With the increasing availability of aerial and satellite imagery, deep learning presents significant potential for transportation asset management, safety analysis, and urban planning. This study introduces CrosswalkNet, a robust and efficient deep learning framework designed to detect various types of pedestrian crosswalks from 15-cm resolution aerial images. CrosswalkNet incorporates a novel detection approach that improves upon traditional object detection strategies by utilizing oriented bounding boxes (OBB), enhancing detection precision by accurately capturing crosswalks regardless of their orientation. Several optimization techniques, including Convolutional Block Attention, a dual-branch Spatial Pyramid Pooling-Fast module, and cosine annealing, are implemented to maximize performance and efficiency. A comprehensive dataset comprising over 23,000 annotated crosswalk instances is utilized to train and validate the proposed framework. The best-performing model achieves an impressive precision of 96.5% and a recall of 93.3% on aerial imagery from Massachusetts, demonstrating its accuracy and effectiveness. CrosswalkNet has also been successfully applied to datasets from New Hampshire, Virginia, and Maine without transfer learning or fine-tuning, showcasing its robustness and strong generalization capability. Additionally, the crosswalk detection results, processed using High-Performance Computing (HPC) platforms and provided in polygon shapefile format, have been shown to accelerate data processing and detection, supporting real-time analysis for safety and mobility applications. This integration offers policymakers, transportation engineers, and urban planners an effective instrument to enhance pedestrian safety and improve urban mobility.
Abstract (translated)
随着高空和卫星图像的日益普及，深度学习在交通资产管理、安全分析和城市规划方面展现出了巨大的潜力。本文介绍了CrosswalkNet，这是一个旨在从15厘米分辨率的航空影像中检测各种类型人行横道的强大且高效的深度学习框架。 CrosswalkNet引入了一种新的检测方法，通过使用定向边界框（OBB）改进了传统的物体检测策略，从而提高了检测精度，无论人行横道的方向如何都能准确捕捉。为了最大化性能和效率，采用了多种优化技术，包括卷积块注意力、双分支空间金字塔池化快速模块以及余弦退火。 研究团队利用了一个包含超过23,000个人行横道路段注释的全面数据集来训练和验证所提出的框架。最佳模型在马萨诸塞州航空影像上达到了令人印象深刻的96.5%精度和93.3%召回率，证明了其准确性和有效性。 此外，CrosswalkNet无需转移学习或微调，已成功应用于新罕布什尔州、弗吉尼亚州和缅因州的数据集，展示了它的稳健性及强大的泛化能力。人行横道检测结果在使用高性能计算（HPC）平台处理后，以多边形shapefile格式提供，这大大加快了数据处理速度，并支持实时安全性和移动应用分析。 该集成向政策制定者、交通工程师和城市规划师提供了一个有效的工具，用于增强行人安全并改善城市流动性。
URL
https://arxiv.org/abs/2506.07885
PDF
-
Transfer Learning and Explainable AI for Brain Tumor Classification: A Study Using MRI Data from Bangladesh
2025-06-08 17:26:13Shuvashis SarkerAbstract
Brain tumors, regardless of being benign or malignant, pose considerable health risks, with malignant tumors being more perilous due to their swift and uncontrolled proliferation, resulting in malignancy. Timely identification is crucial for enhancing patient outcomes, particularly in nations such as Bangladesh, where healthcare infrastructure is constrained. Manual MRI analysis is arduous and susceptible to inaccuracies, rendering it inefficient for prompt diagnosis. This research sought to tackle these problems by creating an automated brain tumor classification system utilizing MRI data obtained from many hospitals in Bangladesh. Advanced deep learning models, including VGG16, VGG19, and ResNet50, were utilized to classify glioma, meningioma, and various brain cancers. Explainable AI (XAI) methodologies, such as Grad-CAM and Grad-CAM++, were employed to improve model interpretability by emphasizing the critical areas in MRI scans that influenced the categorization. VGG16 achieved the most accuracy, attaining 99.17%. The integration of XAI enhanced the system's transparency and stability, rendering it more appropriate for clinical application in resource-limited environments such as Bangladesh. This study highlights the capability of deep learning models, in conjunction with explainable artificial intelligence (XAI), to enhance brain tumor detection and identification in areas with restricted access to advanced medical technologies.
Abstract (translated)
脑肿瘤，无论是良性的还是恶性的，都会带来重大的健康风险。恶性肿瘤由于其快速且不受控制的增殖特性更加危险，并可能导致癌症扩散。及时识别是提高患者预后效果的关键，在诸如孟加拉国等医疗基础设施有限的国家更是如此。手动分析MRI图像既艰难又容易产生误差，这使得即时诊断变得低效。这项研究旨在通过利用来自多个医院的大量MRI数据创建一个自动化的脑肿瘤分类系统来解决这些问题。使用包括VGG16、VGG19和ResNet50在内的先进深度学习模型对胶质瘤、脑膜瘤及其他多种脑癌进行分类。此外，采用可解释的人工智能（XAI）方法如Grad-CAM和Grad-CAM++以强调MRI扫描中影响分类的关键区域，从而提高模型的透明度与理解性。 VGG16实现了最高的准确率，达到了99.17%。结合使用XAI技术增强了系统的透明性和稳定性，在资源有限的环境中更加适合临床应用。这项研究表明，通过将深度学习模型和可解释的人工智能（XAI）相结合，可以在缺乏先进医疗科技手段的情况下显著提高脑肿瘤检测与识别的能力。
URL
https://arxiv.org/abs/2506.07228
PDF
-
Exploring Image Transforms derived from Eye Gaze Variables for Progressive Autism Diagnosis
2025-06-07 19:00:17Abigail Copiaco, Christian Ritz, Yassine Himeur, Valsamma Eapen, Ammar Albanna, Wathiq MansoorAbstract
The prevalence of Autism Spectrum Disorder (ASD) has surged rapidly over the past decade, posing significant challenges in communication, behavior, and focus for affected individuals. Current diagnostic techniques, though effective, are time-intensive, leading to high social and economic costs. This work introduces an AI-powered assistive technology designed to streamline ASD diagnosis and management, enhancing convenience for individuals with ASD and efficiency for caregivers and therapists. The system integrates transfer learning with image transforms derived from eye gaze variables to diagnose ASD. This facilitates and opens opportunities for in-home periodical diagnosis, reducing stress for individuals and caregivers, while also preserving user privacy through the use of image transforms. The accessibility of the proposed method also offers opportunities for improved communication between guardians and therapists, ensuring regular updates on progress and evolving support needs. Overall, the approach proposed in this work ensures timely, accessible diagnosis while protecting the subjects' privacy, improving outcomes for individuals with ASD.
Abstract (translated)
自闭症谱系障碍（ASD）的发病率在过去十年中迅速上升，给受影响个体在沟通、行为和注意力方面带来了重大挑战。尽管当前的诊断技术有效，但耗时较长，导致了较高的社会和经济成本。本文介绍了一种由人工智能驱动的辅助技术，旨在简化自闭症谱系障碍（ASD）的诊断与管理过程，提高患者的便利性，并增强照护者和治疗师的工作效率。该系统结合转移学习与基于眼动变量的图像变换来诊断 ASD，这有助于并为家庭环境中的定期诊断提供了机会，从而减轻患者及照顾者的压力，同时通过使用图像变换保护用户隐私。所提出方法的可访问性也为监护人与治疗师之间的沟通改善带来了机遇，确保了进展和不断变化的支持需求得到及时更新。总体而言，本文提出的方案保证了及时、便捷且尊重受试者隐私的诊断方式，从而提高了自闭症个体的整体结果。
URL
https://arxiv.org/abs/2506.09065
PDF
-
Exploring Visual Prompting: Robustness Inheritance and Beyond
2025-06-07 14:56:32Qi Li, Liangzhi Li, Zhouqiang Jiang, Bowen Wang, Keke TangAbstract
Visual Prompting (VP), an efficient method for transfer learning, has shown its potential in vision tasks. However, previous works focus exclusively on VP from standard source models, it is still unknown how it performs under the scenario of a robust source model: Can the robustness of the source model be successfully inherited? Does VP also encounter the same trade-off between robustness and generalization ability as the source model during this process? If such a trade-off exists, is there a strategy specifically tailored to VP to mitigate this limitation? In this paper, we thoroughly explore these three questions for the first time and provide affirmative answers to them. To mitigate the trade-off faced by VP, we propose a strategy called Prompt Boundary Loosening (PBL). As a lightweight, plug-and-play strategy naturally compatible with VP, PBL effectively ensures the successful inheritance of robustness when the source model is a robust model, while significantly enhancing VP's generalization ability across various downstream datasets. Extensive experiments across various datasets show that our findings are universal and demonstrate the significant benefits of the proposed strategy.
Abstract (translated)
视觉提示法（VP）作为一种高效的迁移学习方法，在视觉任务中展示了其潜力。然而，之前的工作仅关注从标准源模型进行的VP，对于使用鲁棒性更强的源模型时VP的表现尚不清楚：源模型的鲁棒性能否被成功继承？在这一过程中，VP是否会像源模型一样面临鲁棒性和泛化能力之间的权衡？如果这种权衡确实存在，是否有一种专门针对VP的方法来缓解此限制？ 本文首次全面探讨了这三个问题，并给出了肯定的回答。为了缓解VP面临的权衡，我们提出了一种名为“提示边界放松”（PBL）的策略。作为一种轻量级、即插即用且与VP自然兼容的策略，PBL在源模型为鲁棒性模型时有效确保了其鲁棒性的成功继承，并显著提高了VP在各种下游数据集上的泛化能力。跨多个数据集进行的广泛实验表明，我们的发现具有普遍适用性，并展示了所提出策略的重大益处。
URL
https://arxiv.org/abs/2506.06823
PDF
-
The OCR Quest for Generalization: Learning to recognize low-resource alphabets with model editing
2025-06-07 11:05:33Adri\`a Molina Rodr\'iguez, Oriol Ramos Terrades, Josep Llad\'osAbstract
Achieving robustness in recognition systems across diverse domains is crucial for their practical utility. While ample data availability is usually assumed, low-resource languages, such as ancient manuscripts and non-western languages, tend to be kept out of the equations of massive pretraining and foundational techniques due to an under representation. In this work, we aim for building models which can generalize to new distributions of data, such as alphabets, faster than centralized fine-tune strategies. For doing so, we take advantage of the recent advancements in model editing to enhance the incorporation of unseen scripts (low-resource learning). In contrast to state-of-the-art meta-learning, we showcase the effectiveness of domain merging in sparse distributions of data, with agnosticity of its relation to the overall distribution or any other prototyping necessity. Even when using the same exact training data, our experiments showcase significant performance boosts in \textbf{transfer learning} to new alphabets and \textbf{out-of-domain evaluation} in challenging domain shifts, including historical ciphered texts and non-Latin scripts. This research contributes a novel approach into building models that can easily adopt under-represented alphabets and, therefore, enable document recognition to a wider set of contexts and cultures.
Abstract (translated)
在不同领域的识别系统中实现鲁棒性对于其实用性至关重要。虽然通常假设数据充足，但像古代手稿和非西方语言这样的低资源语言由于代表性不足，在大规模预训练和基础技术中的应用较少。本研究旨在构建能够在新数据分布（如字母表）上更快泛化的模型，其速度超过集中式微调策略。为此，我们利用最近的模型编辑进展来增强未见过脚本（低资源学习）的融入。 与当前最先进的元学习方法不同，我们的工作展示了领域合并在稀疏数据分布中的有效性，并且对总体分布或任何原型需求无关联性假设。即使使用相同的训练数据，在迁移学习到新字母表以及面对具有挑战性的域转变（包括历史加密文本和非拉丁文脚本）的域外评估中，实验结果也显示了显著的性能提升。 这项研究贡献了一种新颖的方法来构建能够轻松适应代表性不足的语言文字的模型，并因此使文档识别能够在更广泛的背景和文化情境下得以应用。
URL
https://arxiv.org/abs/2506.06761
PDF
-
Textile Analysis for Recycling Automation using Transfer Learning and Zero-Shot Foundation Models
2025-06-06 22:49:53Yannis Spyridis, Vasileios ArgyriouAbstract
Automated sorting is crucial for improving the efficiency and scalability of textile recycling, but accurately identifying material composition and detecting contaminants from sensor data remains challenging. This paper investigates the use of standard RGB imagery, a cost-effective sensing modality, for key pre-processing tasks in an automated system. We present computer vision components designed for a conveyor belt setup to perform (a) classification of four common textile types and (b) segmentation of non-textile features such as buttons and zippers. For classification, several pre-trained architectures were evaluated using transfer learning and cross-validation, with EfficientNetB0 achieving the best performance on a held-out test set with 81.25\% accuracy. For feature segmentation, a zero-shot approach combining the Grounding DINO open-vocabulary detector with the Segment Anything Model (SAM) was employed, demonstrating excellent performance with a mIoU of 0.90 for the generated masks against ground truth. This study demonstrates the feasibility of using RGB images coupled with modern deep learning techniques, including transfer learning for classification and foundation models for zero-shot segmentation, to enable essential analysis steps for automated textile recycling pipelines.
Abstract (translated)
自动化分拣对于提高纺织品回收的效率和规模至关重要，但准确识别材料组成并从传感器数据中检测污染物仍然是一个挑战。本文研究了使用标准RGB图像（一种成本效益高的传感模式）在自动系统中的关键预处理任务的应用。我们提出了一种计算机视觉组件，专门用于传送带设置，以执行(a)四种常见纺织品类型的分类和(b)非纺织特征（如纽扣和拉链）的分割。 对于分类任务，使用迁移学习和交叉验证评估了几种预训练架构，在保留的测试集上，EfficientNetB0模型实现了最佳性能，准确率为81.25%。对于特征分割，采用了一种零样本方法，结合了Grounding DINO开放词汇检测器与Segment Anything Model（SAM），生成的掩码在与真实标签对比时达到了mIoU为0.90的优秀性能。 这项研究表明，通过结合现代深度学习技术——包括迁移学习用于分类和基础模型用于零样本分割——利用RGB图像可以实现自动化纺织品回收管道所需的关键分析步骤。这种方法展示了使用RGB图像进行有效分拣的可能性，并为进一步提高效率提供了途径。
URL
https://arxiv.org/abs/2506.06569
PDF
