Contour
-
WMamba: Wavelet-based Mamba for Face Forgery Detection
2025-01-16 15:44:24Siran Peng, Tianshuo Zhang, Li Gao, Xiangyu Zhu, Haoyuan Zhang, Kai Pang, Zhen LeiAbstract
With the rapid advancement of deepfake generation technologies, the demand for robust and accurate face forgery detection algorithms has become increasingly critical. Recent studies have demonstrated that wavelet analysis can uncover subtle forgery artifacts that remain imperceptible in the spatial domain. Wavelets effectively capture important facial contours, which are often slender, fine-grained, and global in nature. However, existing wavelet-based approaches fail to fully leverage these unique characteristics, resulting in sub-optimal feature extraction and limited generalizability. To address this challenge, we introduce WMamba, a novel wavelet-based feature extractor built upon the Mamba architecture. WMamba maximizes the utility of wavelet information through two key innovations. First, we propose Dynamic Contour Convolution (DCConv), which employs specially crafted deformable kernels to adaptively model slender facial contours. Second, by leveraging the Mamba architecture, our method captures long-range spatial relationships with linear computational complexity. This efficiency allows for the extraction of fine-grained, global forgery artifacts from small image patches. Extensive experimental results show that WMamba achieves state-of-the-art (SOTA) performance, highlighting its effectiveness and superiority in face forgery detection.
Abstract (translated)
随着深度伪造生成技术的迅速发展，对稳健且准确的人脸伪造检测算法的需求变得愈发重要。近期研究显示，小波分析能够揭示在空间域中难以察觉的细微伪造痕迹。小波变换能有效地捕捉到重要的面部轮廓特征，这些特征往往是纤细、精细和具有全局性的。然而，现有的基于小波的方法未能充分利用这些独特特性，导致了次优的特征提取效果和有限的应用广度。 为了解决这一挑战，我们引入了一种新型的小波基特征提取器——WMamba，它是基于Mamba架构设计的。WMamba通过两个关键创新最大化了小波信息的作用。首先，我们提出了动态轮廓卷积（DCConv），采用特制的可变形核来适应性地建模纤细的面部轮廓；其次，利用Mamba架构的优势，我们的方法能够在线性计算复杂度下捕捉长距离的空间关系。这种效率使得可以从小型图像块中提取出精细、全局性的伪造痕迹。 广泛的实验结果表明，WMamba达到了最先进的性能（SOTA），突显了其在人脸伪造检测中的有效性和优越性。
URL
https://arxiv.org/abs/2501.09617
PDF
-
Toward Realistic Camouflaged Object Detection: Benchmarks and Method
2025-01-13 13:04:00Zhimeng Xin, Tianxu Wu, Shiming Chen, Shuo Ye, Zijing Xie, Yixiong Zou, Xinge You, Yufei GuoAbstract
Camouflaged object detection (COD) primarily relies on semantic or instance segmentation methods. While these methods have made significant advancements in identifying the contours of camouflaged objects, they may be inefficient or cost-effective for tasks that only require the specific location of the object. Object detection algorithms offer an optimized solution for Realistic Camouflaged Object Detection (RCOD) in such cases. However, detecting camouflaged objects remains a formidable challenge due to the high degree of similarity between the features of the objects and their backgrounds. Unlike segmentation methods that perform pixel-wise comparisons to differentiate between foreground and background, object detectors omit this analysis, further aggravating the challenge. To solve this problem, we propose a camouflage-aware feature refinement (CAFR) strategy. Since camouflaged objects are not rare categories, CAFR fully utilizes a clear perception of the current object within the prior knowledge of large models to assist detectors in deeply understanding the distinctions between background and foreground. Specifically, in CAFR, we introduce the Adaptive Gradient Propagation (AGP) module that fine-tunes all feature extractor layers in large detection models to fully refine class-specific features from camouflaged contexts. We then design the Sparse Feature Refinement (SFR) module that optimizes the transformer-based feature extractor to focus primarily on capturing class-specific features in camouflaged scenarios. To facilitate the assessment of RCOD tasks, we manually annotate the labels required for detection on three existing segmentation COD datasets, creating a new benchmark for RCOD tasks. Code and datasets are available at: this https URL.
Abstract (translated)
伪装物体检测（COD）主要依赖于语义或实例分割方法。虽然这些方法在识别伪装物体的轮廓方面取得了显著进展，但对于仅需定位特定物体的任务来说，它们可能不够高效或经济实惠。在这种情况下，对象检测算法为现实中的伪装物体检测（RCOD）提供了优化解决方案。然而，由于物体特征与其背景的高度相似性，探测伪装物体仍然是一项艰巨挑战。与执行逐像素比较以区分前景和背景的分割方法不同，目标检测器忽略了这种分析，进一步加剧了这一挑战。为解决此问题，我们提出了一个“伪装感知特性细化”（CAFR）策略。由于伪装物体并不是罕见类别，CAFR充分利用大型模型中的现有知识来帮助探测器深入理解背景与前景之间的区别。 具体而言，在CAFR中，我们引入了一个自适应梯度传播（AGP）模块，它对大检测模型的所有特征提取层进行了微调，以完全细化来自伪装场景的特定类别的特性。然后设计了稀疏特征优化（SFR）模块，该模块针对基于变压器的特征提取器进行优化，使其主要专注于在伪装场景中捕捉特定类别的特性。 为了便于评估RCOD任务，我们在三个现有的分割COD数据集中手动标注了检测所需的标签，从而创建了一个新的基准测试。相关代码和数据集可在以下链接获得：this https URL.
URL
https://arxiv.org/abs/2501.07297
PDF
-
Evaluating Interval-based Tokenization for Pitch Representation in Symbolic Music Analysis
2025-01-08 17:22:03Dinh-Viet-Toan Le, Louis Bigo, Mikaela KellerAbstract
Symbolic music analysis tasks are often performed by models originally developed for Natural Language Processing, such as Transformers. Such models require the input data to be represented as sequences, which is achieved through a process of tokenization. Tokenization strategies for symbolic music often rely on absolute MIDI values to represent pitch information. However, music research largely promotes the benefit of higher-level representations such as melodic contour and harmonic relations for which pitch intervals turn out to be more expressive than absolute pitches. In this work, we introduce a general framework for building interval-based tokenizations. By evaluating these tokenizations on three music analysis tasks, we show that such interval-based tokenizations improve model performances and facilitate their explainability.
Abstract (translated)
符号音乐分析任务通常由为自然语言处理（如Transformer模型）开发的模型来执行。这些模型要求输入数据以序列形式表示，这是通过分词过程实现的。对于符号音乐来说，分词策略往往依赖于绝对MIDI值来表示音高信息。然而，音乐研究广泛认为更高层次的表示方式，例如旋律轮廓和和声关系，比绝对音高低效得多。这些高层次的表示中，音程（即两个音符之间的间隔）更具表现力。 在本文工作中，我们提出了一种基于音程构建分词策略的通用框架。通过评估这些音程分词策略在三项音乐分析任务中的性能，我们表明这种基于音程的方法能够提升模型的表现，并有助于其可解释性。
URL
https://arxiv.org/abs/2501.04630
PDF
-
DrawSpeech: Expressive Speech Synthesis Using Prosodic Sketches as Control Conditions
2025-01-08 03:47:54Weidong Chen, Shan Yang, Guangzhi Li, Xixin WuAbstract
Controlling text-to-speech (TTS) systems to synthesize speech with the prosodic characteristics expected by users has attracted much attention. To achieve controllability, current studies focus on two main directions: (1) using reference speech as prosody prompt to guide speech synthesis, and (2) using natural language descriptions to control the generation process. However, finding reference speech that exactly contains the prosody that users want to synthesize takes a lot of effort. Description-based guidance in TTS systems can only determine the overall prosody, which has difficulty in achieving fine-grained prosody control over the synthesized speech. In this paper, we propose DrawSpeech, a sketch-conditioned diffusion model capable of generating speech based on any prosody sketches drawn by users. Specifically, the prosody sketches are fed to DrawSpeech to provide a rough indication of the expected prosody trends. DrawSpeech then recovers the detailed pitch and energy contours based on the coarse sketches and synthesizes the desired speech. Experimental results show that DrawSpeech can generate speech with a wide variety of prosody and can precisely control the fine-grained prosody in a user-friendly manner. Our implementation and audio samples are publicly available.
Abstract (translated)
控制文本到语音（TTS）系统，使其生成具有用户期望的韵律特征的语音，已经吸引了广泛的关注。为了实现这种可控性，目前的研究主要集中在两个方向：(1) 使用参考语音作为韵律提示来指导语音合成；(2) 使用自然语言描述来控制生成过程。然而，找到包含用户希望合成的特定韵律特性的参考语音需要大量努力。基于描述的方法在TTS系统中只能确定整体的韵律特征，在实现对合成语音细粒度的韵律控制方面存在困难。 在本文中，我们提出了一种名为DrawSpeech的新模型，这是一种可以根据用户绘制的任何韵律草图生成语音的扩散模型。具体来说，用户的韵律草图被输入到DrawSpeech系统中，以提供期望的韵律趋势的一个粗略指示。然后，DrawSpeech根据这些粗糙的草图恢复出详细的音高和能量轮廓，并最终合成所需的语音。 实验结果表明，DrawSpeech可以生成具有广泛不同韵律特征的语音，并且可以通过用户友好的方式精确控制细粒度的韵律特性。我们的实现代码及音频样本都公开可用。
URL
https://arxiv.org/abs/2501.04256
PDF
-
SYKI-SVC: Advancing Singing Voice Conversion with Post-Processing Innovations and an Open-Source Professional Testset
2025-01-06 11:54:33Yiquan Zhou, Wenyu Wang, Hongwu Ding, Jiacheng Xu, Jihua Zhu, Xin Gao, Shihao LiAbstract
Singing voice conversion aims to transform a source singing voice into that of a target singer while preserving the original lyrics, melody, and various vocal techniques. In this paper, we propose a high-fidelity singing voice conversion system. Our system builds upon the SVCC T02 framework and consists of three key components: a feature extractor, a voice converter, and a post-processor. The feature extractor utilizes the ContentVec and Whisper models to derive F0 contours and extract speaker-independent linguistic features from the input singing voice. The voice converter then integrates the extracted timbre, F0, and linguistic content to synthesize the target speaker's waveform. The post-processor augments high-frequency information directly from the source through simple and effective signal processing to enhance audio quality. Due to the lack of a standardized professional dataset for evaluating expressive singing conversion systems, we have created and made publicly available a specialized test set. Comparative evaluations demonstrate that our system achieves a remarkably high level of naturalness, and further analysis confirms the efficacy of our proposed system design.
Abstract (translated)
歌声转换技术旨在将原唱者的演唱声音转换成目标歌手的声音，同时保留原始歌词、旋律及各种发声技巧。本文提出了一种高保真的歌声转换系统。我们的系统基于SVCC T02框架构建，并包含三个关键组件：特征提取器、声乐转换器和后处理模块。 - **特征提取器**利用ContentVec和Whisper模型来获取F0（基频）曲线并从输入的歌声中提取出与说话人无关的语言特征。 - **声乐转换器**将提取到的声音特质、F0以及语言内容整合，以合成目标歌手的声音波形。 - **后处理模块**通过简单的有效信号处理方法直接从源音频增加高频信息，以此提升音质。 由于缺乏用于评估表现力歌声转换系统的标准化专业数据集，我们创建并公开提供了一个专门的测试集。比较性评测表明，我们的系统在自然度方面达到了非常高的水平，并且进一步分析确认了所提出的设计方案的有效性。
URL
https://arxiv.org/abs/2501.02953
PDF
-
MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention
2025-01-01 12:03:37Rui Yang, Xiaojun Wu, Shengfeng HeAbstract
Current sketch extraction methods either require extensive training or fail to capture a wide range of artistic styles, limiting their practical applicability and versatility. We introduce Mixture-of-Self-Attention (MixSA), a training-free sketch extraction method that leverages strong diffusion priors for enhanced sketch perception. At its core, MixSA employs a mixture-of-self-attention technique, which manipulates self-attention layers by substituting the keys and values with those from reference sketches. This allows for the seamless integration of brushstroke elements into initial outline images, offering precise control over texture density and enabling interpolation between styles to create novel, unseen styles. By aligning brushstroke styles with the texture and contours of colored images, particularly in late decoder layers handling local textures, MixSA addresses the common issue of color averaging by adjusting initial outlines. Evaluated with various perceptual metrics, MixSA demonstrates superior performance in sketch quality, flexibility, and applicability. This approach not only overcomes the limitations of existing methods but also empowers users to generate diverse, high-fidelity sketches that more accurately reflect a wide range of artistic expressions.
Abstract (translated)
目前的草图提取方法要么需要大量的训练，要么无法捕捉广泛的艺术风格，这限制了它们的实际应用性和灵活性。我们引入了一种无需训练的草图提取方法——混合自注意力（MixSA），该方法利用强大的扩散先验来增强对草图的理解和感知。在核心机制上，MixSA采用了混合自注意力技术，通过用参考草图中的键值替换自我注意层中的键值来进行操作。这使得能够将笔触元素无缝地整合到初始轮廓图像中，并提供对纹理密度的精确控制以及风格之间的插值以创造新颖、前所未见的风格。 通过使笔触样式与彩色图像的纹理和轮廓对齐，特别是在处理局部纹理的解码器后期层中进行调整，MixSA解决了常见的颜色平均化问题。经过各种感知度量评估后，结果表明MixSA在草图质量、灵活性和应用性方面表现优越。这种方法不仅克服了现有方法的局限性，还使用户能够生成多样化且高保真的草图，更准确地反映广泛的艺术表达形式。
URL
https://arxiv.org/abs/2501.00816
PDF
-
Gaussian Building Mesh : Extract a Building's 3D Mesh with Google Earth and Gaussian Splatting
2024-12-31 19:53:27Kyle Gao, Liangzhi Li, Hongjie He, Dening Lu, Linlin Xu, Jonathan LiAbstract
Recently released open-source pre-trained foundational image segmentation and object detection models (SAM2+GroundingDINO) allow for geometrically consistent segmentation of objects of interest in multi-view 2D images. Users can use text-based or click-based prompts to segment objects of interest without requiring labeled training datasets. Gaussian Splatting allows for the learning of the 3D representation of a scene's geometry and radiance based on 2D images. Combining Google Earth Studio, SAM2+GroundingDINO, 2D Gaussian Splatting, and our improvements in mask refinement based on morphological operations and contour simplification, we created a pipeline to extract the 3D mesh of any building based on its name, address, or geographic coordinates.
Abstract (translated)
最近发布的开源预训练基础图像分割和目标检测模型（SAM2+GroundingDINO）允许在多视角二维图像中以几何一致性的方式对感兴趣的物体进行分割。用户可以使用基于文本或点击的提示来分割感兴趣的对象，而无需依赖标注的训练数据集。高斯点阵技术则可以根据二维图像学习场景的三维几何和辐射表示。通过结合Google Earth Studio、SAM2+GroundingDINO、二维高斯点阵以及我们在掩码细化方面的改进（基于形态操作和轮廓简化），我们创建了一个管道，可以仅凭建筑物的名字、地址或地理坐标提取其3D网格模型。
URL
https://arxiv.org/abs/2501.00625
PDF
-
Solar Filaments Detection using Active Contours Without Edges
2024-12-30 06:43:22Sanmoy Bandyopadhyay, Vaibhav PantAbstract
In this article, an active contours without edges (ACWE)-based algorithm has been proposed for the detection of solar filaments in H-alpha full-disk solar images. The overall algorithm consists of three main steps of image processing. These are image pre-processing, image segmentation, and image post-processing. Here in the work, contours are initialized on the solar image and allowed to deform based on the energy function. As soon as the contour reaches the boundary of the desired object, the energy function gets reduced, and the contour stops evolving. The proposed algorithm has been applied to few benchmark datasets and has been compared with the classical technique of object detection. The results analysis indicates that the proposed algorithm outperforms the results obtained using the existing classical algorithm of object detection.
Abstract (translated)
在这篇文章中，提出了一种基于无边界的主动轮廓（ACWE）算法，用于检测H-Alpha全太阳盘图像中的日珥丝。整个算法包括三个主要的图像处理步骤：预处理、分割和后处理。在此工作中，初始轮廓在太阳图像上进行初始化，并根据能量函数允许其变形。一旦轮廓到达所需对象的边界，能量函数会减少，从而停止演化。 该提出的算法已应用于一些基准数据集，并与传统的物体检测技术进行了比较。结果分析表明，所提出的算法优于现有传统物体检测算法的结果。
URL
https://arxiv.org/abs/2412.20749
PDF
-
Computational Analysis of Yaredawi YeZema Silt in Ethiopian Orthodox Tewahedo Church Chants
2024-12-25 05:42:56Mequanent Argaw Muluneh, Yan-Tsung Peng, Li SuAbstract
Despite its musicological, cultural, and religious significance, the Ethiopian Orthodox Tewahedo Church (EOTC) chant is relatively underrepresented in music research. Historical records, including manuscripts, research papers, and oral traditions, confirm Saint Yared's establishment of three canonical EOTC chanting modes during the 6th century. This paper attempts to investigate the EOTC chants using music information retrieval (MIR) techniques. Among the research questions regarding the analysis and understanding of EOTC chants, Yaredawi YeZema Silt, namely the mode of chanting adhering to Saint Yared's standards, is of primary importance. Therefore, we consider the task of Yaredawi YeZema Silt classification in EOTC chants by introducing a new dataset and showcasing a series of classification experiments for this task. Results show that using the distribution of stabilized pitch contours as the feature representation on a simple neural network-based classifier becomes an effective solution. The musicological implications and insights of such results are further discussed through a comparative study with the previous ethnomusicology literature on EOTC chants. By making this dataset publicly accessible, we aim to promote future exploration and analysis of EOTC chants and highlight potential directions for further research, thereby fostering a deeper understanding and preservation of this unique spiritual and cultural heritage.
Abstract (translated)
尽管埃塞俄比亚正教提瓦霍多教会（EOTC）唱诵在音乐学、文化和宗教方面具有重要意义，但在音乐研究中却相对被忽视。历史记录包括手稿、研究论文和口头传统都确认了圣耶雷德于公元6世纪建立了三种规范的EOTC唱诵模式。本文尝试使用音乐信息检索（MIR）技术来探讨EOTC唱诵的内容。在关于分析和理解EOTC唱诵的研究问题中，耶雷达威·耶泽玛·西尔特（即遵循圣耶雷德标准的唱诵方式）尤为重要。因此，我们通过介绍一个新数据集并展示一系列针对这一任务的分类实验，来考虑EOTC唱诵中的耶雷达威·耶泽玛·西尔特分类问题。结果显示，在简单的基于神经网络的分类器上使用稳定音高轮廓分布作为特征表示成为有效的解决方案。通过对以前关于EOTC唱诵的民族音乐学文献进行比较研究，进一步讨论了此类结果的音乐学术意义和见解。通过使此数据集公开可用，我们旨在促进未来对EOTC唱诵的探索和分析，并突出潜在的研究方向，从而加深对此独特精神和文化遗产的理解与保护。
URL
https://arxiv.org/abs/2412.18788
PDF
-
A Multimodal Fusion Framework for Bridge Defect Detection with Cross-Verification
2024-12-23 20:33:34Ravi Datta Rachuri, Duoduo Liao, Samhita Sarikonda, Datha Vaishnavi KondurAbstract
This paper presents a pilot study introducing a multimodal fusion framework for the detection and analysis of bridge defects, integrating Non-Destructive Evaluation (NDE) techniques with advanced image processing to enable precise structural assessment. By combining data from Impact Echo (IE) and Ultrasonic Surface Waves (USW) methods, this preliminary investigation focuses on identifying defect-prone regions within concrete structures, emphasizing critical indicators such as delamination and debonding. Using geospatial analysis with alpha shapes, fusion of defect points, and unified lane boundaries, the proposed framework consolidates disparate data sources to enhance defect localization and facilitate the identification of overlapping defect regions. Cross-verification with adaptive image processing further validates detected defects by aligning their coordinates with visual data, utilizing advanced contour-based mapping and bounding box techniques for precise defect identification. The experimental results, with an F1 score of 0.83, demonstrate the potential efficacy of the approach in improving defect localization, reducing false positives, and enhancing detection accuracy, which provides a foundation for future research and larger-scale validation. This preliminary exploration establishes the framework as a promising tool for efficient bridge health assessment, with implications for proactive structural monitoring and maintenance.
Abstract (translated)
本文介绍了一项试点研究，该研究引入了一个多模态融合框架，用于桥梁缺陷的检测与分析。该框架将无损评估（NDE）技术与先进的图像处理相结合，以实现精确的结构评估。通过整合冲击回波(IE)和超声表面波(USW)方法的数据，这项初步调查旨在识别混凝土结构中的易损区域，并重点强调了分层剥离和脱粘等关键指标。利用基于α形状的地理空间分析、缺陷点融合及统一车道边界的方法，所提出的框架整合了不同的数据源以增强缺陷定位并促进重叠缺陷区域的识别。通过自适应图像处理进行交叉验证进一步证实了检测到的缺陷，将坐标与视觉数据对齐，并使用高级轮廓映射和边界框技术实现精确的缺陷识别。实验结果显示出0.83的F1分数，这表明该方法在提高缺陷定位、减少误报率及增强检测准确性方面具有潜在效果，为未来的研究和更大规模的验证奠定了基础。这项初步探索确立了该框架作为高效桥梁健康评估工具的地位，并对主动结构监测与维护具有重要意义。
URL
https://arxiv.org/abs/2412.17968
PDF
-
Adaptive Signal Analysis for Automated Subsurface Defect Detection Using Impact Echo in Concrete Slabs
2024-12-23 20:05:53Deepthi Pavurala, Duoduo Liao, Chaithra Reddy PasunuruAbstract
This pilot study presents a novel, automated, and scalable methodology for detecting and evaluating subsurface defect-prone regions in concrete slabs using Impact Echo (IE) signal analysis. The approach integrates advanced signal processing, clustering, and visual analytics to identify subsurface anomalies. A unique adaptive thresholding method tailors frequency-based defect identification to the distinct material properties of each slab. The methodology generates frequency maps, binary masks, and k-means cluster maps to automatically classify defect and non-defect regions. Key visualizations, including 3D surface plots, cluster maps, and contour plots, are employed to analyze spatial frequency distributions and highlight structural anomalies. The study utilizes a labeled dataset constructed at the Federal Highway Administration (FHWA) Advanced Sensing Technology Nondestructive Evaluation Laboratory. Evaluations involve ground-truth masking, comparing the generated defect maps with top-view binary masks derived from the information provided by the FHWA. The performance metrics, specifically F1-scores and AUC-ROC, achieve values of up to 0.95 and 0.83, respectively. The results demonstrate the robustness of the methodology, consistently identifying defect-prone areas with minimal false positives and few missed defects. Adaptive frequency thresholding ensures flexibility in addressing variations across slabs, providing a scalable framework for detecting structural anomalies. Additionally, the methodology is adaptable to other frequency-based signals due to its generalizable thresholding mechanism and holds potential for integrating multimodal sensor fusion. This automated and scalable pipeline minimizes manual intervention, ensuring accurate and efficient defect detection, further advancing Non-Destructive Evaluation (NDE) techniques.
Abstract (translated)
这项初步研究提出了一种新颖的、自动化的和可扩展的方法，用于通过冲击回波（IE）信号分析来检测和评估混凝土板中的潜在缺陷区域。该方法结合了先进的信号处理、聚类和可视化分析技术，以识别地下异常。一种独特的自适应阈值方法根据每块混凝土板特有的材料属性定制基于频率的缺陷识别。该方法生成频率图、二进制掩码和k-means聚类图，自动分类有缺陷和无缺陷区域。关键可视化工具包括3D表面图、聚类图和等高线图，用于分析空间频率分布并突出结构异常。本研究使用了在美国联邦公路管理局（FHWA）先进传感技术非破坏性评估实验室构建的标注数据集。评估涉及地面真值掩码，将生成的缺陷地图与从FHWA提供的信息衍生出的顶视二进制掩码进行比较。性能指标——特别是F1分数和AUC-ROC——分别达到了高达0.95和0.83的数值。结果表明该方法具有稳健性，能够一致地识别出潜在缺陷区域，并且误报极少，漏检情况也很少。自适应频率阈值确保了在处理不同板材时的灵活性，为检测结构异常提供了可扩展框架。此外，由于其通用化的阈值机制，该方法还可以适应其他基于频率的信号，并具有整合多模态传感器融合的潜力。这种自动化和可扩展的工作流减少了人工干预，保证了准确而高效的缺陷检测，进一步推进了无损评估（NDE）技术的发展。
URL
https://arxiv.org/abs/2412.17953
PDF
-
Learning Cross-Task Generalities Across Graphs via Task-trees
2024-12-21 02:07:43Zehong Wang, Zheyuan Zhang, Tianyi Ma, Nitesh V Chawla, Chuxu Zhang, Yanfang YeAbstract
Foundation models aim to create general, cross-task, and cross-domain machine learning models by pretraining on large-scale datasets to capture shared patterns or concepts (generalities), such as contours, colors, textures, and edges in images, or tokens, words, and sentences in text. However, discovering generalities across graphs remains challenging, which has hindered the development of graph foundation models. To tackle this challenge, in this paper, we propose a novel approach to learn generalities across graphs via task-trees. Specifically, we first define the basic learning instances in graphs as task-trees and assume that the generalities shared across graphs are, at least partially, preserved in the task-trees of the given graphs. To validate the assumption, we first perform a theoretical analysis of task-trees in terms of stability, transferability, and generalization. We find that if a graph neural network (GNN) model is pretrained on diverse task-trees through a reconstruction task, it can learn sufficient transferable knowledge for downstream tasks using an appropriate set of fine-tuning samples. To empirically validate the assumption, we further instantiate the theorems by developing a cross-task, cross-domain graph foundation model named Graph generality Identifier on task-Trees (GIT). The extensive experiments over 30 graphs from five domains demonstrate the effectiveness of GIT in fine-tuning, in-context learning, and zero-shot learning scenarios. Particularly, the general GIT model pretrained on large-scale datasets can be quickly adapted to specific domains, matching or even surpassing expert models designed for those domains. Our data and code are available at this https URL.
Abstract (translated)
基础模型旨在通过在大规模数据集上预训练来创建通用的、跨任务和跨领域的机器学习模型，以捕捉共享模式或概念（普遍性），如图像中的轮廓、颜色、纹理和边缘，或者文本中的标记、单词和句子。然而，在图上发现普遍性仍然是一个挑战，这阻碍了图基础模型的发展。为应对这一挑战，本文提出了一种通过任务树来学习图形间普遍性的新方法。具体来说，我们将图中的基本学习实例定义为任务树，并假设图之间共享的普遍性至少部分保留在给定图的任务树中。为了验证该假设，我们首先从稳定性、可迁移性和泛化能力三个方面对任务树进行了理论分析。发现如果一个图神经网络（GNN）模型通过重建任务在多样化的任务树上进行预训练，它就可以使用一组适当的微调样本学习到足够用于下游任务的可转移知识。为了经验性地验证该假设，我们进一步通过开发一种跨任务、跨领域的图基础模型——基于任务树的图普遍性识别器（GIT），实例化了这些定理。在五个领域中30个图形上的广泛实验表明，GIT在微调、上下文学习和零样本学习场景中的有效性。特别是，预训练于大规模数据集的一般性GIT模型可以快速适应特定领域，并且与专门为那些领域设计的专家模型相比，其表现可匹敌甚至超越后者。我们的数据和代码可以在以下网址获取：[此处提供URL]。
URL
https://arxiv.org/abs/2412.16441
PDF
-
Reconstruction of Contour Lines During the Digitization of Contour Maps to Build a Digital Elevation Model
2024-12-20 03:02:42Aroj Subedi, Pradip Ganesh, Sandip MishraAbstract
Contour map has contour lines that are significant in building a Digital Elevation Model (DEM). During the digitization and pre-processing of contour maps, the contour line intersects with each other or break apart resulting in broken contour segments. These broken segments impose a greater risk while building DEM leading to a faulty model. In this project, a simple yet efficient mechanism is used to match and reconnect the endpoints of the broken segments accurately and efficiently. The matching of the endpoints is done using the concept of minimum Euclidean distance and gradient direction while the Cubic Hermite spline interpolation technique is used to reconnect the endpoints by estimating the values using a mathematical function that minimizes overall surface curvature resulting in a smooth curve. The purpose of this work is to reconnect the broken contour lines generated during the digitization of the contour map, to help build the most appropriate digital elevation model for the corresponding contour map.
Abstract (translated)
等高线图中的等高线在线构建数字高程模型（DEM）中至关重要。在对等高线图进行数字化和预处理的过程中，等高线相互交叉或断裂，导致出现断裂的等高线段。这些断裂的线段会增加构建DEM时的风险，从而可能导致模型出错。在这个项目中，使用了一种简单而高效的机制来准确且高效地匹配并重新连接断裂线段的端点。该匹配过程采用了最小欧几里得距离和梯度方向的概念，同时利用三次Hermite样条插值技术通过一个数学函数估算值来重新连接端点，这个函数能最小化整体曲面弯曲度从而形成一条平滑曲线。本工作的目的是将等高线图数字化过程中生成的断裂等高线重新连接起来，以便为相应的等高线图构建最合适的数字高程模型。
URL
https://arxiv.org/abs/2412.15515
PDF
-
T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation
2024-12-18 04:01:32Zhenhong Sun, Yifu Wang, Yonhon Ng, Yunfei Duan, Daoyi Dong, Hongdong Li, Pan JiAbstract
Scene generation is crucial to many computer graphics applications. Recent advances in generative AI have streamlined sketch-to-image workflows, easing the workload for artists and designers in creating scene concept art. However, these methods often struggle for complex scenes with multiple detailed objects, sometimes missing small or uncommon instances. In this paper, we propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation after reviewing the entire cross-attention mechanism. This scheme revitalizes the existing ControlNet model, enabling effective handling of multi-instance generations, involving prompt balance, characteristics prominence, and dense tuning. Specifically, this approach enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances. It also includes a characteristics prominence module that highlights TopK indices in each channel, ensuring essential features are better represented based on token sketches. Additionally, it employs dense tuning to refine contour details in the attention map, compensating for instance-related regions. Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models. It consistently generates detailed, multi-instance 2D images, closely adhering to the input prompts and enhancing visual quality in complex multi-instance scenes. Code is available at this https URL.
Abstract (translated)
场景生成对许多计算机图形应用至关重要。最近在生成式人工智能方面的进展简化了草图转图像的工作流程，减轻了艺术家和设计师创作场景概念艺术的工作量。然而，这些方法往往难以处理包含多个详细对象的复杂场景，有时会遗漏小或不常见的实例。在这篇论文中，我们在审查整个交叉注意力机制后提出了一个无需训练的三元组调整方案（T3-S2S）以进行草图转场景生成。该方案重新激活了现有的ControlNet模型，使其能够有效处理多实例生成，包括提示平衡、特征突出和密集调优。具体而言，这种方法通过提示平衡模块增强了关键词表示，降低了遗漏关键实例的风险。它还包含一个特征突出模块，在每个通道中突出TopK索引，确保基于标记草图的重要特征得到更好的表现。此外，该方法采用密集调优来细化注意力图中的轮廓细节，弥补与实例相关的区域。实验验证了我们的三元组调整方案显著提高了现有草图转图像模型的性能。它能够一致地生成详细、多实例的2D图像，紧密遵循输入提示并增强复杂多实例场景下的视觉质量。代码可在此 https URL 获取。
URL
https://arxiv.org/abs/2412.13486
PDF
-
Facial Surgery Preview Based on the Orthognathic Treatment Prediction
2024-12-15 04:06:39Huijun Han, Congyi Zhang, Lifeng Zhu, Pradeep Singh, Richard Tai Chiu Hsung, Yiu Yan Leung, Taku Komura, Wenping Wang, Min GuAbstract
Orthognathic surgery consultation is essential to help patients understand the changes to their facial appearance after surgery. However, current visualization methods are often inefficient and inaccurate due to limited pre- and post-treatment data and the complexity of the treatment. To overcome these challenges, this study aims to develop a fully automated pipeline that generates accurate and efficient 3D previews of postsurgical facial appearances for patients with orthognathic treatment without requiring additional medical images. The study introduces novel aesthetic losses, such as mouth-convexity and asymmetry losses, to improve the accuracy of facial surgery prediction. Additionally, it proposes a specialized parametric model for 3D reconstruction of the patient, medical-related losses to guide latent code prediction network optimization, and a data augmentation scheme to address insufficient data. The study additionally employs FLAME, a parametric model, to enhance the quality of facial appearance previews by extracting facial latent codes and establishing dense correspondences between pre- and post-surgery geometries. Quantitative comparisons showed the algorithm's effectiveness, and qualitative results highlighted accurate facial contour and detail predictions. A user study confirmed that doctors and the public could not distinguish between machine learning predictions and actual postoperative results. This study aims to offer a practical, effective solution for orthognathic surgery consultations, benefiting doctors and patients.
Abstract (translated)
正颌手术咨询对于帮助患者理解手术后面部外观的变化至关重要。然而，由于治疗前后数据有限以及治疗方法的复杂性，目前的可视化方法往往效率低下且不够准确。为了克服这些挑战，本研究旨在开发一个全自动流程，为接受正颌治疗的患者生成精准高效的术后三维面部预览，而无需额外的医学影像。该研究引入了新颖的美学损失函数，如嘴巴凸度和不对称性损失，以提高对面部手术预测的准确性。此外，它还提出了用于病人三维重建的专业参数模型、与医疗相关的损失来指导潜在代码预测网络优化以及一个数据增强方案来解决数据不足的问题。本研究进一步采用了FLAME（一种参数化模型），通过提取面部潜在代码并建立术前术后几何结构之间的密集对应关系，以提高面部外观预览的质量。定量比较显示了算法的有效性，定性结果突出了准确的面部轮廓和细节预测。一项用户研究表明，医生和公众无法区分机器学习预测与实际术后结果。本研究旨在为正颌手术咨询提供一个实用有效的解决方案，惠及医生和患者。
URL
https://arxiv.org/abs/2412.11045
PDF
-
Reliable and superior elliptic Fourier descriptor normalization and its application software ElliShape with efficient image processing
2024-12-14 11:14:05Hui Wu (Big Data and AI Biodiversity Conservation Research Center, Institute of Botany, Chinese Academy of Sciences, Beijing, China, State Key Laboratory of Plant Diversity and Specialty Crops and Key Laboratory of Systematic and Evolutionary Botany, Institute of Botany, Chinese Academy of Sciences, Beijing, China, Plant Science Data Center, Chinese Academy of Sciences, Beijing, China, China National Botanical Garden, Beijing, China), Jia-Jie Yang (Big Data and AI Biodiversity Conservation Research Center, Institute of Botany, Chinese Academy of Sciences, Beijing, China, Plant Science Data Center, Chinese Academy of Sciences, Beijing, China, China National Botanical Garden, Beijing, China), Chao-Qun Li (School of Life Sciences, Qilu Normal University, Jinan, China), Jin-Hua Ran (State Key Laboratory of Plant Diversity and Specialty Crops and Key Laboratory of Systematic and Evolutionary Botany, Institute of Botany, Chinese Academy of Sciences, Beijing, China, China National Botanical Garden, Beijing, China, University of Chinese Academy of Sciences, Beijing, China), Ren-Hua Peng (University of Chinese Academy of Sciences, Beijing, China, Key Laboratory of Noise and Vibration Control, Institute of Acoustics, Chinese Academy of Sciences, Beijing, China), Xiao-Quan Wang (Big Data and AI Biodiversity Conservation Research Center, Institute of Botany, Chinese Academy of Sciences, Beijing, China, State Key Laboratory of Plant Diversity and Specialty Crops and Key Laboratory of Systematic and Evolutionary Botany, Institute of Botany, Chinese Academy of Sciences, Beijing, China, Plant Science Data Center, Chinese Academy of Sciences, Beijing, China, China National Botanical Garden, Beijing, China, University of Chinese Academy of Sciences, Beijing, China)Abstract
Elliptic Fourier analysis (EFA) is a powerful tool for shape analysis, which is often employed in geometric morphometrics. However, the normalization of elliptic Fourier descriptors has persistently posed challenges in obtaining unique results in basic contour transformations, requiring extensive manual alignment. Additionally, contemporary contour/outline extraction methods often struggle to handle complex digital images. Here, we reformulated the procedure of EFDs calculation to improve computational efficiency and introduced a novel approach for EFD normalization, termed true EFD normalization, which remains invariant under all basic contour transformations. These improvements are crucial for processing large sets of contour curves collected from different platforms with varying transformations. Based on these improvements, we developed ElliShape, a user-friendly software. Particularly, the improved contour/outline extraction employs an interactive approach that combines automatic contour generation for efficiency with manual correction for essential modifications and refinements. We evaluated ElliShape's stability, robustness, and ease of use by comparing it with existing software using standard datasets. ElliShape consistently produced reliable reconstructed shapes and normalized EFD values across different contours and transformations, and it demonstrated superior performance in visualization and efficient processing of various digital images for contour this http URL output annotated images and EFDs could be utilized in deep learning-based data training, thereby advancing artificial intelligence in botany and offering innovative solutions for critical challenges in biodiversity conservation, species classification, ecosystem function assessment, and related critical issues.
Abstract (translated)
椭圆傅里叶分析（EFA）是形状分析的强大工具，常用于几何形态学中。然而，在进行基本轮廓变换时，椭圆傅里叶描述符的标准化一直面临着挑战，需要大量的手动对齐来获得独特结果。此外，现代轮廓/边缘提取方法在处理复杂数字图像方面常常遇到困难。在此基础上，我们重新制定了EFDs计算程序以提高计算效率，并引入了一种新的规范化方法——真正的EFD规范化，它在所有基本轮廓变换下保持不变。这些改进对于处理从不同平台收集并经历各种转换的大规模轮廓曲线集至关重要。基于这些改进，我们开发了ElliShape这一用户友好的软件。特别地，改进的轮廓/边缘提取采用了一种结合自动轮廓生成以提高效率和手动校正进行必要修改和细化的交互式方法。通过使用标准数据集与现有软件进行比较，评估了ElliShape的稳定性、鲁棒性和易用性。结果显示，ElliShape能够始终如一地生成可靠的重建形状和标准化后的EFD值，并在不同轮廓和变换下表现出卓越的可视化能力和各种数字图像处理效率。由此产生的带注释的图像和EFDs可用于基于深度学习的数据训练，从而推动植物学领域的人工智能发展，并为生物多样性保护、物种分类、生态系统功能评估等关键问题提供创新解决方案。
URL
https://arxiv.org/abs/2412.10795
PDF
-
Cycle-Consistent Bridge Diffusion Model for Accelerated MRI Reconstruction
2024-12-13 09:35:34Tao Song, Yicheng Wu, Minhao Hu, Xiangde Luo, Guoting Luo, Guotai Wang, Yi Guo, Feng Xu, Shaoting ZhangAbstract
Accelerated MRI reconstruction techniques aim to reduce examination time while maintaining high image fidelity, which is highly desirable in clinical settings for improving patient comfort and hospital efficiency. Existing deep learning methods typically reconstruct images from under-sampled data with traditional reconstruction approaches, but they still struggle to provide high-fidelity results. Diffusion models show great potential to improve fidelity of generated images in recent years. However, their inference process starting with a random Gaussian noise introduces instability into the results and usually requires thousands of sampling steps, resulting in sub-optimal reconstruction quality and low efficiency. To address these challenges, we propose Cycle-Consistent Bridge Diffusion Model (CBDM). CBDM employs two bridge diffusion models to construct a cycle-consistent diffusion process with a consistency loss, enhancing the fine-grained details of reconstructed images and reducing the number of diffusion steps. Moreover, CBDM incorporates a Contourlet Decomposition Embedding Module (CDEM) which captures multi-scale structural texture knowledge in images through frequency domain decomposition pyramids and directional filter banks to improve structural fidelity. Extensive experiments demonstrate the superiority of our model by higher reconstruction quality and fewer training iterations, achieving a new state of the art for accelerated MRI reconstruction in both fastMRI and IXI datasets.
Abstract (translated)
加速的磁共振成像（MRI）重建技术旨在减少检查时间同时保持高图像保真度，这对于临床环境中提高患者舒适度和医院效率非常理想。现有的深度学习方法通常从欠采样数据中进行传统重建方式来重构图像，但它们仍难以提供高质量的结果。近年来，扩散模型显示出显著提升生成图像保真度的潜力。然而，这些模型的推理过程从随机高斯噪声开始引入了结果不稳定，并且通常需要数千次采样步骤，导致重建质量不佳和效率低下。为了解决这些问题，我们提出了循环一致性桥接扩散模型（CBDM）。CBDM采用两个桥接扩散模型构建一个带有一致损失的循环一致性扩散过程，增强重建图像的细节并减少扩散步骤的数量。此外，CBDM集成了轮廓波分解嵌入模块（CDEM），通过频率域分解金字塔和方向滤波器组捕捉图像中的多尺度结构纹理知识，以提高结构保真度。广泛的实验表明，我们的模型在快MRI和IXI数据集中通过更高的重建质量和更少的训练迭代次数实现了加速MRI重建的新技术标杆。
URL
https://arxiv.org/abs/2412.09998
PDF
-
Soybean Maturity Prediction using 2D Contour Plots from Drone based Time Series Imagery
2024-12-12 19:23:50Bitgoeul Kim, Samuel W. Blair, Talukder Z. Jubery, Soumik Sarkar, Arti Singh, Asheesh K. Singh, Baskar GanapathysubramanianAbstract
Plant breeding programs require assessments of days to maturity for accurate selection and placement of entries in appropriate tests. In the early stages of the breeding pipeline, soybean breeding programs assign relative maturity ratings to experimental varieties that indicate their suitable maturity zones. Traditionally, the estimation of maturity value for breeding varieties has involved breeders manually inspecting fields and assessing maturity value visually. This approach relies heavily on rater judgment, making it subjective and time-consuming. This study aimed to develop a machine-learning model for evaluating soybean maturity using UAV-based time-series imagery. Images were captured at three-day intervals, beginning as the earliest varieties started maturing and continuing until the last varieties fully matured. The data collected for this experiment consisted of 22,043 plots collected across three years (2021 to 2023) and represent relative maturity groups 1.6 - 3.9. We utilized contour plot images extracted from the time-series UAV RGB imagery as input for a neural network model. This contour plot approach encoded the temporal and spatial variation within each plot into a single image. A deep learning model was trained to utilize this contour plot to predict maturity ratings. This model significantly improves accuracy and robustness, achieving up to 85% accuracy. We also evaluate the model's accuracy as we reduce the number of time points, quantifying the trade-off between temporal resolution and maturity prediction. The predictive model offers a scalable, objective, and efficient means of assessing crop maturity, enabling phenomics and ML approaches to reduce the reliance on manual inspection and subjective assessment. This approach enables the automatic prediction of relative maturity ratings in a breeding program, saving time and resources.
Abstract (translated)
植物育种项目需要评估成熟所需天数，以确保准确选择并将其安排在合适的测试中。在育种流程的早期阶段，大豆育种项目会给实验品种分配相对成熟的评级，以指示其适合的成熟区域。传统上，对育种品种成熟度值的估计涉及育种者手动检查田地，并通过视觉评估来确定成熟度值。这种方法高度依赖于评估者的判断，因此具有主观性和耗时性。本研究旨在开发一种基于无人机时间序列图像评估大豆成熟的机器学习模型。图像采集每隔三天进行一次，从最早开始成熟的品种开始，持续到最晚成熟的品种完全成熟为止。该实验收集的数据包括三年（2021年至2023年）内收集的22,043个地块，代表相对成熟组别1.6至3.9。我们使用从时间序列无人机RGB图像中提取的等高线图作为神经网络模型的输入。这种等高线图方法将每个地块内的时空变化编码为单张图像。训练了一个深度学习模型来利用此等高线图预测成熟度评级，该模型显著提高了准确性和鲁棒性，达到了最高85%的准确性。我们还评估了减少时间点数量时模型的准确性，量化了时间分辨率与成熟预测之间的权衡。这一预测模型提供了一种可扩展、客观且高效的作物成熟度评估手段，使表型学和机器学习方法能够减少对人工检查和主观评价的依赖。这种方法使得在育种项目中自动预测相对成熟评级成为可能，节省时间和资源。
URL
https://arxiv.org/abs/2412.09696
PDF
-
Local Features Meet Stochastic Anonymization: Revolutionizing Privacy-Preserving Face Recognition for Black-Box Models
2024-12-11 10:49:15Yuanwei Liu, Chengyu Jia, Ruqi Xiao, Xuemai Jia, Hui Wei, Kui Jiang, Zheng WangAbstract
The task of privacy-preserving face recognition (PPFR) currently faces two major unsolved challenges: (1) existing methods are typically effective only on specific face recognition models and struggle to generalize to black-box face recognition models; (2) current methods employ data-driven reversible representation encoding for privacy protection, making them susceptible to adversarial learning and reconstruction of the original image. We observe that face recognition models primarily rely on local features ({e.g., face contour, skin texture, and so on) for identification. Thus, by disrupting global features while enhancing local features, we achieve effective recognition even in black-box environments. Additionally, to prevent adversarial models from learning and reversing the anonymization process, we adopt an adversarial learning-based approach with irreversible stochastic injection to ensure the stochastic nature of the anonymization. Experimental results demonstrate that our method achieves an average recognition accuracy of 94.21\% on black-box models, outperforming existing methods in both privacy protection and anti-reconstruction capabilities.
Abstract (translated)
隐私保护人脸识别（PPFR）当前面临着两大未解决的挑战：(1) 现有方法通常仅在特定的人脸识别模型上有效，难以泛化到黑盒人脸识别模型；(2) 当前的方法使用数据驱动的可逆表示编码来保护隐私，使其容易受到对抗性学习和原始图像重建的影响。我们观察到，人脸识别模型主要依赖局部特征（如人脸轮廓、皮肤纹理等）进行识别。因此，通过扰乱全局特征同时增强局部特征，我们在黑盒环境中也能实现有效的识别。此外，为了防止对抗模型学习并逆转匿名化过程，我们采用基于对抗学习的方法，并引入不可逆的随机注入以确保匿名化的随机性。实验结果显示，我们的方法在黑盒模型上实现了平均94.21%的识别准确率，在隐私保护和防重建能力方面均优于现有方法。
URL
https://arxiv.org/abs/2412.08276
PDF
-
Automatic Tongue Delineation from MRI Images with a Convolutional Neural Network Approach
2024-12-06 09:49:24Karyna Isaieva (IADI), Yves Laprie (LORIA, MULTISPEECH), Nicolas Turpault (MULTISPEECH), Alexis Houssard (MULTISPEECH), Jacques Felblinger (IADI, CIC-IT), Pierre-Andr\'e Vuissoz (IADI)Abstract
Tongue contour extraction from real-time magnetic resonance images is a nontrivial task due to the presence of artifacts manifesting in form of blurring or ghostly contours. In this work, we present results of automatic tongue delineation achieved by means of U-Net auto-encoder convolutional neural network. We present both intra- and inter-subject validation. We used real-time magnetic resonance images and manually annotated 1-pixel wide contours as inputs. Predicted probability maps were post-processed in order to obtain 1-pixel wide tongue contours. The results are very good and slightly outperform published results on automatic tongue segmentation.
Abstract (translated)
从实时磁共振图像中提取舌轮廓是一项非 trivial（复杂）的任务，因为存在以模糊或幽灵状轮廓形式显现的伪影。在本研究中，我们展示了通过 U-Net 自编码卷积神经网络实现自动舌界定的结果。我们同时进行了受试者内和受试者间验证。我们使用了实时磁共振图像，并将手动注释的 1 像素宽的轮廓作为输入。预测的概率图经过后处理以获得 1 像素宽的舌轮廓。结果非常好，略优于已发布的自动舌分割结果。
URL
https://arxiv.org/abs/2412.04893
PDF
