Contour
-
Active Contour Models Driven by Hyperbolic Mean Curvature Flow for Image Segmentation
2025-06-07 08:29:40Saiyu Hu, Chunlei He, Jianfeng Zhang, Dexing Kong, Shoujun HuangAbstract
Parabolic mean curvature flow-driven active contour models (PMCF-ACMs) are widely used in image segmentation, which however depend heavily on the selection of initial curve configurations. In this paper, we firstly propose several hyperbolic mean curvature flow-driven ACMs (HMCF-ACMs), which introduce tunable initial velocity fields, enabling adaptive optimization for diverse segmentation scenarios. We shall prove that HMCF-ACMs are indeed normal flows and establish the numerical equivalence between dissipative HMCF formulations and certain wave equations using the level set method with signed distance function. Building on this framework, we furthermore develop hyperbolic dual-mode regularized flow-driven ACMs (HDRF-ACMs), which utilize smooth Heaviside functions for edge-aware force modulation to suppress over-diffusion near weak boundaries. Then, we optimize a weighted fourth-order Runge-Kutta algorithm with nine-point stencil spatial discretization when solving the above-mentioned wave equations. Experiments show that both HMCF-ACMs and HDRF-ACMs could achieve more precise segmentations with superior noise resistance and numerical stability due to task-adaptive configurations of initial velocities and initial contours.
Abstract (translated)
抛物型平均曲率流驱动的活动轮廓模型（PMCF-ACMs）在图像分割中被广泛应用，然而这些方法严重依赖于初始曲线配置的选择。在这篇论文中，我们首先提出了几种双曲平均曲率流驱动的活动轮廓模型（HMCF-ACMs），引入了可调的初始速度场，使它们能够针对不同的分割场景进行自适应优化。我们将证明HMCF-ACMs确实是规范流，并通过带符号距离函数的水平集方法建立了耗散型HMCF形式与某些波动方程之间的数值等价性。基于这一框架，我们进一步开发了双曲双模式正则化流动驱动的活动轮廓模型（HDRF-ACMs），利用平滑海维赛德函数进行边缘感知力调制以抑制弱边界附近的过度扩散现象。然后，我们在求解上述波动方程时优化了一种加权四阶龙格库塔算法，并采用了九点模板的空间离散化方法。 实验表明，由于初始速度和轮廓的配置可以根据任务需求进行自适应调整，HMCF-ACMs 和 HDRF-ACMs 都能够实现更精确的分割效果，并且具有更好的抗噪性能和数值稳定性。
URL
https://arxiv.org/abs/2506.06712
PDF
-
Through-the-Wall Radar Human Activity Recognition WITHOUT Using Neural Networks
2025-06-05 15:45:08Weicheng GaoAbstract
After a few years of research in the field of through-the-wall radar (TWR) human activity recognition (HAR), I found that we seem to be stuck in the mindset of training on radar image data through neural network models. The earliest related works in this field based on template matching did not require a training process, and I believe they have never died. Because these methods possess a strong physical interpretability and are closer to the basis of theoretical signal processing research. In this paper, I would like to try to return to the original path by attempting to eschew neural networks to achieve the TWR HAR task and challenge to achieve intelligent recognition as neural network models. In detail, the range-time map and Doppler-time map of TWR are first generated. Then, the initial regions of the human target foreground and noise background on the maps are determined using corner detection method, and the micro-Doppler signature is segmented using the multiphase active contour model. The micro-Doppler segmentation feature is discretized into a two-dimensional point cloud. Finally, the topological similarity between the resulting point cloud and the point clouds of the template data is calculated using Mapper algorithm to obtain the recognition results. The effectiveness of the proposed method is demonstrated by numerical simulated and measured experiments. The open-source code of this work is released at: this https URL.
Abstract (translated)
经过几年对穿墙雷达（TWR）人体活动识别（HAR）领域的研究，我发现我们似乎陷入了通过神经网络模型训练雷达图像数据的思维定势中。最早的相关工作基于模板匹配方法，并不需要训练过程，我认为这些方法从未真正消失过。因为这些方法具有较强的物理可解释性，并且更接近于理论信号处理研究的基础。在这篇论文中，我试图回到原来的路径上，尝试摒弃神经网络来完成TWR HAR任务，并挑战仅依靠智能识别实现与基于神经网络模型的同等效果。 具体而言，在本方案中，首先生成TWR的范围-时间图和多普勒-时间图。然后，使用角点检测方法确定地图上的人体目标前景区域和噪声背景初始区域，接着利用多相活动轮廓模型对微多普勒特征进行分割。将微多普勒分割特征离散化为二维点云。最后，使用Mapper算法计算所得点云与模板数据点云之间的拓扑相似性以获得识别结果。 通过数值模拟实验和实测实验验证了所提出方法的有效性。本工作的开源代码可在以下网址获取：[此URL]（请将“this https URL”替换为实际的链接）。
URL
https://arxiv.org/abs/2506.05169
PDF
-
Physical Annotation for Automated Optical Inspection: A Concept for In-Situ, Pointer-Based Trainingdata Generation
2025-06-05 13:37:24Oliver Krumpek, Oliver Heimann, J\"org Kr\"ugerAbstract
This paper introduces a novel physical annotation system designed to generate training data for automated optical inspection. The system uses pointer-based in-situ interaction to transfer the valuable expertise of trained inspection personnel directly into a machine learning (ML) training pipeline. Unlike conventional screen-based annotation methods, our system captures physical trajectories and contours directly on the object, providing a more intuitive and efficient way to label data. The core technology uses calibrated, tracked pointers to accurately record user input and transform these spatial interactions into standardised annotation formats that are compatible with open-source annotation software. Additionally, a simple projector-based interface projects visual guidance onto the object to assist users during the annotation process, ensuring greater accuracy and consistency. The proposed concept bridges the gap between human expertise and automated data generation, enabling non-IT experts to contribute to the ML training pipeline and preventing the loss of valuable training samples. Preliminary evaluation results confirm the feasibility of capturing detailed annotation trajectories and demonstrate that integration with CVAT streamlines the workflow for subsequent ML tasks. This paper details the system architecture, calibration procedures and interface design, and discusses its potential contribution to future ML data generation for automated optical inspection.
Abstract (translated)
本文介绍了一种新颖的物理标注系统，旨在为自动化光学检测生成训练数据。该系统使用基于指针的现场互动方法，将受过培训的操作人员的专业知识直接转化为机器学习（ML）训练管道的一部分。与传统的屏幕式注释方法不同，我们的系统可以直接在物体上捕捉物理轨迹和轮廓，提供一种更直观且高效的标注方式。 核心技术采用校准、跟踪指针来准确记录用户输入，并将这些空间互动转换为与开源注释软件兼容的标准注释格式。此外，一个简单的投影仪界面将视觉指导直接投射到物体表面，帮助用户在注释过程中提高准确性并保持一致性。所提出的概念弥合了人类专业知识和自动数据生成之间的差距，使非IT专家也能参与到ML训练管道中，并防止有价值的训练样本丢失。 初步评估结果证实了捕捉详细标注轨迹的可行性，并展示了与CVAT集成可以简化后续ML任务的工作流程。本文详细介绍了系统架构、校准程序和界面设计，并讨论了其对未来自动化光学检测中的ML数据生成的潜在贡献。
URL
https://arxiv.org/abs/2506.05026
PDF
-
Contour Errors: An Ego-Centric Metric for Reliable 3D Multi-Object Tracking
2025-06-04 16:15:04Sharang Kaul, Mario Berk, Thiemo Gerbich, Abhinav ValadaAbstract
Finding reliable matches is essential in multi-object tracking to ensure the accuracy and reliability of perception systems in safety-critical applications such as autonomous vehicles. Effective matching mitigates perception errors, enhancing object identification and tracking for improved performance and safety. However, traditional metrics such as Intersection over Union (IoU) and Center Point Distances (CPDs), which are effective in 2D image planes, often fail to find critical matches in complex 3D scenes. To address this limitation, we introduce Contour Errors (CEs), an ego or object-centric metric for identifying matches of interest in tracking scenarios from a functional perspective. By comparing bounding boxes in the ego vehicle's frame, contour errors provide a more functionally relevant assessment of object matches. Extensive experiments on the nuScenes dataset demonstrate that contour errors improve the reliability of matches over the state-of-the-art 2D IoU and CPD metrics in tracking-by-detection methods. In 3D car tracking, our results show that Contour Errors reduce functional failures (FPs/FNs) by 80% at close ranges and 60% at far ranges compared to IoU in the evaluation stage.
Abstract (translated)
在多目标跟踪中，找到可靠的匹配是确保感知系统在自动驾驶等安全关键应用中的准确性和可靠性的重要因素。有效的匹配可以减少感知错误，提高物体识别和追踪的性能与安全性。然而，传统的度量标准如交并比（IoU）和中心点距离（CPD），虽然在二维图像平面上效果显著，但在复杂的三维场景中往往难以找到关键匹配。为了解决这一局限性，我们引入了轮廓误差（CEs），这是一种从功能角度出发、以自身车辆或目标为中心的度量标准，用于识别跟踪场景中的感兴趣匹配。通过比较自身车辆坐标系中的边界框，轮廓误差提供了更为功能相关的物体匹配评估。在nuScenes数据集上的广泛实验表明，与当前最优的2D IoU和CPD指标相比，轮廓误差提高了检测到追踪方法中匹配的可靠性。 特别是在三维汽车跟踪方面，我们的结果显示，在评估阶段，与IoU相比，轮廓误差可以将近距离的功能失败（FPs/FNs）减少80%，远距离的则减少60%。
URL
https://arxiv.org/abs/2506.04122
PDF
-
The Harmonic Structure of Information Contours
2025-06-04 12:56:30Eleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Wilcox, Alex Warstadt, Ryan Cotterell, Mario GiulianelliAbstract
The uniform information density (UID) hypothesis proposes that speakers aim to distribute information evenly throughout a text, balancing production effort and listener comprehension difficulty. However, language typically does not maintain a strictly uniform information rate; instead, it fluctuates around a global average. These fluctuations are often explained by factors such as syntactic constraints, stylistic choices, or audience design. In this work, we explore an alternative perspective: that these fluctuations may be influenced by an implicit linguistic pressure towards periodicity, where the information rate oscillates at regular intervals, potentially across multiple frequencies simultaneously. We apply harmonic regression and introduce a novel extension called time scaling to detect and test for such periodicity in information contours. Analyzing texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese, we find consistent evidence of periodic patterns in information rate. Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization. Beyond highlighting the connection between information rate and discourse structure, our approach offers a general framework for uncovering structural pressures at various levels of linguistic granularity.
Abstract (translated)
统一信息密度（UID）假说认为，说话者在文本中力求均匀地分配信息，平衡生产努力与听众理解的难度。然而，语言通常不会维持严格的恒定信息率；相反，它会围绕一个全球平均值波动。这些波动往往由诸如句法约束、风格选择或受众设计等因素解释。在这项工作中，我们探讨了一个替代视角：认为这些波动可能受到周期性隐含语言压力的影响，在这种情况下，信息速率会在规则的时间间隔内振荡，并且可能同时在多个频率上进行。 为了检测并测试这种周期性，我们应用了谐波回归方法，并引入了一种称为时间缩放的新扩展技术。通过分析英语、西班牙语、德语、荷兰语、巴斯克语和巴西葡萄牙语的文本，我们发现信息率中的周期性模式一致存在。许多主要频率与话语结构相符，表明这些振荡反映了有意义的语言组织。 我们的方法不仅强调了信息速率与话语结构之间的联系，还为在语言各个层次的颗粒度上揭露结构性压力提供了一个通用框架。
URL
https://arxiv.org/abs/2506.03902
PDF
-
Probabilistic Online Event Downsampling
2025-06-03 07:33:11Andreu Girbau-Xalabarder, Jun Nagata, Shinichi SumiyoshiAbstract
Event cameras capture scene changes asynchronously on a per-pixel basis, enabling extremely high temporal resolution. However, this advantage comes at the cost of high bandwidth, memory, and computational demands. To address this, prior work has explored event downsampling, but most approaches rely on fixed heuristics or threshold-based strategies, limiting their adaptability. Instead, we propose a probabilistic framework, POLED, that models event importance through an event-importance probability density function (ePDF), which can be arbitrarily defined and adapted to different applications. Our approach operates in a purely online setting, estimating event importance on-the-fly from raw event streams, enabling scene-specific adaptation. Additionally, we introduce zero-shot event downsampling, where downsampled events must remain usable for models trained on the original event stream, without task-specific adaptation. We design a contour-preserving ePDF that prioritizes structurally important events and evaluate our method across four datasets and tasks--object classification, image interpolation, surface normal estimation, and object detection--demonstrating that intelligent sampling is crucial for maintaining performance under event-budget constraints.
Abstract (translated)
事件相机以每像素异步方式捕捉场景变化，从而实现极高的时间分辨率。然而，这种优势也带来了高带宽、内存和计算需求的代价。为了解决这个问题，先前的研究探讨了事件降采样的方法，但大多数方法依赖于固定的启发式或基于阈值的策略，这限制了它们的适应性。为此，我们提出了一种概率框架POLED，通过定义事件重要性的概率密度函数（ePDF）来建模事件的重要性，该函数可以任意定义并根据不同的应用场景进行调整。我们的方法在完全在线环境中运行，直接从原始事件流中实时估计事件的重要性，从而实现针对特定场景的适应性。 此外，我们引入了无损事件降采样概念，在这种情况下，经过降采样的事件必须保持可用状态，以用于训练过程中使用完整事件流所得到的任务模型。为此，我们设计了一个边缘保留的ePDF来优先处理结构上重要的事件，并在四个数据集和任务（物体分类、图像插值、表面法线估计及物体检测）中评估了我们的方法，证明了智能采样对于在事件预算限制下保持性能至关重要。
URL
https://arxiv.org/abs/2506.02547
PDF
-
From Local Cues to Global Percepts: Emergent Gestalt Organization in Self-Supervised Vision Models
2025-05-31 21:35:54Tianqin Li, Ziqi Wen, Leiran Song, Jun Liu, Zhi Jing, Tai Sing LeeAbstract
Human vision organizes local cues into coherent global forms using Gestalt principles like closure, proximity, and figure-ground assignment -- functions reliant on global spatial structure. We investigate whether modern vision models show similar behaviors, and under what training conditions these emerge. We find that Vision Transformers (ViTs) trained with Masked Autoencoding (MAE) exhibit activation patterns consistent with Gestalt laws, including illusory contour completion, convexity preference, and dynamic figure-ground segregation. To probe the computational basis, we hypothesize that modeling global dependencies is necessary for Gestalt-like organization. We introduce the Distorted Spatial Relationship Testbench (DiSRT), which evaluates sensitivity to global spatial perturbations while preserving local textures. Using DiSRT, we show that self-supervised models (e.g., MAE, CLIP) outperform supervised baselines and sometimes even exceed human performance. ConvNeXt models trained with MAE also exhibit Gestalt-compatible representations, suggesting such sensitivity can arise without attention architectures. However, classification finetuning degrades this ability. Inspired by biological vision, we show that a Top-K activation sparsity mechanism can restore global sensitivity. Our findings identify training conditions that promote or suppress Gestalt-like perception and establish DiSRT as a diagnostic for global structure sensitivity across models.
Abstract (translated)
人类视觉通过诸如封闭性、邻近性和正负形分配等格式塔原则，将局部线索组织成连贯的整体形式。这些功能依赖于全局空间结构。我们研究了现代视觉模型是否表现出类似的特性，并探讨了在何种训练条件下它们会出现这种行为。 我们的发现表明，使用掩码自动编码（MAE）进行训练的Vision Transformers (ViTs) 展现出了与格式塔定律一致的激活模式，包括幻觉轮廓完成、凸性偏好和动态正负形分离。为了探究计算基础，我们假设建模全局依赖关系对于形成类似的格式塔组织是必要的。 为此，我们引入了扭曲空间关系测试台（Distorted Spatial Relationship Testbench, DiSRT），它评估模型对全局空间扰动的敏感度，同时保持局部纹理不变。通过使用DiSRT，我们展示了自监督模型（例如MAE、CLIP）在处理全球结构变化时的表现优于监督基准，并且有时甚至超越了人类表现。此外，经过MAE训练的ConvNeXt模型也表现出与格式塔兼容的表示形式，这表明这种敏感度可以在没有注意力架构的情况下产生。 然而，在进行分类微调后，这些能力会退化。受生物视觉启发，我们展示了一种Top-K激活稀疏机制可以恢复全局敏感性。我们的研究确定了促进或抑制类似格式塔感知的训练条件，并将DiSRT确立为跨模型检测全球结构敏感性的诊断工具。 简而言之，这项研究表明，在特定条件下训练的现代计算机视觉模型能够模拟人类视觉中的某些高级认知过程，特别是那些涉及从局部细节中提取全局信息的过程。这不仅加深了我们对这些模型如何工作的理解，还可能对开发更智能、更具适应性的机器学习系统具有重要影响。
URL
https://arxiv.org/abs/2506.00718
PDF
-
Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
2025-05-31 15:12:04Danfeng li, Hui Zhang, Sheng Wang, Jiacheng Li, Zuxuan WuAbstract
Despite recent advances in diffusion models, top-tier text-to-image (T2I) models still struggle to achieve precise spatial layout control, i.e. accurately generating entities with specified attributes and locations. Segmentation-mask-to-image (S2I) generation has emerged as a promising solution by incorporating pixel-level spatial guidance and regional text prompts. However, existing S2I methods fail to simultaneously ensure semantic consistency and shape consistency. To address these challenges, we propose Seg2Any, a novel S2I framework built upon advanced multimodal diffusion transformers (e.g. FLUX). First, to achieve both semantic and shape consistency, we decouple segmentation mask conditions into regional semantic and high-frequency shape components. The regional semantic condition is introduced by a Semantic Alignment Attention Mask, ensuring that generated entities adhere to their assigned text prompts. The high-frequency shape condition, representing entity boundaries, is encoded as an Entity Contour Map and then introduced as an additional modality via multi-modal attention to guide image spatial structure. Second, to prevent attribute leakage across entities in multi-entity scenarios, we introduce an Attribute Isolation Attention Mask mechanism, which constrains each entity's image tokens to attend exclusively to themselves during image self-attention. To support open-set S2I generation, we construct SACap-1M, a large-scale dataset containing 1 million images with 5.9 million segmented entities and detailed regional captions, along with a SACap-Eval benchmark for comprehensive S2I evaluation. Extensive experiments demonstrate that Seg2Any achieves state-of-the-art performance on both open-set and closed-set S2I benchmarks, particularly in fine-grained spatial and attribute control of entities.
Abstract (translated)
尽管在扩散模型方面取得了近期进展，顶级的文本到图像（T2I）模型仍然难以实现精确的空间布局控制，即准确生成具有指定属性和位置的实体。分割掩码到图像（S2I）生成作为一种有前景的解决方案应运而生，它通过引入像素级别的空间指导和区域文本提示来改进这一过程。然而，现有的S2I方法未能同时确保语义一致性和形状一致性。 为了解决这些挑战，我们提出了Seg2Any，这是一种基于先进多模态扩散变换器（例如FLUX）构建的新型S2I框架。首先，为了实现语义和形状的一致性，我们将分割掩码条件解耦为区域语义和高频形状成分。通过引入一个语义对齐注意力掩码来提出区域语义条件，确保生成的实体符合其分配的文本提示。高频形状条件表示实体边界，并被编码成实体轮廓图，然后作为额外模态通过多模态注意机制引入，以指导图像的空间结构。 其次，在涉及多个实体的情况下，为了防止属性在不同实体之间泄漏，我们引入了一种属性隔离注意力掩码机制，该机制约束每个实体的图像令牌仅在图像自我关注时关注自身。为支持开放集S2I生成，我们构建了SACap-1M，这是一个大规模数据集，包含100万张图片和590万个分割实体及其详细的区域描述文本，以及一个用于全面评估的SACap-Eval基准。 广泛的实验表明，Seg2Any在开放集和封闭集S2I基准测试中均表现出色，在细粒度的空间和属性控制方面尤其突出。
URL
https://arxiv.org/abs/2506.00596
PDF
-
Real-Time Blind Defocus Deblurring for Earth Observation: The IMAGIN-e Mission Approach
2025-05-28 08:52:38Alejandro D. MousistAbstract
This work addresses mechanical defocus in Earth observation images from the IMAGIN-e mission aboard the ISS, proposing a blind deblurring approach adapted to space-based edge computing constraints. Leveraging Sentinel-2 data, our method estimates the defocus kernel and trains a restoration model within a GAN framework, effectively operating without reference images. On Sentinel-2 images with synthetic degradation, SSIM improved by 72.47% and PSNR by 25.00%, confirming the model's ability to recover lost details when the original clean image is known. On IMAGIN-e, where no reference images exist, perceptual quality metrics indicate a substantial enhancement, with NIQE improving by 60.66% and BRISQUE by 48.38%, validating real-world onboard restoration. The approach is currently deployed aboard the IMAGIN-e mission, demonstrating its practical application in an operational space environment. By efficiently handling high-resolution images under edge computing constraints, the method enables applications such as water body segmentation and contour detection while maintaining processing viability despite resource limitations.
Abstract (translated)
这项工作解决了国际空间站（ISS）上IMAGIN-e任务地球观测图像中的机械失焦问题，提出了一种适应基于太空边缘计算限制的盲去模糊方法。通过利用Sentinel-2数据，我们的方法估计了失焦核，并在一个生成对抗网络（GAN）框架内训练了一个恢复模型，能够在没有参考图像的情况下有效运行。在具有合成退化处理的Sentinel-2图像上，SSIM（结构相似性指数）提高了72.47%，PSNR（峰值信噪比）提高了25.00%，这证实了该模型能够恢复已知原始清晰图像时丢失的细节。在没有参考图像存在的IMAGIN-e任务中，感知质量度量表明有了显著提升：NIQE（自然图像质量评价器）提高了60.66%，BRISQUE（基于回归的无监督图像质量评估）提高了48.38%，这验证了其在实际环境中的恢复能力。该方法目前部署于IMAGIN-e任务中，展示了其在操作性太空环境中应用的实际价值。 通过高效处理边缘计算限制下的高分辨率图像，该方法使诸如水体分割和轮廓检测等应用程序得以实现，并且即使在资源受限的情况下也能保持处理的可行性。
URL
https://arxiv.org/abs/2505.22128
PDF
-
MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-on
2025-05-27 15:22:02Guangyuan Li, Siming Zheng, Hao Zhang, Jinwei Chen, Junsheng Luan, Binkai Ou, Lei Zhao, Bo Li, Peng-Tao JiangAbstract
Video Virtual Try-On (VVT) aims to simulate the natural appearance of garments across consecutive video frames, capturing their dynamic variations and interactions with human body motion. However, current VVT methods still face challenges in terms of spatiotemporal consistency and garment content preservation. First, they use diffusion models based on the U-Net, which are limited in their expressive capability and struggle to reconstruct complex details. Second, they adopt a separative modeling approach for spatial and temporal attention, which hinders the effective capture of structural relationships and dynamic consistency across frames. Third, their expression of garment details remains insufficient, affecting the realism and stability of the overall synthesized results, especially during human motion. To address the above challenges, we propose MagicTryOn, a video virtual try-on framework built upon the large-scale video diffusion this http URL replace the U-Net architecture with a diffusion Transformer and combine full self-attention to jointly model the spatiotemporal consistency of videos. We design a coarse-to-fine garment preservation strategy. The coarse strategy integrates garment tokens during the embedding stage, while the fine strategy incorporates multiple garment-based conditions, such as semantics, textures, and contour lines during the denoising stage. Moreover, we introduce a mask-aware loss to further optimize garment region fidelity. Extensive experiments on both image and video try-on datasets demonstrate that our method outperforms existing SOTA methods in comprehensive evaluations and generalizes to in-the-wild scenarios.
Abstract (translated)
视频虚拟试穿（VVT）旨在模拟服装在连续视频帧中的自然外观，捕捉其动态变化及其与人体动作的互动。然而，当前的VVT方法仍然面临着时空一致性和衣物内容保留方面的挑战。 首先，它们使用基于U-Net的扩散模型，这种模型表达能力有限，在重建复杂细节时遇到困难。 其次，它们采用分离式建模方法来处理空间和时间注意力问题，这阻碍了对帧间结构关系及动态一致性有效捕捉的能力。 第三，他们对于衣物细节的表现仍然不足，影响整体合成结果的真实感与稳定性，尤其是在人体动作期间。 为了解决上述挑战，我们提出了MagicTryOn框架。该框架基于大规模视频扩散模型构建，并用扩散Transformer替代U-Net架构，同时结合全面的自注意力机制来共同建模视频的时空一致性。我们设计了一种从粗到细的衣物保留策略：粗略策略在嵌入阶段集成服装令牌，而精细策略则在去噪阶段整合多个基于衣物条件的因素（如语义、纹理和轮廓线）。此外，我们引入了一个遮罩感知损失来进一步优化衣物区域的真实感。 通过针对图像及视频试穿数据集的广泛实验表明，我们的方法在全面评估中超越了现有的最先进的技术，并且能够应用于现实场景。
URL
https://arxiv.org/abs/2505.21325
PDF
-
Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning
2025-05-27 14:15:19Lintao Xu, Yinghao Wang, Chaohui WangAbstract
Occlusion Boundary Estimation (OBE) identifies boundaries arising from both inter-object occlusions and self-occlusion within individual objects, distinguishing intrinsic object edges from occlusion-induced contours to improve scene understanding and 3D reconstruction capacity. This is closely related to Monocular Depth Estimation (MDE), which infers depth from a single image, as occlusion boundaries provide critical geometric cues for resolving depth ambiguities, while depth priors can conversely refine occlusion reasoning in complex scenes. In this paper, we propose a novel network, MoDOT, that first jointly estimates depth and OBs. We propose CASM, a cross-attention multi-scale strip convolution module, leverages mid-level OB features to significantly enhance depth prediction. Additionally, we introduce an occlusion-aware loss function, OBDCL, which encourages sharper and more accurate depth boundaries. Extensive experiments on both real and synthetic datasets demonstrate the mutual benefits of jointly estimating depth and OB, and highlight the effectiveness of our model design. Our method achieves the state-of-the-art (SOTA) on both our proposed synthetic datasets and one popular real dataset, NYUD-v2, significantly outperforming multi-task baselines. Besides, without domain adaptation, results on real-world depth transfer are comparable to the competitors, while preserving sharp occlusion boundaries for geometric fidelity. We will release our code, pre-trained models, and datasets to support future research in this direction.
Abstract (translated)
遮挡边界估计（OBE）旨在识别由物体间遮挡和单个对象内部自我遮挡产生的边界，区分固有的对象边缘与遮挡引起的轮廓，以此来提高场景理解和三维重建的能力。这与单目深度估计（MDE）密切相关，后者通过单一图像推断深度。遮挡边界的几何线索对于解决深度模糊性至关重要，而深度先验信息则可以反过来在复杂场景中改进遮挡推理。 本文提出了一种新的网络模型MoDOT，该模型首先联合估计深度和遮挡边界（OB）。我们提出了一个跨注意力多尺度带状卷积模块CASM，利用中级的OB特征显著提高深度预测能力。此外，还引入了一个感知遮挡的损失函数OBDCL，以鼓励更尖锐、更准确的深度边界。 在真实数据集和合成数据集上的广泛实验表明了联合估计深度和OB之间的相互利益，并突显了我们模型设计的有效性。我们的方法在我们提出的合成数据集中以及流行的NYUD-v2现实数据集中达到了最先进的性能（SOTA），显著超越了多任务基线。此外，无需领域适应的情况下，在实际世界的深度转移结果可以与竞争对手相媲美，同时保持清晰的遮挡边界以确保几何精度。 我们将发布我们的代码、预训练模型和数据集，以支持未来在此方向上的研究工作。
URL
https://arxiv.org/abs/2505.21231
PDF
-
ReassembleNet: Learnable Keypoints and Diffusion for 2D Fresco Reconstruction
2025-05-27 12:38:06Adeela Islam, Stefano Fiorini, Stuart James, Pietro Morerio, Alessio Del BueAbstract
The task of reassembly is a significant challenge across multiple domains, including archaeology, genomics, and molecular docking, requiring the precise placement and orientation of elements to reconstruct an original structure. In this work, we address key limitations in state-of-the-art Deep Learning methods for reassembly, namely i) scalability; ii) multimodality; and iii) real-world applicability: beyond square or simple geometric shapes, realistic and complex erosion, or other real-world problems. We propose ReassembleNet, a method that reduces complexity by representing each input piece as a set of contour keypoints and learning to select the most informative ones by Graph Neural Networks pooling inspired techniques. ReassembleNet effectively lowers computational complexity while enabling the integration of features from multiple modalities, including both geometric and texture data. Further enhanced through pretraining on a semi-synthetic dataset. We then apply diffusion-based pose estimation to recover the original structure. We improve on prior methods by 55% and 86% for RMSE Rotation and Translation, respectively.
Abstract (translated)
重组任务在考古学、基因组学和分子对接等多个领域都面临着重大挑战，需要精确地定位和定向各个元素以重建原始结构。在这项工作中，我们解决了最先进的深度学习方法在重组任务中的三个关键限制：i) 可扩展性；ii) 多模态处理能力；iii) 现实世界适用性（即超出正方形或其他简单几何形状的范围，包括现实世界的复杂侵蚀问题等）。为此，我们提出了一种名为ReassembleNet的方法。该方法通过将每个输入块表示为一组轮廓关键点，并利用图神经网络池化技术来学习选择最具信息量的关键点，从而降低了任务的复杂性。这种方法在减少计算复杂度的同时，还支持多模态特征（包括几何和纹理数据）的集成。通过半合成数据集上的预训练进一步增强ReassembleNet的能力后，我们接着应用基于扩散的方法进行姿态估计以恢复原始结构。我们的方法分别使旋转和平移方向上均方根误差（RMSE）相对于先前的方法提高了55% 和86%。
URL
https://arxiv.org/abs/2505.21117
PDF
-
VibE-SVC: Vibrato Extraction with High-frequency F0 Contour for Singing Voice Conversion
2025-05-27 06:56:13Joon-Seung Choi, Dong-Min Byun, Hyung-Seok Oh, Seong-Whan LeeAbstract
Controlling singing style is crucial for achieving an expressive and natural singing voice. Among the various style factors, vibrato plays a key role in conveying emotions and enhancing musical depth. However, modeling vibrato remains challenging due to its dynamic nature, making it difficult to control in singing voice conversion. To address this, we propose VibESVC, a controllable singing voice conversion model that explicitly extracts and manipulates vibrato using discrete wavelet transform. Unlike previous methods that model vibrato implicitly, our approach decomposes the F0 contour into frequency components, enabling precise transfer. This allows vibrato control for enhanced flexibility. Experimental results show that VibE-SVC effectively transforms singing styles while preserving speaker similarity. Both subjective and objective evaluations confirm high-quality conversion.
Abstract (translated)
控制歌唱风格对于实现富有表现力和自然的歌声至关重要。在各种风格因素中，颤音（vibrato）在传达情感和增强音乐深度方面起着关键作用。然而，由于其动态特性，模拟颤音仍然具有挑战性，并且难以在声乐转换中进行精确控制。 为了应对这一挑战，我们提出了VibESVC模型，这是一种可控制的歌唱声音转换模型，它使用离散小波变换（discrete wavelet transform）明确地提取和操作颤音。与以往隐式建模颤音的方法不同，我们的方法将基频轮廓分解为不同的频率成分，从而能够进行精确的声音转移。这使得我们可以更灵活地控制颤音。 实验结果显示，VibESVC模型能够在转换歌唱风格的同时保持说话人的一致性。无论是主观评价还是客观评估都确认了这种高质量的转换效果。
URL
https://arxiv.org/abs/2505.20794
PDF
-
Decoding Speaker-Normalized Pitch from EEG for Mandarin Perception
2025-05-26 07:46:48Jiaxin Chen, Yiming Wang, Ziyu Zhang, Jiayang Han, Yin-Long Liu, Rui Feng, Xiuyuan Liang, Zhen-Hua Ling, Jiahong YuanAbstract
The same speech content produced by different speakers exhibits significant differences in pitch contour, yet listeners' semantic perception remains unaffected. This phenomenon may stem from the brain's perception of pitch contours being independent of individual speakers' pitch ranges. In this work, we recorded electroencephalogram (EEG) while participants listened to Mandarin monosyllables with varying tones, phonemes, and speakers. The CE-ViViT model is proposed to decode raw or speaker-normalized pitch contours directly from EEG. Experimental results demonstrate that the proposed model can decode pitch contours with modest errors, achieving performance comparable to state-of-the-art EEG regression methods. Moreover, speaker-normalized pitch contours were decoded more accurately, supporting the neural encoding of relative pitch.
Abstract (translated)
不同说话者发出的相同演讲内容在音高曲线上表现出显著差异，但听众的语义感知并未受到影响。这一现象可能源于大脑对音高曲线的感知与个别说话人的音域独立。在这项工作中，我们记录了参与者聆听具有不同声调、音素和说话者的普通话单音节词时的脑电图（EEG）。提出了CE-ViViT模型来直接从EEG解码原始或说话人归一化的音高曲线。实验结果表明，所提出的模型可以以适度误差解码音高曲线，并且性能与最先进的EEG回归方法相当。此外，更准确地解码了说话人归一化的音高曲线，支持相对音高的神经编码。
URL
https://arxiv.org/abs/2505.19626
PDF
-
Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation
2025-05-26 02:32:22Yuhao He, Jinyu Tian, Haiwei Wu, Jianqing LiAbstract
The rapid advancement of diffusion models has enhanced their image inpainting and editing capabilities but also introduced significant societal risks. Adversaries can exploit user images from social media to generate misleading or harmful content. While adversarial perturbations can disrupt inpainting, global perturbation-based methods fail in mask-guided editing tasks due to spatial constraints. To address these challenges, we propose Structure Disruption Attack (SDA), a powerful protection framework for safeguarding sensitive image regions against inpainting-based editing. Building upon the contour-focused nature of self-attention mechanisms of diffusion models, SDA optimizes perturbations by disrupting queries in self-attention during the initial denoising step to destroy the contour generation process. This targeted interference directly disrupts the structural generation capability of diffusion models, effectively preventing them from producing coherent images. We validate our motivation through visualization techniques and extensive experiments on public datasets, demonstrating that SDA achieves state-of-the-art (SOTA) protection performance while maintaining strong robustness.
Abstract (translated)
扩散模型的迅速发展增强了其图像修复和编辑能力，但也带来了重要的社会风险。对手可以利用社交媒体上的用户图片生成误导性或有害内容。虽然对抗性扰动能够干扰图像修复过程，但基于全局扰动的方法在引导式编辑任务中由于空间约束而失效。为了解决这些挑战，我们提出了结构破坏攻击（SDA），这是一种强大的保护框架，用于防御扩散模型对敏感图像区域进行修复式编辑的威胁。基于扩散模型中的自注意力机制以轮廓为重点的特点，SDA通过在初始去噪步骤中干扰自注意力中的查询来优化扰动，从而破坏轮廓生成过程。这种有针对性的干扰直接中断了扩散模型的结构生成能力，有效防止其产生连贯图像。我们通过可视化技术和在公开数据集上的广泛实验验证了我们的动机，证明SDA在保持强大鲁棒性的同时达到了最先进的（SOTA）保护性能。
URL
https://arxiv.org/abs/2505.19425
PDF
-
LlamaSeg: Image Segmentation via Autoregressive Mask Generation
2025-05-26 02:22:41Jiru Deng, Tengjin Weng, Tianyu Yang, Wenhan Luo, Zhiheng Li, Wenhao JiangAbstract
We present LlamaSeg, a visual autoregressive framework that unifies multiple image segmentation tasks via natural language instructions. We reformulate image segmentation as a visual generation problem, representing masks as "visual" tokens and employing a LLaMA-style Transformer to predict them directly from image inputs. By adhering to the next-token prediction paradigm, our approach naturally integrates segmentation tasks into autoregressive architectures. To support large-scale training, we introduce a data annotation pipeline and construct the SA-OVRS dataset, which contains 2M segmentation masks annotated with over 5,800 open-vocabulary labels or diverse textual descriptions, covering a wide spectrum of real-world scenarios. This enables our model to localize objects in images based on text prompts and to generate fine-grained masks. To more accurately evaluate the quality of masks produced by visual generative models, we further propose a composite metric that combines Intersection over Union (IoU) with Average Hausdorff Distance (AHD), offering a more precise assessment of contour fidelity. Experimental results demonstrate that our method surpasses existing generative models across multiple datasets and yields more detailed segmentation masks.
Abstract (translated)
我们介绍了LlamaSeg，这是一种视觉自回归框架，通过自然语言指令统一了多种图像分割任务。我们将图像分割重新表述为一个视觉生成问题，将掩码表示为“视觉”令牌，并采用LLaMA风格的Transformer模型直接从图像输入预测它们。通过遵循下一个标记预测范式，我们的方法自然地将分割任务整合到自回归架构中。 为了支持大规模训练，我们引入了一个数据标注流水线并构建了SA-OVRS数据集，该数据集中包含200万张带有超过5800个开放词汇标签或多样化文本描述的分割掩码，涵盖了广泛的现实世界场景。这使得我们的模型能够在基于文本提示的情况下在图像中定位对象，并生成精细的掩码。 为了更准确地评估视觉生成模型产生的掩码质量，我们进一步提出了一种复合度量标准，该标准结合了交并比（IoU）和平均豪斯多夫距离（AHD），为轮廓保真度提供了更为精确的评价。实验结果表明，我们的方法在多个数据集上超越了现有的生成模型，并产生了更详细的分割掩码。
URL
https://arxiv.org/abs/2505.19422
PDF
-
Towards Generalized Proactive Defense against Face Swappingwith Contour-Hybrid Watermark
2025-05-25 10:29:23Ruiyang Xia, Dawei Zhou, Decheng Liu, Lin Yuan, Jie Li, Nannan Wang, Xinbo GaoAbstract
Face swapping, recognized as a privacy and security concern, has prompted considerable defensive research. With the advancements in AI-generated content, the discrepancies between the real and swapped faces have become nuanced. Considering the difficulty of forged traces detection, we shift the focus to the face swapping purpose and proactively embed elaborate watermarks against unknown face swapping techniques. Given that the constant purpose is to swap the original face identity while preserving the background, we concentrate on the regions surrounding the face to ensure robust watermark generation, while embedding the contour texture and face identity information to achieve progressive image determination. The watermark is located in the facial contour and contains hybrid messages, dubbed the contour-hybrid watermark (CMark). Our approach generalizes face swapping detection without requiring any swapping techniques during training and the storage of large-scale messages in advance. Experiments conducted across 8 face swapping techniques demonstrate the superiority of our approach compared with state-of-the-art passive and proactive detectors while achieving a favorable balance between the image quality and watermark robustness.
Abstract (translated)
面部替换，作为一种隐私和安全问题，已经引起了大量防御性研究的关注。随着人工智能生成内容的进步，真实人脸与被替换后的人脸之间的差异变得更加微妙。鉴于伪造痕迹检测的难度，我们将重点转移到面部替换的目的上，并主动嵌入复杂的水印以对抗未知的面部替换技术。考虑到面部替换的主要目的是在保持背景不变的情况下更换原始面部身份，我们专注于面部周围的区域来确保生成稳健的水印，同时嵌入口轮纹理和面部识别信息，实现渐进式图像判定。该水印位于面部轮廓中，并包含混合消息，被称为轮廓混合水印（CMark）。我们的方法可以泛化面部替换检测，在训练过程中不需要任何特定的替换技术，也不需要提前存储大规模的消息。 实验在八种不同的面部替换技术上进行，证明了我们方法相较于现有的被动和主动检测器具有优越性，同时实现了图像质量和水印稳健性的良好平衡。
URL
https://arxiv.org/abs/2505.19081
PDF
-
Audio-to-Audio Emotion Conversion With Pitch And Duration Style Transfer
2025-05-23 09:18:12Soumya Dutta, Avni Jain, Sriram GanapathyAbstract
Given a pair of source and reference speech recordings, audio-to-audio (A2A) style transfer involves the generation of an output speech that mimics the style characteristics of the reference while preserving the content and speaker attributes of the source. In this paper, we propose a novel framework, termed as A2A Zero-shot Emotion Style Transfer (A2A-ZEST), that enables the transfer of reference emotional attributes to the source while retaining its speaker and speech contents. The A2A-ZEST framework consists of an analysis-synthesis pipeline, where the analysis module decomposes speech into semantic tokens, speaker representations, and emotion embeddings. Using these representations, a pitch contour estimator and a duration predictor are learned. Further, a synthesis module is designed to generate speech based on the input representations and the derived factors. This entire paradigm of analysis-synthesis is trained purely in a self-supervised manner with an auto-encoding loss. For A2A emotion style transfer, the emotion embedding extracted from the reference speech along with the rest of the representations from the source speech are used in the synthesis module to generate the style translated speech. In our experiments, we evaluate the converted speech on content/speaker preservation (w.r.t. source) as well as on the effectiveness of the emotion style transfer (w.r.t. reference). The proposal, A2A-ZEST, is shown to improve over other prior works on these evaluations, thereby enabling style transfer without any parallel training data. We also illustrate the application of the proposed work for data augmentation in emotion recognition tasks.
Abstract (translated)
给定一对源录音和参考录音，音频到音频（A2A）风格迁移旨在生成一个输出语音，在保留源语音内容和说话人属性的同时模仿参考录音的风格特性。本文提出了一种新颖框架，命名为 A2A 零样本情感风格迁移 (A2A-ZEST)，该框架允许将参考语音的情感属性转移到源语音中，同时保持其说话人身份和语音内容不变。A2A-ZEST 框架包括一个分析-综合流水线，在其中分析模块将语音分解为语义标记、说话人表示和情感嵌入。利用这些表示形式学习音高曲线估计器和持续时间预测器。此外，设计了一个合成模块，用于根据输入表示及推导出的参数生成语音。整个分析-综合流程完全以自监督方式训练，并使用自动编码损失函数。 对于 A2A 情感风格迁移，从参考录音中提取的情感嵌入与源录音中的其余表示共同被用来在合成模块内生成风格转换后的语音。在实验过程中，我们评估了转换语音在保留内容和说话人身份（相对于原始音频）以及情感风格转移效果（相对于参考音频）方面的表现。A2A-ZEST 方法在这类评价中超越了先前的工作，从而实现在没有平行训练数据的情况下进行样式迁移。此外，我们也展示了所提出的方案在情感识别任务中的数据增强应用。
URL
https://arxiv.org/abs/2505.17655
PDF
-
Enhancing Shape Perception and Segmentation Consistency for Industrial Image Inspection
2025-05-19 09:57:00Guoxuan Mao, Ting Cao, Ziyang Li, Yuan DongAbstract
Semantic segmentation stands as a pivotal research focus in computer vision. In the context of industrial image inspection, conventional semantic segmentation models fail to maintain the segmentation consistency of fixed components across varying contextual environments due to a lack of perception of object contours. Given the real-time constraints and limited computing capability of industrial image detection machines, it is also necessary to create efficient models to reduce computational complexity. In this work, a Shape-Aware Efficient Network (SPENet) is proposed, which focuses on the shapes of objects to achieve excellent segmentation consistency by separately supervising the extraction of boundary and body information from images. In SPENet, a novel method is introduced for describing fuzzy boundaries to better adapt to real-world scenarios named Variable Boundary Domain (VBD). Additionally, a new metric, Consistency Mean Square Error(CMSE), is proposed to measure segmentation consistency for fixed components. Our approach attains the best segmentation accuracy and competitive speed on our dataset, showcasing significant advantages in CMSE among numerous state-of-the-art real-time segmentation networks, achieving a reduction of over 50% compared to the previously top-performing models.
Abstract (translated)
语义分割是计算机视觉领域的核心研究方向之一。在工业图像检测的背景下，传统的语义分割模型由于缺乏对物体轮廓的认知，在不同上下文环境中难以保持固定组件的一致性分割结果。鉴于工业图像检测设备实时性和计算能力有限的要求，创建高效的模型以降低计算复杂度也显得尤为重要。 本文提出了一种基于形状感知的高效网络（Shape-Aware Efficient Network, SPENet），该网络通过分别监督图像边界和主体信息的提取来实现优异的分割一致性，并专注于物体形状。在SPENet中引入了一种名为可变边界域（Variable Boundary Domain, VBD）的新方法，用于描述模糊边界以更好地适应现实世界场景。 此外，本文还提出了一项新的度量标准——一致性均方误差(Consistency Mean Square Error, CMSE)，用以衡量固定组件的分割一致性。我们的方法在我们构建的数据集中取得了最佳的分割精度和竞争力的速度，在众多实时语义分割网络中展现了显著的优势，并且在CMSE指标上相比之前性能最优的模型减少了超过50%。
URL
https://arxiv.org/abs/2505.14718
PDF
-
Dynamic Graph Induced Contour-aware Heat Conduction Network for Event-based Object Detection
2025-05-19 09:44:01Xiao Wang, Yu Jin, Lan Chen, Bo Jiang, Lin Zhu, Yonghong Tian, Jin Tang, Bin LuoAbstract
Event-based Vision Sensors (EVS) have demonstrated significant advantages over traditional RGB frame-based cameras in low-light conditions, high-speed motion capture, and low latency. Consequently, object detection based on EVS has attracted increasing attention from researchers. Current event stream object detection algorithms are typically built upon Convolutional Neural Networks (CNNs) or Transformers, which either capture limited local features using convolutional filters or incur high computational costs due to the utilization of self-attention. Recently proposed vision heat conduction backbone networks have shown a good balance between efficiency and accuracy; however, these models are not specifically designed for event stream data. They exhibit weak capability in modeling object contour information and fail to exploit the benefits of multi-scale features. To address these issues, this paper proposes a novel dynamic graph induced contour-aware heat conduction network for event stream based object detection, termed CvHeat-DET. The proposed model effectively leverages the clear contour information inherent in event streams to predict the thermal diffusivity coefficients within the heat conduction model, and integrates hierarchical structural graph features to enhance feature learning across multiple scales. Extensive experiments on three benchmark datasets for event stream-based object detection fully validated the effectiveness of the proposed model. The source code of this paper will be released on this https URL.
Abstract (translated)
基于事件的视觉传感器（EVS）在低光环境、高速运动捕捉和低延迟方面展示了相对于传统RGB帧相机的巨大优势。因此，基于EVS的对象检测吸引了越来越多的研究人员的关注。目前，基于事件流的对象检测算法通常建立在卷积神经网络（CNN）或Transformer的基础上，前者利用卷积滤波器捕获有限的局部特征，而后者则由于使用自注意力机制而导致计算成本高昂。最近提出的视觉热传导骨干网表现出效率和准确性之间的良好平衡；然而，这些模型并没有专门针对事件流数据进行设计，因此在建模物体轮廓信息方面能力较弱，并且未能充分利用多尺度特性带来的优势。 为了解决这些问题，本文提出了一种新颖的动态图诱导的感知轮廓的热传导网络用于基于事件流的对象检测，称为CvHeat-DET。所提出的模型有效利用了事件流中固有的清晰轮廓信息来预测热传导模型中的热扩散系数，并通过整合分层结构化图形特征来增强多尺度特征学习。在三个基准数据集上进行的广泛实验全面验证了所提出模型的有效性。 该论文的源代码将在此网址发布：[https URL]（请根据实际情况替换为具体的URL）。
URL
https://arxiv.org/abs/2505.12908
PDF
