Contour
-
Revisiting Audio-Visual Segmentation with Vision-Centric Transformer
2025-06-30 08:40:36Shaofei Huang, Rui Ling, Tianrui Hui, Hongyu Li, Xu Zhou, Shifeng Zhang, Si Liu, Richang Hong, Meng WangAbstract
Audio-Visual Segmentation (AVS) aims to segment sound-producing objects in video frames based on the associated audio signal. Prevailing AVS methods typically adopt an audio-centric Transformer architecture, where object queries are derived from audio features. However, audio-centric Transformers suffer from two limitations: perception ambiguity caused by the mixed nature of audio, and weakened dense prediction ability due to visual detail loss. To address these limitations, we propose a new Vision-Centric Transformer (VCT) framework that leverages vision-derived queries to iteratively fetch corresponding audio and visual information, enabling queries to better distinguish between different sounding objects from mixed audio and accurately delineate their contours. Additionally, we also introduce a Prototype Prompted Query Generation (PPQG) module within our VCT framework to generate vision-derived queries that are both semantically aware and visually rich through audio prototype prompting and pixel context grouping, facilitating audio-visual information aggregation. Extensive experiments demonstrate that our VCT framework achieves new state-of-the-art performances on three subsets of the AVSBench dataset. The code is available at this https URL.
Abstract (translated)
音频视觉分割（AVS）的目标是基于关联的音频信号在视频帧中对发声物体进行分割。现有的AVS方法通常采用以音频为中心的Transformer架构，其中对象查询是从音频特征派生出来的。然而，这类音频中心的Transformer存在两个限制：一是由于音频混合特性导致的感觉模糊问题，二是因为视觉细节损失而导致密集预测能力减弱的问题。 为了解决这些问题，我们提出了一种新的基于视觉的Transformer（VCT）框架，该框架利用来自视觉的信息生成查询，并迭代地获取相应的音频和视觉信息。这使得查询能够更好地从混合音频中区分不同的发声物体，并准确描绘它们的轮廓。 此外，我们在VCT框架内还引入了一个原型提示查询生成（PPQG）模块，通过音频原型提示和像素上下文分组来生成既具有语义感知又能丰富视觉特征的基于视觉的查询。这有助于实现音频与视觉信息的有效聚合。 经过广泛的实验验证，我们的VCT框架在AVSBench数据集的三个子集中实现了新的最先进性能。相关代码可在[此链接]（请将此处的URL替换为实际提供的URL）获取。
URL
https://arxiv.org/abs/2506.23623
PDF
-
Multimodal image registration for effective thermographic fever screening
2025-06-29 20:02:23C. Y. N. Dwith, Pejhman Ghassemi, Joshua Pfefer, Jon Casamento, Quanzeng WangAbstract
Fever screening based on infrared thermographs (IRTs) is a viable mass screening approach during infectious disease pandemics, such as Ebola and SARS, for temperature monitoring in public places like hospitals and airports. IRTs have found to be powerful, quick and non-invasive methods to detect elevated temperatures. Moreover, regions medially adjacent to the inner canthi (called the canthi regions in this paper) are preferred sites for fever screening. Accurate localization of the canthi regions can be achieved through multi-modal registration of infrared (IR) and white-light images. We proposed a registration method through a coarse-fine registration strategy using different registration models based on landmarks and edge detection on eye contours. We evaluated the registration accuracy to be within 2.7 mm, which enables accurate localization of the canthi regions.
Abstract (translated)
基于红外热图（IRT）的发热筛查在埃博拉和SARS等传染病大流行期间，是一种可行的大规模筛查方法，适用于医院、机场等人流密集场所的体温监测。IRT已被证明是检测高体温的有效、快速且无创的方法。此外，在此研究中被称为内眼角区域（即靠近内眼角的区域）被认为是发热筛查的理想部位。通过红外图像和白光图像的多模态配准技术，可以实现这些内眼角区域的精确定位。 我们提出了一种粗细结合的注册方法策略，该策略采用基于地标和眼睛轮廓边缘检测的不同注册模型。我们评估了这种配准方法的准确性在2.7毫米以内，这使得能够精确地定位内眼角区域。
URL
https://arxiv.org/abs/2507.02955
PDF
-
SmoothSinger: A Conditional Diffusion Model for Singing Voice Synthesis with Multi-Resolution Architecture
2025-06-26 17:07:45Kehan Sui, Jinxu Xiang, Fang JinAbstract
Singing voice synthesis (SVS) aims to generate expressive and high-quality vocals from musical scores, requiring precise modeling of pitch, duration, and articulation. While diffusion-based models have achieved remarkable success in image and video generation, their application to SVS remains challenging due to the complex acoustic and musical characteristics of singing, often resulting in artifacts that degrade naturalness. In this work, we propose SmoothSinger, a conditional diffusion model designed to synthesize high quality and natural singing voices. Unlike prior methods that depend on vocoders as a final stage and often introduce distortion, SmoothSinger refines low-quality synthesized audio directly in a unified framework, mitigating the degradation associated with two-stage pipelines. The model adopts a reference-guided dual-branch architecture, using low-quality audio from any baseline system as a reference to guide the denoising process, enabling more expressive and context-aware synthesis. Furthermore, it enhances the conventional U-Net with a parallel low-frequency upsampling path, allowing the model to better capture pitch contours and long term spectral dependencies. To improve alignment during training, we replace reference audio with degraded ground truth audio, addressing temporal mismatch between reference and target signals. Experiments on the Opencpop dataset, a large-scale Chinese singing corpus, demonstrate that SmoothSinger achieves state-of-the-art results in both objective and subjective evaluations. Extensive ablation studies confirm its effectiveness in reducing artifacts and improving the naturalness of synthesized voices.
Abstract (translated)
歌声合成（SVS）的目标是从乐谱生成具有表现力和高质量的声乐，这需要对音高、时长和发音进行精确建模。虽然基于扩散的方法在图像和视频生成方面取得了显著的成功，但由于歌唱中复杂的声学和音乐特性，这些方法在SVS中的应用仍然面临挑战，常常导致降低自然度的声音伪影。为此，我们提出了SmoothSinger，这是一种条件式扩散模型，旨在合成高质量且自然的歌声。与以往依赖于编码器作为最终阶段的方法不同，并且这些方法往往会引入失真，SmoothSinger直接在一个统一框架中对低质量的合成音频进行精炼，从而减轻了两阶段流水线所带来的降质问题。 该模型采用了参考引导式双分支架构，使用任何基准系统生成的低质量音频作为参考来指导去噪过程，这使得更具有表现力和上下文感知的合成成为可能。此外，它通过添加一个并行的低频上采样路径增强了传统的U-Net结构，从而使模型能够更好地捕捉音高轮廓和长期频谱依赖关系。 为了在训练过程中改善对齐情况，我们用退化的真实音频替换了参考音频，解决了参考信号与目标信号之间的时序不匹配问题。在大规模中文歌唱语料库Opencpop数据集上的实验表明，SmoothSinger在客观评价和主观评价中均达到了当前最佳水平。广泛的消融研究表明，它在减少伪影和提升合成声音的自然度方面非常有效。
URL
https://arxiv.org/abs/2506.21478
PDF
-
Image Segmentation using Chan-Vese Active Contours
2025-06-24 06:16:56Pranav Shenoy K. PAbstract
This paper presents a comprehensive derivation and implementation of the Chan-Vese active contour model for image segmentation. The model, derived from the Mumford-Shah variational framework, evolves contours based on regional intensity differences rather than image gradients, making it highly effective for segmenting noisy images or images with weak boundaries. We provide a rigorous mathematical derivation of the level set formulation, including detailed treatment of each energy term using the divergence theorem and curve evolution theory. The resulting algorithm is implemented in Python using finite difference methods with special care to numerical stability, including an upwind entropy scheme and curvature-based regularization. Experimental results on medical and synthetic images demonstrate accurate segmentation, robustness to noise, and superior performance compared to classical edge-based methods. This study confirms the suitability of the Chan-Vese model for complex segmentation tasks and highlights its potential for use in real-world imaging applications.
Abstract (translated)
本文介绍了Chan-Vese活动轮廓模型在图像分割中的全面推导和实现。该模型基于Mumford-Shah变分框架，通过区域强度差异来演化轮廓线，而不是依赖于图像梯度，因此对于处理噪声大或边界弱的图像非常有效。我们提供了对水平集形式化的严格的数学推导，包括使用散度定理和曲线演化理论详细分析每个能量项。 算法采用Python实现，并利用有限差分方法进行数值稳定性的特别注意，其中包括迎风熵方案和基于曲率的正则化技术。通过医学图像和合成图像的实验结果表明，该模型能够实现准确的分割、对噪声具有鲁棒性，并且相较于传统的边缘基础的方法表现出更优越的性能。 本研究确认了Chan-Vese模型在复杂分割任务中的适用性，并强调其在实际成像应用中使用的潜力。
URL
https://arxiv.org/abs/2506.19344
PDF
-
A Robust Method for Pitch Tracking in the Frequency Following Response using Harmonic Amplitude Summation Filterbank
2025-06-24 02:33:18Sajad Sadeghkhani, Maryam Karimi Boroujeni, Hilmi R. Dajani, Saeid R. Seydnejad, Christian Gigu\`ereAbstract
The Frequency Following Response (FFR) reflects the brain's neural encoding of auditory stimuli including speech. Because the fundamental frequency (F0), a physical correlate of pitch, is one of the essential features of speech, there has been particular interest in characterizing the FFR at F0, especially when F0 varies over time. The standard method for extracting F0 in FFRs has been the Autocorrelation Function (ACF). This paper investigates harmonic-structure-based F0 estimation algorithms, originally developed for speech and music, and resolves their poor performance when applied to FFRs in two steps. Firstly, given that unlike in speech or music, stimulus F0 of FFRs is already known, we introduce a stimulus-aware filterbank that selectively aggregates amplitudes at F0 and its harmonics while suppressing noise at non-harmonic frequencies. This method, called Harmonic Amplitude Summation (HAS), evaluates F0 candidates only within a range centered around the stimulus F0. Secondly, unlike other pitch tracking methods that select the highest peak, our method chooses the most prominent one, as it better reflects the underlying periodicity of FFRs. To the best of our knowledge, this is the first study to propose an F0 estimation algorithm for FFRs that relies on harmonic structure. Analyzing recorded FFRs from 16 normal hearing subjects to 4 natural speech stimuli with a wide F0 variation from 89 Hz to 452 Hz showed that this method outperformed ACF by reducing the average Root-Mean-Square-Error (RMSE) within each response and stimulus F0 contour pair by 8.8% to 47.4%, depending on the stimulus.
Abstract (translated)
频率跟随响应（FFR）反映了大脑对包括语音在内的听觉刺激的神经编码。由于基频（F0），即音高的物理对应物，是语音的基本特征之一，因此在研究FFR中的F0特性时，尤其是在F0随时间变化的情况下，人们对此特别感兴趣。传统上，自相关函数（ACF）被用作从FFR中提取F0的标准方法。本文探讨了基于谐波结构的F0估计算法，这些算法最初是为了语音和音乐开发的，并通过两个步骤解决了当应用于FFRs时性能不佳的问题。 首先，考虑到与语音或音乐不同，在FFRs中的刺激F0是已知的，我们引入了一种了解刺激的滤波器组。这种名为谐振幅叠加（HAS）的方法选择性地在F0及其谐音频率处聚合幅度，并抑制非谐音频率的噪声。这种方法仅在一个围绕刺激F0为中心的范围内评估F0候选值。 其次，不同于其他音高追踪方法会选择最高峰，我们的方法选择了最显著的一个峰值，因为这更好地反映了FFRs中的周期性特征。据我们所知，这是首次提出依赖于谐波结构来估计FFR中基频（F0）的方法的研究。 通过对16名正常听力受试者对4种自然语音刺激（其基频范围从89Hz到452Hz）的记录进行分析，结果显示该方法在减少每个响应和刺激F0轮廓对之间的平均均方根误差（RMSE）方面优于ACF方法。具体而言，根据不同的刺激类型，这种新方法将RMSE减少了8.8%至47.4%不等。
URL
https://arxiv.org/abs/2506.19253
PDF
-
MARL-MambaContour: Unleashing Multi-Agent Deep Reinforcement Learning for Active Contour Optimization in Medical Image Segmentation
2025-06-23 14:22:49Ruicheng Zhang, Yu Sun, Zeyu Zhang, Jinai Li, Xiaofan Liu, Au Hoi Fan, Haowei Guo, Puxin YanAbstract
We introduce MARL-MambaContour, the first contour-based medical image segmentation framework based on Multi-Agent Reinforcement Learning (MARL). Our approach reframes segmentation as a multi-agent cooperation task focused on generate topologically consistent object-level contours, addressing the limitations of traditional pixel-based methods which could lack topological constraints and holistic structural awareness of anatomical regions. Each contour point is modeled as an autonomous agent that iteratively adjusts its position to align precisely with the target boundary, enabling adaptation to blurred edges and intricate morphologies common in medical images. This iterative adjustment process is optimized by a contour-specific Soft Actor-Critic (SAC) algorithm, further enhanced with the Entropy Regularization Adjustment Mechanism (ERAM) which dynamically balance agent exploration with contour smoothness. Furthermore, the framework incorporates a Mamba-based policy network featuring a novel Bidirectional Cross-attention Hidden-state Fusion Mechanism (BCHFM). This mechanism mitigates potential memory confusion limitations associated with long-range modeling in state space models, thereby facilitating more accurate inter-agent information exchange and informed decision-making. Extensive experiments on five diverse medical imaging datasets demonstrate the state-of-the-art performance of MARL-MambaContour, highlighting its potential as an accurate and robust clinical application.
Abstract (translated)
我们介绍了MARL-MambaContour，这是首个基于多智能体强化学习（MARL）的轮廓导向医学图像分割框架。我们的方法将分割任务重新定义为一个多代理合作问题，专注于生成具有拓扑一致性的对象级轮廓，以此来解决传统像素基础方法缺乏拓扑约束和整体结构感知的问题，这些问题在解剖区域中尤为明显。每个轮廓点都被建模成一个自治的智能体，它们会迭代地调整自己的位置以精确对齐目标边界，从而能够适应医学图像中常见的模糊边缘和复杂形态。 这一迭代调整过程通过特定于轮廓的软行为者-评论家（SAC）算法进行优化，并进一步利用熵调节机制（ERAM），该机制动态平衡了智能体探索与轮廓平滑度之间的关系。此外，框架还集成了一个基于蟒蛇策略网络，其中包括一种新颖的双向交叉注意力隐藏状态融合机制（BCHFM）。这一机制减轻了长距离建模在状态空间模型中可能产生的记忆混淆限制，从而促进更准确的代理间信息交换和知情决策制定。 广泛的实验表明，在五个不同的医学成像数据集上，MARL-MambaContour展现了顶尖水平的表现，突显了其作为精确且稳健临床应用的巨大潜力。
URL
https://arxiv.org/abs/2506.18679
PDF
-
RaCalNet: Radar Calibration Network for Sparse-Supervised Metric Depth Estimation
2025-06-18 15:35:16Xingrui Qin, Wentao Zhao, Chuan Cao, Yihe Niu, Houcheng Jiang, Jingchuan WangAbstract
Dense metric depth estimation using millimeter-wave radar typically requires dense LiDAR supervision, generated via multi-frame projection and interpolation, to guide the learning of accurate depth from sparse radar measurements and RGB images. However, this paradigm is both costly and data-intensive. To address this, we propose RaCalNet, a novel framework that eliminates the need for dense supervision by using sparse LiDAR to supervise the learning of refined radar measurements, resulting in a supervision density of merely around 1% compared to dense-supervised methods. Unlike previous approaches that associate radar points with broad image regions and rely heavily on dense labels, RaCalNet first recalibrates and refines sparse radar points to construct accurate depth priors. These priors then serve as reliable anchors to guide monocular depth prediction, enabling metric-scale estimation without resorting to dense supervision. This design improves structural consistency and preserves fine details. Despite relying solely on sparse supervision, RaCalNet surpasses state-of-the-art dense-supervised methods, producing depth maps with clear object contours and fine-grained textures. Extensive experiments on the ZJU-4DRadarCam dataset and real-world deployment scenarios demonstrate its effectiveness, reducing RMSE by 35.30% and 34.89%, respectively.
Abstract (translated)
使用毫米波雷达进行密集度量深度估计通常需要通过多帧投影和插值生成的密集LiDAR监督来指导从稀疏雷达测量和RGB图像中学习准确的深度信息。然而，这种范式既昂贵又需要大量的数据。为了解决这个问题，我们提出了RaCalNet，这是一种新的框架，它通过使用稀疏LiDAR来监督精炼后的雷达测量的学习过程，从而消除了对密集监督的需求，其监督密度仅为密集监督方法的大约1%。 与之前的方法将雷达点关联到宽泛的图像区域并依赖于密集标签不同，RaCalNet首先重新校准和细化稀疏雷达点以构建准确的深度先验。这些先验作为可靠的锚点来引导单目深度预测，从而实现没有密集监督下的度量级估计。这种设计提高了结构一致性，并保留了细粒度细节。 尽管仅依赖于稀疏监督，RaCalNet在深度图生成方面超过了最先进的密集监督方法，产生了具有清晰物体轮廓和细腻纹理的深度图。在ZJU-4DRadarCam数据集和真实世界部署场景中的广泛实验验证了其有效性，分别将RMSE降低了35.30%和34.89%。
URL
https://arxiv.org/abs/2506.15560
PDF
-
FADPNet: Frequency-Aware Dual-Path Network for Face Super-Resolution
2025-06-17 02:33:42Siyu Xu, Wenjie Li, Guangwei Gao, Jian Yang, Guo-Jun Qi, Chia-Wen LinAbstract
Face super-resolution (FSR) under limited computational costs remains an open problem. Existing approaches typically treat all facial pixels equally, resulting in suboptimal allocation of computational resources and degraded FSR performance. CNN is relatively sensitive to high-frequency facial features, such as component contours and facial outlines. Meanwhile, Mamba excels at capturing low-frequency features like facial color and fine-grained texture, and does so with lower complexity than Transformers. Motivated by these observations, we propose FADPNet, a Frequency-Aware Dual-Path Network that decomposes facial features into low- and high-frequency components and processes them via dedicated branches. For low-frequency regions, we introduce a Mamba-based Low-Frequency Enhancement Block (LFEB), which combines state-space attention with squeeze-and-excitation operations to extract low-frequency global interactions and emphasize informative channels. For high-frequency regions, we design a CNN-based Deep Position-Aware Attention (DPA) module to enhance spatially-dependent structural details, complemented by a lightweight High-Frequency Refinement (HFR) module that further refines frequency-specific representations. Through the above designs, our method achieves an excellent balance between FSR quality and model efficiency, outperforming existing approaches.
Abstract (translated)
在计算成本有限的情况下，人脸超分辨率（FSR）仍然是一个开放性问题。现有方法通常对所有面部像素一视同仁，导致计算资源分配不均衡和FSR性能下降。卷积神经网络（CNN）对于如面部轮廓和边界的高频特征较为敏感；同时，Mamba模型擅长捕捉肤色和细腻纹理等低频特征，并且其复杂度低于Transformer模型。基于这些观察结果，我们提出了频率感知双路径网络FADPNet。该网络将面部特征分解为低频和高频成分，并通过专用分支进行处理。 对于低频区域，我们引入了一个基于Mamba的低频增强块（LFEB），结合状态空间注意力与挤压激励操作以提取低频全局互动并强调信息通道。针对高频区域，设计了一种基于CNN的位置感知深度注意模块DPA来加强依赖于位置的结构细节，并辅以轻量级的高频细化（HFR）模块进一步优化频率特定表示。 通过上述设计，我们的方法在FSR质量和模型效率之间实现了优异的平衡，超越了现有方法。
URL
https://arxiv.org/abs/2506.14121
PDF
-
FOAM: A General Frequency-Optimized Anti-Overlapping Framework for Overlapping Object Perception
2025-06-16 13:58:49Mingyuan Li, Tong Jia, Han Gu, Hui Lu, Hao Wang, Bowen Ma, Shuyang Lin, Shiyi Guo, Shizhuo Deng, Dongyue ChenAbstract
Overlapping object perception aims to decouple the randomly overlapping foreground-background features, extracting foreground features while suppressing background features, which holds significant application value in fields such as security screening and medical auxiliary diagnosis. Despite some research efforts to tackle the challenge of overlapping object perception, most solutions are confined to the spatial domain. Through frequency domain analysis, we observe that the degradation of contours and textures due to the overlapping phenomenon can be intuitively reflected in the magnitude spectrum. Based on this observation, we propose a general Frequency-Optimized Anti-Overlapping Framework (FOAM) to assist the model in extracting more texture and contour information, thereby enhancing the ability for anti-overlapping object perception. Specifically, we design the Frequency Spatial Transformer Block (FSTB), which can simultaneously extract features from both the frequency and spatial domains, helping the network capture more texture features from the foreground. In addition, we introduce the Hierarchical De-Corrupting (HDC) mechanism, which aligns adjacent features in the separately constructed base branch and corruption branch using a specially designed consistent loss during the training phase. This mechanism suppresses the response to irrelevant background features of FSTBs, thereby improving the perception of foreground contour. We conduct extensive experiments to validate the effectiveness and generalization of the proposed FOAM, which further improves the accuracy of state-of-the-art models on four datasets, specifically for the three overlapping object perception tasks: Prohibited Item Detection, Prohibited Item Segmentation, and Pneumonia Detection. The code will be open source once the paper is accepted.
Abstract (translated)
重叠物体感知旨在解耦随机重叠的前景和背景特征，在提取前景特征的同时抑制背景特征，这一技术在安全检查和医学辅助诊断等领域具有重要的应用价值。尽管已有一些研究努力来应对重叠物体感知挑战，但大多数解决方案仍然局限于空间领域内。通过频域分析，我们观察到由于重叠现象导致边缘和纹理质量下降的情况可以直观地反映在幅度谱上。基于这一观察结果，我们提出了一种通用的频率优化抗重叠框架（FOAM），旨在帮助模型提取更多的纹理和轮廓信息，从而增强其对抗重叠物体感知的能力。 具体而言，我们设计了频域空间变换器模块（FSTB），该模块可以同时从频域和空间域中提取特征，有助于网络捕捉前景中的更多纹理特征。此外，我们引入了分层去混淆机制（HDC），在训练阶段通过特别设计的一致性损失将独立构建的基础分支和干扰分支的相邻特征对齐。这一机制抑制了FSTB对无关背景特征的响应，从而改善了前景轮廓的感知能力。 为了验证所提出的FOAM的有效性和泛化性能，我们进行了广泛的实验，并进一步提高了四个数据集上现有最先进模型在三种重叠物体感知任务——禁止物品检测、禁止物品分割和肺炎检测中的准确性。论文一旦被接受，代码将开源共享。
URL
https://arxiv.org/abs/2506.13501
PDF
-
Automatic Multi-View X-Ray/CT Registration Using Bone Substructure Contours
2025-06-16 09:33:37Roman Flepp, Leon Nissen, Bastian Sigrist, Arend Nieuwland, Nicola Cavalcanti, Philipp F\"urnstahl, Thomas Dreher, Lilian CalvetAbstract
Purpose: Accurate intraoperative X-ray/CT registration is essential for surgical navigation in orthopedic procedures. However, existing methods struggle with consistently achieving sub-millimeter accuracy, robustness under broad initial pose estimates or need manual key-point annotations. This work aims to address these challenges by proposing a novel multi-view X-ray/CT registration method for intraoperative bone registration. Methods: The proposed registration method consists of a multi-view, contour-based iterative closest point (ICP) optimization. Unlike previous methods, which attempt to match bone contours across the entire silhouette in both imaging modalities, we focus on matching specific subcategories of contours corresponding to bone substructures. This leads to reduced ambiguity in the ICP matches, resulting in a more robust and accurate registration solution. This approach requires only two X-ray images and operates fully automatically. Additionally, we contribute a dataset of 5 cadaveric specimens, including real X-ray images, X-ray image poses and the corresponding CT scans. Results: The proposed registration method is evaluated on real X-ray images using mean reprojection error (mRPD). The method consistently achieves sub-millimeter accuracy with a mRPD 0.67mm compared to 5.35mm by a commercial solution requiring manual intervention. Furthermore, the method offers improved practical applicability, being fully automatic. Conclusion: Our method offers a practical, accurate, and efficient solution for multi-view X-ray/CT registration in orthopedic surgeries, which can be easily combined with tracking systems. By improving registration accuracy and minimizing manual intervention, it enhances intraoperative navigation, contributing to more accurate and effective surgical outcomes in computer-assisted surgery (CAS).
Abstract (translated)
目的：在骨科手术中，术中X射线/CT配准对于手术导航至关重要。然而，现有方法难以持续实现亚毫米级精度、广泛初始姿态估计下的鲁棒性或需要手动关键点标注。本研究旨在通过提出一种新的多视角X射线/CT配准方法来解决这些问题，该方法适用于术中骨配准。 **方法：** 所提出的注册方法包括一个基于轮廓的多视图迭代最近点（ICP）优化过程。与以往试图在两种成像模式下在整个剪影范围内匹配骨骼轮廓的方法不同，我们专注于匹配特定子类别的对应于骨骼亚结构的轮廓。这减少了ICP匹配中的歧义，从而提高了配准解决方案的稳健性和准确性。这种方法仅需两张X射线图像，并且完全自动运行。 此外，我们贡献了一组包含5个解剖标本的真实数据集，包括X射线影像、X射线影像姿态和相应的CT扫描。 **结果：** 所提出的注册方法在使用均方根重投影误差（mRPD）评估真实X射线图像时表现出色。该方法持续实现亚毫米级精度，达到0.67mm的mRPD，而商业解决方案需手动干预且仅为5.35mm。此外，这种方法提供了更实用的应用性，因为它是完全自动化的。 **结论：** 我们的方法为多视角X射线/CT配准提供了一种实际、准确和有效的解决方案，在骨科手术中可以轻松与跟踪系统结合使用。通过提高注册精度并减少人工干预，该方法改善了术中的导航性能，有助于计算机辅助外科手术（CAS）实现更精确和有效的结果。
URL
https://arxiv.org/abs/2506.13292
PDF
-
Real-Time Per-Garment Virtual Try-On with Temporal Consistency for Loose-Fitting Garments
2025-06-14 04:57:21Zaiqiang Wu, I-Chao Shen, Takeo IgarashiAbstract
Per-garment virtual try-on methods collect garment-specific datasets and train networks tailored to each garment to achieve superior results. However, these approaches often struggle with loose-fitting garments due to two key limitations: (1) They rely on human body semantic maps to align garments with the body, but these maps become unreliable when body contours are obscured by loose-fitting garments, resulting in degraded outcomes; (2) They train garment synthesis networks on a per-frame basis without utilizing temporal information, leading to noticeable jittering artifacts. To address these challenges, we propose a two-stage approach for robust semantic map estimation. First, we extract a garment-invariant representation from the raw input image. This representation is then passed through an auxiliary network to estimate the semantic map. This enhances the robustness of semantic map estimation under loose-fitting garments during garment-specific dataset generation. Furthermore, we introduce a recurrent garment synthesis framework that incorporates temporal dependencies to improve frame-to-frame coherence while maintaining real-time performance. We conducted qualitative and quantitative evaluations to demonstrate that our method outperforms existing approaches in both image quality and temporal coherence. Ablation studies further validate the effectiveness of the garment-invariant representation and the recurrent synthesis framework.
Abstract (translated)
基于每件衣物的虚拟试穿方法会收集特定于某件衣物的数据集，并训练针对每种衣物定制的网络以获得更佳效果。然而，这些方法在处理宽松衣物时往往面临两个关键限制：（1）它们依赖于人体语义地图来将衣物与身体对齐，但当人体轮廓被宽松衣物遮挡时，这种地图变得不可靠，导致结果质量下降；（2）它们基于每帧训练服装合成网络，而不利用时间信息，这会导致明显的抖动伪影。为了解决这些挑战，我们提出了一种两阶段的方法来估计稳健的语义图。 首先，从原始输入图像中提取一种与衣物无关的表示。这种表示然后通过一个辅助网络来估算语义地图。这种方法增强了在生成特定于某件服装的数据集时，在宽松服饰下语义地图估算的鲁棒性。此外，我们引入了一种递归服装合成框架，该框架结合了时间依赖关系以改善帧与帧之间的连贯性，并保持实时性能。 我们进行了定性和定量评估来证明我们的方法在图像质量和时间一致性方面优于现有的方法。消融研究进一步验证了衣物无关表示和递归合成框架的有效性。
URL
https://arxiv.org/abs/2506.12348
PDF
-
Modality-AGnostic Image Cascade for Multi-Modality Cardiac Substructure Segmentation
2025-06-12 15:10:24Nicholas Summerfield, Qisheng He, Alex Kuo, Ahmed I. Ghanem, Simeng Zhu, Chase Ruff, Joshua Pan, Anudeep Kumar, Prashant Nagpal, Jiwei Zhao, Ming Dong, Carri K. Glide-HurstAbstract
Cardiac substructures are essential in thoracic radiation therapy planning to minimize risk of radiation-induced heart disease. Deep learning (DL) offers efficient methods to reduce contouring burden but lacks generalizability across different modalities and overlapping structures. This work introduces and validates a Modality-AGnostic Image Cascade (MAGIC) for comprehensive and multi-modal cardiac substructure segmentation. MAGIC is implemented through replicated encoding and decoding branches of an nnU-Net-based, U-shaped backbone conserving the function of a single model. Twenty cardiac substructures (heart, chambers, great vessels (GVs), valves, coronary arteries (CAs), and conduction nodes) from simulation CT (Sim-CT), low-field MR-Linac, and cardiac CT angiography (CCTA) modalities were manually delineated and used to train (n=76), validate (n=15), and test (n=30) MAGIC. Twelve comparison models (four segmentation subgroups across three modalities) were equivalently trained. All methods were compared for training efficiency and against reference contours using the Dice Similarity Coefficient (DSC) and two-tailed Wilcoxon Signed-Rank test (threshold, p<0.05). Average DSC scores were 0.75(0.16) for Sim-CT, 0.68(0.21) for MR-Linac, and 0.80(0.16) for CCTA. MAGIC outperforms the comparison in 57% of cases, with limited statistical differences. MAGIC offers an effective and accurate segmentation solution that is lightweight and capable of segmenting multiple modalities and overlapping structures in a single model. MAGIC further enables clinical implementation by simplifying the computational requirements and offering unparalleled flexibility for clinical settings.
Abstract (translated)
心脏亚结构在胸部放射治疗规划中至关重要，可降低由辐射引起的疾病风险。深度学习（DL）提供有效的方法来减轻轮廓绘制负担，但缺乏跨不同模态和重叠结构的泛化能力。本工作引入并验证了一种无模式感知图像级联（MAGIC），用于全面且多模态的心脏亚结构分割。MAGIC通过复制基于nnU-Net、呈U形架构模型的编码和解码分支来实现，该架构保留了单一模型的功能。 从模拟CT（Sim-CT）、低场MR-Linac和心脏CT血管造影（CCTA）模态中手动勾画出20个心脏亚结构（包括心脏、心室、大血管(GVs)、瓣膜、冠状动脉(CAs)以及传导节点），用于训练(n=76)，验证(n=15)和测试MAGIC(n=30)。同样，十二种对比模型（四种分割子组跨三种模态）被同等培训。所有方法均根据训练效率进行比较，并通过Dice相似系数(DSC)与双尾Wilcoxon符号秩检验（阈值p<0.05）参考轮廓进行比对。 Sim-CT、MR-Linac和CCTA的平均DSC分数分别为0.75(0.16)、0.68(0.21)和0.80(0.16)。在57%的情况下，MAGIC优于对比模型，但统计差异有限。 MAGIC提供了一种有效的且准确的分割解决方案，它是轻量级的，并能够在单一模型中对多种模态及重叠结构进行分割。此外，MAGIC通过简化计算需求和为临床环境提供无与伦比的灵活性而促进了其在临床上的应用。
URL
https://arxiv.org/abs/2506.10797
PDF
-
Rethinking Brain Tumor Segmentation from the Frequency Domain Perspective
2025-06-11 19:44:51Minye Shao, Zeyu Wang, Haoran Duan, Yawen Huang, Bing Zhai, Shizheng Wang, Yang Long, Yefeng ZhengAbstract
Precise segmentation of brain tumors, particularly contrast-enhancing regions visible in post-contrast MRI (areas highlighted by contrast agent injection), is crucial for accurate clinical diagnosis and treatment planning but remains challenging. However, current methods exhibit notable performance degradation in segmenting these enhancing brain tumor areas, largely due to insufficient consideration of MRI-specific tumor features such as complex textures and directional variations. To address this, we propose the Harmonized Frequency Fusion Network (HFF-Net), which rethinks brain tumor segmentation from a frequency-domain perspective. To comprehensively characterize tumor regions, we develop a Frequency Domain Decomposition (FDD) module that separates MRI images into low-frequency components, capturing smooth tumor contours and high-frequency components, highlighting detailed textures and directional edges. To further enhance sensitivity to tumor boundaries, we introduce an Adaptive Laplacian Convolution (ALC) module that adaptively emphasizes critical high-frequency details using dynamically updated convolution kernels. To effectively fuse tumor features across multiple scales, we design a Frequency Domain Cross-Attention (FDCA) integrating semantic, positional, and slice-specific information. We further validate and interpret frequency-domain improvements through visualization, theoretical reasoning, and experimental analyses. Extensive experiments on four public datasets demonstrate that HFF-Net achieves an average relative improvement of 4.48\% (ranging from 2.39\% to 7.72\%) in the mean Dice scores across the three major subregions, and an average relative improvement of 7.33% (ranging from 5.96% to 8.64%) in the segmentation of contrast-enhancing tumor regions, while maintaining favorable computational efficiency and clinical applicability. Code: this https URL.
Abstract (translated)
脑肿瘤的精确分割，尤其是通过注射对比剂后在磁共振成像（MRI）中可见的增强区域，对于临床诊断和治疗计划的准确性至关重要，但仍然具有挑战性。然而，当前的方法在分割这些增强型脑瘤区域时表现出显著性能下降，主要原因是未能充分考虑如复杂纹理和方向变化等特定于MRI的肿瘤特征。为了解决这个问题，我们提出了一种谐波频率融合网络（HFF-Net），该网络从频域角度重新思考了脑瘤分割的问题。 为了全面表征肿瘤区域，我们开发了一个频域分解（FDD）模块，将MRI图像分离成低频部分和高频部分。低频部分捕捉平滑的肿瘤轮廓，而高频部分则强调细节纹理和方向边缘。为进一步提高对肿瘤边界的敏感度，我们引入了自适应拉普拉斯卷积（ALC）模块，该模块通过动态更新的卷积核来增强关键的高频细节。 为了有效融合跨多个尺度的肿瘤特征，我们设计了一个频域交叉注意力（FDCA）机制，它整合了语义、位置和切片特定的信息。此外，我们还通过可视化分析、理论推理以及实验研究验证并解释了频率域改进的方法。 在四个公共数据集上的广泛实验证明，HFF-Net实现了平均相对4.48%的Dice分数提升（范围从2.39%到7.72%），主要表现在三大子区域中；同时，在对比增强肿瘤区域的分割上，其平均相对改进达到了7.33%（范围从5.96%至8.64%）。此外，HFF-Net在保持计算效率和临床应用性的同时展现了出色的性能。 代码链接：[此 URL](https://this-url.com)
URL
https://arxiv.org/abs/2506.10142
PDF
-
The Four Color Theorem for Cell Instance Segmentation
2025-06-11 13:33:58Ye Zhang, Yu Zhou, Yifeng Wang, Jun Xiao, Ziyue Wang, Yongbing Zhang, Jianxu ChenAbstract
Cell instance segmentation is critical to analyzing biomedical images, yet accurately distinguishing tightly touching cells remains a persistent challenge. Existing instance segmentation frameworks, including detection-based, contour-based, and distance mapping-based approaches, have made significant progress, but balancing model performance with computational efficiency remains an open problem. In this paper, we propose a novel cell instance segmentation method inspired by the four-color theorem. By conceptualizing cells as countries and tissues as oceans, we introduce a four-color encoding scheme that ensures adjacent instances receive distinct labels. This reformulation transforms instance segmentation into a constrained semantic segmentation problem with only four predicted classes, substantially simplifying the instance differentiation process. To solve the training instability caused by the non-uniqueness of four-color encoding, we design an asymptotic training strategy and encoding transformation method. Extensive experiments on various modes demonstrate our approach achieves state-of-the-art performance. The code is available at this https URL.
Abstract (translated)
细胞实例分割对于分析生物医学图像至关重要，但准确地区分紧密接触的细胞仍然是一个持久的挑战。现有的基于检测、轮廓和距离映射的方法虽然在实例分割框架方面取得了显著进展，但在保持模型性能的同时实现计算效率仍然是一项未解难题。在这篇论文中，我们提出了一种受四色定理启发的新颖细胞实例分割方法。通过将细胞视为国家并将组织视为海洋，我们引入了一个四色编码方案，确保相邻实例获得不同的标签。这种重新定义将实例分割问题转化为一个仅有四个预测类别的约束语义分割问题，从而大大简化了实例区分的过程。 为了应对由四色编码非唯一性导致的训练不稳定性问题，我们设计了一种渐近式训练策略和编码转换方法。在各种模式下的广泛实验表明，我们的方法达到了最先进的性能水平。代码可在提供的链接处获取（请将“this https URL”替换为实际链接）。
URL
https://arxiv.org/abs/2506.09724
PDF
-
How attention simplifies mental representations for planning
2025-06-11 08:46:05Jason da Silva Castanheira, Nicholas Shea, Stephen M. FlemingAbstract
Human planning is efficient -- it frugally deploys limited cognitive resources to accomplish difficult tasks -- and flexible -- adapting to novel problems and environments. Computational approaches suggest that people construct simplified mental representations of their environment, balancing the complexity of a task representation with its utility. These models imply a nested optimisation in which planning shapes perception, and perception shapes planning -- but the perceptual and attentional mechanisms governing how this interaction unfolds remain unknown. Here, we harness virtual maze navigation to characterise how spatial attention controls which aspects of a task representation enter subjective awareness and are available for planning. We find that spatial proximity governs which aspects of a maze are available for planning, and that when task-relevant information follows natural (lateralised) contours of attention, people can more easily construct simplified and useful maze representations. This influence of attention varies considerably across individuals, explaining differences in people's task representations and behaviour. Inspired by the 'spotlight of attention' analogy, we incorporate the effects of visuospatial attention into existing computational accounts of value-guided construal. Together, our work bridges computational perspectives on perception and decision-making to better understand how individuals represent their environments in aid of planning.
Abstract (translated)
人类的规划既高效又灵活。它以节俭的方式部署有限的认知资源，以完成困难的任务，并能适应新出现的问题和环境。计算方法表明，人们会构建自己环境的简化心理表征，在任务表示的复杂性和其实用性之间取得平衡。这些模型暗示了一个嵌套优化的过程：规划塑造感知，而感知又反过来影响规划——但控制这种互动展开的认知和注意力机制仍然未知。 在这里，我们利用虚拟迷宫导航来研究空间注意如何控制哪些任务方面的信息进入主观意识，并可用于规划。我们的发现表明，空间上的接近性决定了迷宫中哪些方面可以被用于规划，并且当与任务相关的数据遵循自然（偏侧化）的注意力轮廓时，人们更容易构建出简化和有用的迷宫表征。这种注意力的影响在个体间差异很大，解释了不同人在任务表示和行为方面的差异。 受“注意聚焦”类比的启发，我们将视觉空间注意的效果融入现有的价值导向建构计算模型中。我们这项工作将感知与决策制定的计算观点相结合，以更好地理解个人如何表征环境来帮助规划。
URL
https://arxiv.org/abs/2506.09520
PDF
-
A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning
2025-06-11 06:24:02Swadhin Das, Divyansh Mundra, Priyanshu Dayal, Raksha SharmaAbstract
Transformer-based models have achieved strong performance in remote sensing image captioning by capturing long-range dependencies and contextual information. However, their practical deployment is hindered by high computational costs, especially in multi-modal frameworks that employ separate transformer-based encoders and decoders. In addition, existing remote sensing image captioning models primarily focus on high-level semantic extraction while often overlooking fine-grained structural features such as edges, contours, and object boundaries. To address these challenges, a lightweight transformer architecture is proposed by reducing the dimensionality of the encoder layers and employing a distilled version of GPT-2 as the decoder. A knowledge distillation strategy is used to transfer knowledge from a more complex teacher model to improve the performance of the lightweight network. Furthermore, an edge-aware enhancement strategy is incorporated to enhance image representation and object boundary understanding, enabling the model to capture fine-grained spatial details in remote sensing images. Experimental results demonstrate that the proposed approach significantly improves caption quality compared to state-of-the-art methods.
Abstract (translated)
基于Transformer的模型通过捕捉长距离依赖和上下文信息，在遥感图像描述生成中取得了优异的成绩。然而，由于计算成本高，尤其是在使用独立的Transformer编码器和解码器的多模态框架中，这些模型的实际部署受到了阻碍。此外，现有的遥感图像描述生成模型主要关注高层次语义提取，而往往忽视了边缘、轮廓和对象边界等细粒度结构特征。 为了解决这些问题，提出了一种轻量级的Transformer架构，通过减少编码器层的维度并使用GPT-2的精简版本作为解码器来实现。采用知识蒸馏策略，将更复杂的教师模型的知识转移到轻量级网络中以提高性能。此外，还引入了边缘感知增强策略，用以提升图像表示和对象边界理解能力，使模型能够捕捉到遥感图像中的细粒度空间细节。 实验结果表明，所提出的方法在图像描述质量方面显著优于现有的先进方法。
URL
https://arxiv.org/abs/2506.09429
PDF
-
Grids Often Outperform Implicit Neural Representations
2025-06-10 23:52:09Namhoon Kim, Sara Fridovich-KeilAbstract
Implicit Neural Representations (INRs) have recently shown impressive results, but their fundamental capacity, implicit biases, and scaling behavior remain poorly understood. We investigate the performance of diverse INRs across a suite of 2D and 3D real and synthetic signals with varying effective bandwidth, as well as both overfitting and generalization tasks including tomography, super-resolution, and denoising. By stratifying performance according to model size as well as signal type and bandwidth, our results shed light on how different INR and grid representations allocate their capacity. We find that, for most tasks and signals, a simple regularized grid with interpolation trains faster and to higher quality than any INR with the same number of parameters. We also find limited settings where INRs outperform grids -- namely fitting signals with underlying lower-dimensional structure such as shape contours -- to guide future use of INRs towards the most advantageous applications. Code and synthetic signals used in our analysis are available at this https URL.
Abstract (translated)
最近，隐式神经表示（INR）展示了令人印象深刻的结果，但它们的基本容量、内在偏置和扩展行为仍然缺乏深入理解。我们研究了不同种类的INRs在一系列2D和3D真实及合成信号上的表现，这些信号的有效带宽各不相同，并且还涵盖了过拟合和泛化任务，如断层成像、超分辨率处理以及去噪等。 通过根据不同模型大小、信号类型和有效带宽对性能进行分类，我们的结果揭示了不同INR和网格表示如何分配其容量。我们发现，在大多数任务和信号上，一个简单的正则化网格（带有插值功能）训练速度更快，并且能达到更高的质量，与具有相同数量参数的任何INR相比都是如此。 此外，我们也找到了一些特定情况下INRs优于网格的情况——例如拟合具有潜在低维结构的信号，如形状轮廓。这些发现可以为未来的INR应用提供指导，使其应用于最有利的应用场景中。 用于我们分析的代码和合成信号可在以下网址获取：[此链接]（请将[此链接]替换为您实际提供的URL）。
URL
https://arxiv.org/abs/2506.11139
PDF
-
Adapting Vision-Language Foundation Model for Next Generation Medical Ultrasound Image Analysis
2025-06-10 14:37:51Jingguo Qu, Xinyang Han, Tonghuan Xiao, Jia Ai, Juan Wu, Tong Zhao, Jing Qin, Ann Dorothy King, Winnie Chiu-Wing Chu, Jing Cai, Michael Tin-Cheung Ying{\i}nstAbstract
Medical ultrasonography is an essential imaging technique for examining superficial organs and tissues, including lymph nodes, breast, and thyroid. It employs high-frequency ultrasound waves to generate detailed images of the internal structures of the human body. However, manually contouring regions of interest in these images is a labor-intensive task that demands expertise and often results in inconsistent interpretations among individuals. Vision-language foundation models, which have excelled in various computer vision applications, present new opportunities for enhancing ultrasound image analysis. Yet, their performance is hindered by the significant differences between natural and medical imaging domains. This research seeks to overcome these challenges by developing domain adaptation methods for vision-language foundation models. In this study, we explore the fine-tuning pipeline for vision-language foundation models by utilizing large language model as text refiner with special-designed adaptation strategies and task-driven heads. Our approach has been extensively evaluated on six ultrasound datasets and two tasks: segmentation and classification. The experimental results show that our method can effectively improve the performance of vision-language foundation models for ultrasound image analysis, and outperform the existing state-of-the-art vision-language and pure foundation models. The source code of this study is available at \href{this https URL}{GitHub}.
Abstract (translated)
医学超声成像是检查浅表器官和组织（包括淋巴结、乳腺和甲状腺）的重要影像技术。它利用高频超声波生成人体内部结构的详细图像。然而，手动勾画这些图像中感兴趣区域轮廓是一项劳动密集型任务，需要专业知识，并且通常会导致个体间解释的一致性问题。视觉-语言基础模型在各种计算机视觉应用中的表现优异，为提升超声影像分析提供了新的机会。然而，由于自然和医学成像领域之间的显著差异，其性能受到了限制。 本研究旨在通过开发面向视觉-语言基础模型的领域适应方法来克服这些挑战。在这项研究中，我们探索了使用大规模语言模型作为文本精炼器并结合特殊设计的适应策略和任务驱动头部的微调流程。我们的方法已经在六种超声数据集以及分割和分类两项任务上进行了广泛的评估。实验结果表明，本方法能够有效提升视觉-语言基础模型在超声图像分析中的性能，并优于现有的最先进的视觉-语言及纯基础模型。 这项研究的源代码可在[GitHub](https://this%20URL)获取。
URL
https://arxiv.org/abs/2506.08849
PDF
-
Image-Based Method For Measuring And Classification Of Iron Ore Pellets Using Star-Convex Polygons
2025-06-10 11:30:46Artem Solomko, Oleg Kartashev, Andrey Golov, Mikhail Deulin, Vadim Valynkin, Vasily KharinAbstract
We would like to present a comprehensive study on the classification of iron ore pellets, aimed at identifying quality violations in the final product, alongside the development of an innovative imagebased measurement method utilizing the StarDist algorithm, which is primarily employed in the medical field. This initiative is motivated by the necessity to accurately identify and analyze objects within densely packed and unstable environments. The process involves segmenting these objects, determining their contours, classifying them, and measuring their physical dimensions. This is crucial because the size distribution and classification of pellets such as distinguishing between nice (quality) and joint (caused by the presence of moisture or indicating a process of production failure) types are among the most significant characteristics that define the quality of the final product. Traditional algorithms, including image classification techniques using Vision Transformer (ViT), instance segmentation methods like Mask R-CNN, and various anomaly segmentation algorithms, have not yielded satisfactory results in this context. Consequently, we explored methodologies from related fields to enhance our approach. The outcome of our research is a novel method designed to detect objects with smoothed boundaries. This advancement significantly improves the accuracy of physical dimension measurements and facilitates a more precise analysis of size distribution among the iron ore pellets. By leveraging the strengths of the StarDist algorithm, we aim to provide a robust solution that addresses the challenges posed by the complex nature of pellet classification and measurement.
Abstract (translated)
我们希望介绍一项关于铁矿石球团分类的全面研究，旨在识别最终产品中的质量违规情况，并开发一种创新的基于图像测量方法，该方法利用StarDist算法。尽管StarDist算法主要应用于医学领域，但我们希望通过这项研究解决在密集且不稳定环境中准确识别和分析对象的需求。这一过程包括分割这些对象、确定其轮廓、对其进行分类以及测量它们的物理尺寸。这非常重要，因为球团如区分优质（质量良好）与结块（由水分引起或表明生产过程中存在问题）类型时的大小分布和分类是定义最终产品质量的关键特征之一。 传统的算法，例如使用Vision Transformer (ViT)进行图像分类的技术、实例分割方法Mask R-CNN以及各种异常分割算法，在这种情况下未能提供令人满意的解决方案。因此，我们探索了相关领域的方法以改进我们的方法。研究结果是一种新型对象检测技术，该技术可以识别具有平滑边界的物体。这项进展显著提高了物理尺寸测量的准确性，并促进了铁矿石球团大小分布的更精确分析。 通过利用StarDist算法的优势，我们旨在提供一个强大的解决方案，解决由复杂性质导致的球团分类和测量挑战。
URL
https://arxiv.org/abs/2506.11126
PDF
-
Locating Tennis Ball Impact on the Racket in Real Time Using an Event Camera
2025-06-10 01:29:32Yuto Kase, Kai Ishibe, Ryoma Yasuda, Yudai Washida, Sakiko HashimotoAbstract
In racket sports, such as tennis, locating the ball's position at impact is important in clarifying player and equipment characteristics, thereby aiding in personalized equipment design. High-speed cameras are used to measure the impact location; however, their excessive memory consumption limits prolonged scene capture, and manual digitization for position detection is time-consuming and prone to human error. These limitations make it difficult to effectively capture the entire playing scene, hindering the ability to analyze the player's performance. We propose a method for locating the tennis ball impact on the racket in real time using an event camera. Event cameras efficiently measure brightness changes (called `events') with microsecond accuracy under high-speed motion while using lower memory consumption. These cameras enable users to continuously monitor their performance over extended periods. Our method consists of three identification steps: time range of swing, timing at impact, and contours of ball and racket. Conventional computer vision techniques are utilized along with an original event-based processing to detect the timing at impact (PATS: the amount of polarity asymmetry in time symmetry). The results of the experiments were within the permissible range for measuring tennis players' performance. Moreover, the computation time was sufficiently short for real-time applications.
Abstract (translated)
在网球等球拍运动中，确定击球瞬间球的位置对于明确球员和器材特性非常重要，并有助于个性化设备设计。高速相机常用于测量撞击位置；然而，由于其过高的内存消耗限制了长时间场景捕捉的能力，且手动数字化以检测位置耗时长并且容易出现人为错误。这些局限性使得有效捕获整个比赛场面变得困难，从而阻碍了分析球员表现的能力。 我们提出了一种使用事件相机实时定位网球与球拍撞击点的方法。事件相机能够在高速运动中准确地以微秒级精度测量亮度变化（称为“事件”），并且内存消耗较低，使用户能够长时间连续监测其表现。我们的方法包含三个识别步骤：挥拍时间范围、击球时刻以及球和球拍轮廓的确定。 我们采用传统的计算机视觉技术，并结合一种原始的时间对称性中的极性不对称量测法（PATS）来检测击球时间。实验结果显示，测量结果在允许范围内可以准确衡量网球运动员的表现。此外，计算时间对于实时应用来说足够短。
URL
https://arxiv.org/abs/2506.08327
PDF
