Contour
-
ViEEG: Hierarchical Neural Coding with Cross-Modal Progressive Enhancement for EEG-Based Visual Decoding
2025-05-18 13:19:08Minxu Liu, Donghai Guan, Chuhang Zheng, Chunwei Tian, Jie Wen, Qi ZhuAbstract
Understanding and decoding brain activity into visual representations is a fundamental challenge at the intersection of neuroscience and artificial intelligence. While EEG-based visual decoding has shown promise due to its non-invasive, low-cost nature and millisecond-level temporal resolution, existing methods are limited by their reliance on flat neural representations that overlook the brain's inherent visual hierarchy. In this paper, we introduce ViEEG, a biologically inspired hierarchical EEG decoding framework that aligns with the Hubel-Wiesel theory of visual processing. ViEEG decomposes each visual stimulus into three biologically aligned components-contour, foreground object, and contextual scene-serving as anchors for a three-stream EEG encoder. These EEG features are progressively integrated via cross-attention routing, simulating cortical information flow from V1 to IT to the association cortex. We further adopt hierarchical contrastive learning to align EEG representations with CLIP embeddings, enabling zero-shot object recognition. Extensive experiments on the THINGS-EEG dataset demonstrate that ViEEG achieves state-of-the-art performance, with 40.9% Top-1 accuracy in subject-dependent and 22.9% Top-1 accuracy in cross-subject settings, surpassing existing methods by over 45%. Our framework not only advances the performance frontier but also sets a new paradigm for biologically grounded brain decoding in AI.
Abstract (translated)
理解并解码脑活动以生成视觉表示是神经科学与人工智能交叉领域中的一个基本挑战。尽管基于EEG的视觉解码由于其非侵入性、低成本以及毫秒级的时间分辨率而展现出巨大潜力，现有的方法却受到依赖于忽视大脑固有视觉层次结构的扁平化神经表示的限制。在本文中，我们提出了ViEEG——一种受生物启发的分层EEG解码框架，该框架与Hubel-Wiesel关于视觉处理理论相一致。ViEEG将每个视觉刺激分解为三个与生物学相符的部分：轮廓、前景对象和背景场景，作为三流EEG编码器的基础。这些EEG特征通过交叉注意力路由进行逐步融合，模拟从初级视皮层（V1）到颞叶内侧区域（IT）再到关联皮层的信息流动过程。我们进一步采用了层次对比学习方法来使EEG表示与CLIP嵌入对齐，从而实现零样本对象识别功能。在THINGS-EEG数据集上的大量实验表明，ViEEG取得了最先进的性能，在依赖于单个受试者的情况下实现了40.9%的Top-1准确率，并且在跨受试者设置中达到了22.9%的Top-1准确率，这比现有方法高出超过45%。我们的框架不仅提升了性能边界，还为人工智能领域中的生物基础脑解码设定了新的范式。
URL
https://arxiv.org/abs/2505.12408
PDF
-
What's Inside Your Diffusion Model? A Score-Based Riemannian Metric to Explore the Data Manifold
Abstract
Recent advances in diffusion models have demonstrated their remarkable ability to capture complex image distributions, but the geometric properties of the learned data manifold remain poorly understood. We address this gap by introducing a score-based Riemannian metric that leverages the Stein score function from diffusion models to characterize the intrinsic geometry of the data manifold without requiring explicit parameterization. Our approach defines a metric tensor in the ambient space that stretches distances perpendicular to the manifold while preserving them along tangential directions, effectively creating a geometry where geodesics naturally follow the manifold's contours. We develop efficient algorithms for computing these geodesics and demonstrate their utility for both interpolation between data points and extrapolation beyond the observed data distribution. Through experiments on synthetic data with known geometry, Rotated MNIST, and complex natural images via Stable Diffusion, we show that our score-based geodesics capture meaningful transformations that respect the underlying data distribution. Our method consistently outperforms baseline approaches on perceptual metrics (LPIPS) and distribution-level metrics (FID, KID), producing smoother, more realistic image transitions. These results reveal the implicit geometric structure learned by diffusion models and provide a principled way to navigate the manifold of natural images through the lens of Riemannian geometry.
Abstract (translated)
最近在扩散模型方面的进展已经展示了它们捕捉复杂图像分布的非凡能力，但学习到的数据流形的几何属性仍然鲜为人知。为了解决这一问题，我们引入了一种基于评分函数的黎曼度量方法，该方法利用扩散模型中的Stein评分函数来表征数据流形的内在几何性质，并且不需要显式的参数化方式。 我们的方法在包围空间中定义了一个度量张量，在这个度量张量中，沿着与流形垂直的方向拉伸距离，而在沿切线方向则保持不变。这有效地创建了一种几何结构，在这种结构下测地线自然地遵循流形的轮廓。我们开发了用于计算这些测地线的有效算法，并展示了它们在数据点之间插值以及超出观察到的数据分布外推中的实用性。 通过使用具有已知几何特征的人工数据、旋转后的MNIST数据集和通过Stable Diffusion生成的复杂自然图像进行实验，我们表明我们的基于评分函数的测地线捕捉到了有意义的变化过程，这些变化过程尊重底层数据分布。在感知度量（LPIPS）和分布级度量（FID, KID）上，我们的方法始终优于基线方法，生成了更平滑、更逼真的图像转换。 这些结果揭示了扩散模型所学到的隐含几何结构，并通过黎曼几何提供了一种导航自然图象流形的合理途径。
URL
https://arxiv.org/abs/2505.11128
PDF
-
Unifying Segment Anything in Microscopy with Multimodal Large Language Model
2025-05-16 00:55:56Manyu Li, Ruian He, Zixian Zhang, Weimin Tan, Bo YanAbstract
Accurate segmentation of regions of interest in biomedical images holds substantial value in image analysis. Although several foundation models for biomedical segmentation have currently achieved excellent performance on certain datasets, they typically demonstrate sub-optimal performance on unseen domain data. We owe the deficiency to lack of vision-language knowledge before segmentation. Multimodal Large Language Models (MLLMs) bring outstanding understanding and reasoning capabilities to multimodal tasks, which inspires us to leverage MLLMs to inject Vision-Language Knowledge (VLK), thereby enabling vision models to demonstrate superior generalization capabilities on cross-domain datasets. In this paper, we propose using MLLMs to guide SAM in learning microscopy crose-domain data, unifying Segment Anything in Microscopy, named uLLSAM. Specifically, we propose the Vision-Language Semantic Alignment (VLSA) module, which injects VLK into Segment Anything Model (SAM). We find that after SAM receives global VLK prompts, its performance improves significantly, but there are deficiencies in boundary contour perception. Therefore, we further propose Semantic Boundary Regularization (SBR) to prompt SAM. Our method achieves performance improvements of 7.71% in Dice and 12.10% in SA across 9 in-domain microscopy datasets, achieving state-of-the-art performance. Our method also demonstrates improvements of 6.79% in Dice and 10.08% in SA across 10 out-ofdomain datasets, exhibiting strong generalization capabilities. Code is available at this https URL.
Abstract (translated)
在生物医学图像中精确分割感兴趣区域对图像分析具有重要价值。尽管目前有一些用于生物医学分割的基础模型已经在某些数据集上取得了优秀的性能，但它们通常在未见过的数据域（unseen domain data）上的表现较差。我们认为这一不足主要是由于缺乏视觉-语言知识（Vision-Language Knowledge, VLK）。多模态大型语言模型（Multimodal Large Language Models, MLLMs）在多模态任务中带来了出色的理解和推理能力，这激发了我们利用MLLMs注入VLK的想法，从而使视觉模型能够在跨域数据集上表现出色的泛化能力。在这篇论文中，我们提出了一种方法，即使用MLLMs来指导SAM（Segment Anything Model）学习显微图像跨域数据，命名为uLLSAM（unifying Segment Anything in Microscopy）。具体来说，我们提出了视觉-语言语义对齐（Vision-Language Semantic Alignment, VLSA）模块，该模块将VLK注入到SAM中。我们发现，在SAM接收到全局VLK提示后，其性能显著提高，但存在边界轮廓感知方面的不足。因此，我们进一步提出了一种语义边界正则化（Semantic Boundary Regularization, SBR），用于引导SAM。我们的方法在9个内域显微图像数据集上实现了Dice指标提高了7.71%，SA（可能指的是Segment Anything模型的评价标准之一）指标提高了12.10%的成绩，达到了最先进的性能水平。此外，在10个外域数据集中，我们的方法分别使Dice和SA指标提升了6.79%和10.08%，展示了强大的泛化能力。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2505.10769
PDF
-
CSPENet: Contour-Aware and Saliency Priors Embedding Network for Infrared Small Target Detection
2025-05-15 03:56:36Jiakun Deng, Kexuan Li, Xingye Cui, Jiaxuan Li, Chang Long, Tian Pu, Zhenming PengAbstract
Infrared small target detection (ISTD) plays a critical role in a wide range of civilian and military applications. Existing methods suffer from deficiencies in the localization of dim targets and the perception of contour information under dense clutter environments, severely limiting their detection performance. To tackle these issues, we propose a contour-aware and saliency priors embedding network (CSPENet) for ISTD. We first design a surround-convergent prior extraction module (SCPEM) that effectively captures the intrinsic characteristic of target contour pixel gradients converging toward their center. This module concurrently extracts two collaborative priors: a boosted saliency prior for accurate target localization and multi-scale structural priors for comprehensively enriching contour detail representation. Building upon this, we propose a dual-branch priors embedding architecture (DBPEA) that establishes differentiated feature fusion pathways, embedding these two priors at optimal network positions to achieve performance enhancement. Finally, we develop an attention-guided feature enhancement module (AGFEM) to refine feature representations and improve saliency estimation accuracy. Experimental results on public datasets NUDT-SIRST, IRSTD-1k, and NUAA-SIRST demonstrate that our CSPENet outperforms other state-of-the-art methods in detection performance. The code is available at this https URL.
Abstract (translated)
红外小目标检测（ISTD）在众多民用和军事应用中扮演着关键角色。现有的方法在定位暗淡目标以及在密集杂波环境中感知轮廓信息方面存在缺陷，这严重限制了它们的检测性能。为了解决这些问题，我们提出了一种基于轮廓感知和显著性先验嵌入的网络（CSPENet）用于ISTD。首先，我们设计了一个环绕汇聚先验提取模块（SCPEM），该模块能够有效捕捉目标轮廓像素梯度向中心汇聚的内在特性。此模块同时提取两种协同工作的先验：一种是增强型显著性先验，有助于精确的目标定位；另一种是多尺度结构化先验，用于全面丰富轮廓细节表示。 在此基础上，我们提出了一种双支路先验嵌入架构（DBPEA），建立了差异化特征融合路径，在最佳网络位置嵌入这两种先验以实现性能提升。最后，我们开发了一个基于注意力引导的特征增强模块（AGFEM）来优化特征表示并提高显著性估计的准确性。 在公开数据集NUDT-SIRST、IRSTD-1k和NUAA-SIRST上的实验结果表明，我们的CSPENet在检测性能上优于其他最先进的方法。代码可在[此处](https://example.com)获取（请将URL更改为实际链接）。
URL
https://arxiv.org/abs/2505.09943
PDF
-
OXSeg: Multidimensional attention UNet-based lip segmentation using semi-supervised lip contours
2025-05-08 14:06:20Hanie Moghaddasi, Christina Chambers, Sarah N. Mattson, Jeffrey R. Wozniak, Claire D. Coles, Raja Mukherjee, Michael SuttieAbstract
Lip segmentation plays a crucial role in various domains, such as lip synchronization, lipreading, and diagnostics. However, the effectiveness of supervised lip segmentation is constrained by the availability of lip contour in the training phase. A further challenge with lip segmentation is its reliance on image quality , lighting, and skin tone, leading to inaccuracies in the detected boundaries. To address these challenges, we propose a sequential lip segmentation method that integrates attention UNet and multidimensional input. We unravel the micro-patterns in facial images using local binary patterns to build multidimensional inputs. Subsequently, the multidimensional inputs are fed into sequential attention UNets, where the lip contour is reconstructed. We introduce a mask generation method that uses a few anatomical landmarks and estimates the complete lip contour to improve segmentation accuracy. This mask has been utilized in the training phase for lip segmentation. To evaluate the proposed method, we use facial images to segment the upper lips and subsequently assess lip-related facial anomalies in subjects with fetal alcohol syndrome (FAS). Using the proposed lip segmentation method, we achieved a mean dice score of 84.75%, and a mean pixel accuracy of 99.77% in upper lip segmentation. To further evaluate the method, we implemented classifiers to identify those with FAS. Using a generative adversarial network (GAN), we reached an accuracy of 98.55% in identifying FAS in one of the study populations. This method could be used to improve lip segmentation accuracy, especially around Cupid's bow, and shed light on distinct lip-related characteristics of FAS.
Abstract (translated)
嘴唇分割在唇同步、唇读和诊断等多个领域中扮演着重要角色。然而，监督式嘴唇分割的有效性受限于训练阶段中可用的嘴唇轮廓。此外，嘴唇分割还依赖于图像质量、光照条件以及肤色等因素，导致检测边界不准确的问题。为了解决这些挑战，我们提出了一种结合注意力UNet和多维输入的序列化嘴唇分割方法。 我们的方法首先利用局部二值模式技术揭示面部图像中的微小图案特征，并以此构建多维输入数据集。然后将这些多维输入数据送入序列化的注意力UNet中，重建嘴唇轮廓。我们还引入了一种生成掩码的方法，该方法使用少量解剖学标志点估计完整的嘴唇轮廓，以提高分割精度。此掩码在训练阶段被用于优化嘴唇的精确分割。 为了评估所提出的分割方法的有效性，我们在一系列面部图像上对其进行了测试，特别关注对胎儿酒精综合症（FAS）患者进行上唇分割，并进一步分析了与FAS相关的面部异常特征。通过使用我们的方法，在上唇分割任务中我们取得了平均Dice分数为84.75%，以及平均像素准确率为99.77%的成绩。 为了更全面地评估该方法，我们在一个包含FAS患者的数据集上实现了分类器，使用生成对抗网络（GAN）对数据进行了训练。结果表明，在区分FAS患者和对照组时，我们的模型达到了98.55%的准确率。 这一方法不仅能够提升嘴唇分割特别是围绕人中部分区域的精度，而且还能帮助识别与胎儿酒精综合症相关的独特唇部特征，从而为相关研究提供新的视角和技术手段。
URL
https://arxiv.org/abs/2505.05531
PDF
-
An Active Contour Model for Silhouette Vectorization using B'ezier Curves
Abstract
In this paper, we propose an active contour model for silhouette vectorization using cubic Bézier curves. Among the end points of the Bézier curves, we distinguish between corner and regular points where the orientation of the tangent vector is prescribed. By minimizing the distance of the Bézier curves to the silhouette boundary, the active contour model optimizes the location of the Bézier curves end points, the orientation of the tangent vectors in the regular points, and the estimation of the Bézier curve parameters. This active contour model can use the silhouette vectorization obtained by any method as an initial guess. The proposed method significantly reduces the average distance between the silhouette boundary and its vectorization obtained by the world-class graphic software Inkscape, Adobe Illustrator, and a curvature-based vectorization method, which we introduce for comparison. Our method also allows us to impose additional regularity on the Bézier curves by reducing their lengths.
Abstract (translated)
在这篇论文中，我们提出了一种使用三次Bézier曲线进行轮廓向矢量图转化的主动轮廓模型。在Bézier曲线的端点之间，我们将它们区分为角点和普通点，在这些普通点上切线向量的方向被规定。通过最小化Bézier曲线与轮廓边界之间的距离，该主动轮廓模型优化了Bézier曲线端点的位置、普通点处切线向量的方向以及对Bézier曲线参数的估计。这种主动轮廓模型可以使用任何方法获得的初始轮廓矢量化结果作为起始猜测。 我们提出的方法显著减少了轮廓边界与其通过世界一流的图形软件（如Inkscape和Adobe Illustrator）以及基于曲率的矢量化方法得到的矢量图之间的平均距离，后者用于比较参考。我们的方法还允许通过对缩短Bézier曲线长度来施加额外的规则性。
URL
https://arxiv.org/abs/2505.05132
PDF
-
Very High-Resolution Forest Mapping with TanDEM-X InSAR Data and Self-Supervised Learning
2025-05-06 08:54:28Jos\'e-Luis Bueso-Bello, Benjamin Chauvel, Daniel Carcereri, Philipp Posovszky, Pietro Milillo, Jennifer Ruiz, Juan-Carlos Fern\'andez-Diaz, Carolina Gonz\'alez, Michele Martone, Ronny H\"ansch, Paola RizzoliAbstract
Deep learning models have shown encouraging capabilities for mapping accurately forests at medium resolution with TanDEM-X interferometric SAR data. Such models, as most of current state-of-the-art deep learning techniques in remote sensing, are trained in a fully-supervised way, which requires a large amount of labeled data for training and validation. In this work, our aim is to exploit the high-resolution capabilities of the TanDEM-X mission to map forests at 6 m. The goal is to overcome the intrinsic limitations posed by midresolution products, which affect, e.g., the detection of narrow roads within vegetated areas and the precise delineation of forested regions contours. To cope with the lack of extended reliable reference datasets at such a high resolution, we investigate self-supervised learning techniques for extracting highly informative representations from the input features, followed by a supervised training step with a significantly smaller number of reliable labels. A 1 m resolution forest/non-forest reference map over Pennsylvania, USA, allows for comparing different training approaches for the development of an effective forest mapping framework with limited labeled samples. We select the best-performing approach over this test region and apply it in a real-case forest mapping scenario over the Amazon rainforest, where only very few labeled data at high resolution are available. In this challenging scenario, the proposed self-supervised framework significantly enhances the classification accuracy with respect to fully-supervised methods, trained using the same amount of labeled data, representing an extremely promising starting point for large-scale, very high-resolution forest mapping with TanDEM-X data.
Abstract (translated)
深度学习模型在利用TanDEM-X干涉SAR数据进行中分辨率森林地图绘制方面展示了令人鼓舞的能力。这类模型，以及大多数当前遥感领域的先进深度学习技术，都是通过完全监督的方式训练的，这需要大量标记的数据用于训练和验证。在这项工作中，我们的目标是利用TanDEM-X任务的高分辨率能力来在6米尺度上进行森林制图。该目标旨在克服中等分辨率产品内在限制的影响，例如，在植被区域检测狭窄道路以及精确界定森林地区轮廓的问题。 为了应对这种高分辨率下可靠参考数据集缺乏的情况，我们研究了自监督学习技术以从输入特征中提取高度相关信息，并随后使用少量可靠的标签进行监督训练。美国宾夕法尼亚州1米分辨率的森林/非森林参考图允许我们比较不同训练方法，以便开发一个有效的森林制图框架，在这种情况下标注样本数量有限。 在测试区域中选择了最佳性能的方法后，我们在亚马逊雨林的真实案例森林制图场景中应用了该方法。在这个具有挑战性的场景中，仅存在很少的高分辨率标签数据，提出的自监督框架显著提高了分类准确率，相对于完全监督的方法而言，在使用相同量级标记数据的情况下效果更佳。这为利用TanDEM-X数据进行大规模、极高分辨率森林制图提供了一个极具前景的起点。
URL
https://arxiv.org/abs/2505.03327
PDF
-
Interactive Instance Annotation with Siamese Networks
2025-05-06 05:04:56Xiang Xu, Ruotong Li, Mengjun Yi, Baile XU, Furao Shen, Jian ZhaoAbstract
Annotating instance masks is time-consuming and labor-intensive. A promising solution is to predict contours using a deep learning model and then allow users to refine them. However, most existing methods focus on in-domain scenarios, limiting their effectiveness for cross-domain annotation tasks. In this paper, we propose SiamAnno, a framework inspired by the use of Siamese networks in object tracking. SiamAnno leverages one-shot learning to annotate previously unseen objects by taking a bounding box as input and predicting object boundaries, which can then be adjusted by annotators. Trained on one dataset and tested on another without fine-tuning, SiamAnno achieves state-of-the-art (SOTA) performance across multiple datasets, demonstrating its ability to handle domain and environment shifts in cross-domain tasks. We also provide more comprehensive results compared to previous work, establishing a strong baseline for future research. To our knowledge, SiamAnno is the first model to explore Siamese architecture for instance annotation.
Abstract (translated)
标注实例掩模是一项耗时且费力的工作。一个有前景的解决方案是使用深度学习模型预测轮廓，然后允许用户进行调整。然而，大多数现有的方法主要关注于领域内的场景，这限制了它们在跨域标注任务中的有效性。在这篇论文中，我们提出了SiamAnno框架，该框架借鉴了Siamese网络在物体跟踪中的应用。SiamAnno利用一次性学习的方法，通过输入一个边界框来预测未见过的对象边界，并允许注释者进行调整。经过一个数据集的训练并在另一个未经微调的数据集中测试后，SiamAnno在多个数据集中取得了最先进的（SOTA）性能，展示了其处理跨域任务中领域和环境变化的能力。我们还提供了比先前工作更全面的结果，为未来的研究奠定了坚实的基础。据我们所知，SiamAnno是第一个探索Siamese架构用于实例标注的模型。
URL
https://arxiv.org/abs/2505.03184
PDF
-
Path and Bone-Contour Regularized Unpaired MRI-to-CT Translation
2025-05-06 02:08:35Teng Zhou, Jax Luo, Yuping Sun, Yiheng Tan, Shun Yao, Nazim Haouchine, Scott RaymondAbstract
Accurate MRI-to-CT translation promises the integration of complementary imaging information without the need for additional imaging sessions. Given the practical challenges associated with acquiring paired MRI and CT scans, the development of robust methods capable of leveraging unpaired datasets is essential for advancing the MRI-to-CT translation. Current unpaired MRI-to-CT translation methods, which predominantly rely on cycle consistency and contrastive learning frameworks, frequently encounter challenges in accurately translating anatomical features that are highly discernible on CT but less distinguishable on MRI, such as bone structures. This limitation renders these approaches less suitable for applications in radiation therapy, where precise bone representation is essential for accurate treatment planning. To address this challenge, we propose a path- and bone-contour regularized approach for unpaired MRI-to-CT translation. In our method, MRI and CT images are projected to a shared latent space, where the MRI-to-CT mapping is modeled as a continuous flow governed by neural ordinary differential equations. The optimal mapping is obtained by minimizing the transition path length of the flow. To enhance the accuracy of translated bone structures, we introduce a trainable neural network to generate bone contours from MRI and implement mechanisms to directly and indirectly encourage the model to focus on bone contours and their adjacent regions. Evaluations conducted on three datasets demonstrate that our method outperforms existing unpaired MRI-to-CT translation approaches, achieving lower overall error rates. Moreover, in a downstream bone segmentation task, our approach exhibits superior performance in preserving the fidelity of bone structures. Our code is available at: this https URL.
Abstract (translated)
准确的MRI到CT图像转换有望整合互补成像信息，而无需进行额外的成像会话。鉴于获取配对的MRI和CT扫描在实践中所面临的挑战，开发能够利用未配对数据集的强大方法对于推进MRI至CT的转换至关重要。目前基于未配对数据的方法主要依赖于循环一致性及对比学习框架，在将MRI图像准确翻译为含有CT图像中明显可见但在MRI上不太可区分的解剖特征（如骨骼结构）时，经常遇到困难。这一限制使得这些方法在放射治疗应用中的适用性降低，因为在该领域精确地表示骨骼对于制定准确的治疗计划至关重要。 为了应对这一挑战，我们提出了一种针对未配对MRI到CT转换的路径和骨轮廓正则化的方法。在此方法中，我们将MRI和CT图像映射到一个共享的潜在空间，在此空间内，从MRI到CT的映射被视为由神经常微分方程所调控的连续流。通过最小化该流动过程中的过渡路径长度来获得最佳映射。为了提高翻译后的骨骼结构的准确性，我们引入了一种可训练的神经网络以生成MRI上的骨轮廓，并实施机制直接和间接地鼓励模型专注于骨轮廓及其附近区域。 在三个数据集上进行的评估表明，我们的方法超越了现有的未配对MRI至CT转换方案，在总体错误率方面表现更佳。此外，在下游骨骼分割任务中，我们的方法展示了保持骨骼结构保真度方面的卓越性能。我们的代码可在此链接获取：this https URL.
URL
https://arxiv.org/abs/2505.03114
PDF
-
Finger Pose Estimation for Under-screen Fingerprint Sensor
2025-05-05 09:05:47Xiongjun Guan, Zhiyu Pan, Jianjiang Feng, Jie ZhouAbstract
Two-dimensional pose estimation plays a crucial role in fingerprint recognition by facilitating global alignment and reduce pose-induced variations. However, existing methods are still unsatisfactory when handling with large angle or small area inputs. These limitations are particularly pronounced on fingerprints captured by under-screen fingerprint sensors in smartphones. In this paper, we present a novel dual-modal input based network for under-screen fingerprint pose estimation. Our approach effectively integrates two distinct yet complementary modalities: texture details extracted from ridge patches through the under-screen fingerprint sensor, and rough contours derived from capacitive images obtained via the touch screen. This collaborative integration endows our network with more comprehensive and discriminative information, substantially improving the accuracy and stability of pose estimation. A decoupled probability distribution prediction task is designed, instead of the traditional supervised forms of numerical regression or heatmap voting, to facilitate the training process. Additionally, we incorporate a Mixture of Experts (MoE) based feature fusion mechanism and a relationship driven cross-domain knowledge transfer strategy to further strengthen feature extraction and fusion capabilities. Extensive experiments are conducted on several public datasets and two private datasets. The results indicate that our method is significantly superior to previous state-of-the-art (SOTA) methods and remarkably boosts the recognition ability of fingerprint recognition algorithms. Our code is available at this https URL.
Abstract (translated)
二维姿态估计在指纹识别中扮演着重要角色，通过促进全局对齐并减少姿态引起的变异来提高系统的性能。然而，现有的方法在处理大角度或小区域输入时仍不尽如人意。这些限制尤其体现在智能手机下屏指纹传感器捕获的指纹图像上。 本文提出了一种基于双模态输入的新网络架构，专门用于手机屏幕下方指纹的姿态估计。我们的方法有效地融合了两种互补的信息模式：一种是通过下屏指纹传感器提取的纹理细节（即从脊纹区域获取），另一种是从触控屏幕获得的电容图像中得出的大致轮廓。这种协作式的集成使我们的网络能够利用更全面和具有区分性的信息，从而显著提高姿态估计的准确性和稳定性。 我们设计了一个解耦的概率分布预测任务，而非传统的数值回归或热图投票方法，以促进训练过程。此外，还引入了一种基于专家混合（MoE）的特征融合机制以及一种驱动关系的跨域知识转移策略，进一步增强了特征提取和融合的能力。 我们在多个公开数据集和两个私有数据集上进行了广泛的实验。结果显示，我们的方法显著优于现有的最先进方法，并大幅提升了指纹识别算法的识别能力。 代码可在以下网址获取：[此处应为URL，原文中未提供具体链接，请访问相关论文或项目页面获取]
URL
https://arxiv.org/abs/2505.02481
PDF
-
A Dual-Task Synergy-Driven Generalization Framework for Pancreatic Cancer Segmentation in CT Scans
2025-05-03 00:54:00Jun Li, Yijue Zhang, Haibo Shi, Minhong Li, Qiwei Li, Xiaohua QianAbstract
Pancreatic cancer, characterized by its notable prevalence and mortality rates, demands accurate lesion delineation for effective diagnosis and therapeutic interventions. The generalizability of extant methods is frequently compromised due to the pronounced variability in imaging and the heterogeneous characteristics of pancreatic lesions, which may mimic normal tissues and exhibit significant inter-patient variability. Thus, we propose a generalization framework that synergizes pixel-level classification and regression tasks, to accurately delineate lesions and improve model stability. This framework not only seeks to align segmentation contours with actual lesions but also uses regression to elucidate spatial relationships between diseased and normal tissues, thereby improving tumor localization and morphological characterization. Enhanced by the reciprocal transformation of task outputs, our approach integrates additional regression supervision within the segmentation context, bolstering the model's generalization ability from a dual-task perspective. Besides, dual self-supervised learning in feature spaces and output spaces augments the model's representational capability and stability across different imaging views. Experiments on 594 samples composed of three datasets with significant imaging differences demonstrate that our generalized pancreas segmentation results comparable to mainstream in-domain validation performance (Dice: 84.07%). More importantly, it successfully improves the results of the highly challenging cross-lesion generalized pancreatic cancer segmentation task by 9.51%. Thus, our model constitutes a resilient and efficient foundational technological support for pancreatic disease management and wider medical applications. The codes will be released at this https URL.
Abstract (translated)
胰腺癌因其高发病率和死亡率，需要准确地对病灶进行勾画以实现有效的诊断和治疗。现有的方法通常由于成像的显著变异性以及胰腺病变异质性特征（这些特征可能与正常组织相似，并表现出明显的患者间差异）而难以泛化应用。因此，我们提出了一种综合框架，该框架结合了像素级分类和回归任务，以准确勾画病灶并提高模型稳定性。此框架不仅力求使分割轮廓与实际病变对齐，还利用回归来阐明患病组织和正常组织之间的空间关系，从而改善肿瘤定位和形态特征描述。通过任务输出的相互转换，我们的方法在分割上下文中集成了额外的回归监督，从双任务的角度增强了模型的泛化能力。此外，在特征空间和输出空间中进行双重自监督学习增强了模型在不同成像视图下的表示能力和稳定性。 我们在一个由三组具有显著影像差异性的594个样本组成的实验数据集中进行了测试，结果表明我们的通用胰腺分割方法在主流领域内验证性能方面（Dice系数：84.07%）表现相当。更重要的是，在高挑战性的跨病灶泛化胰腺癌分割任务中提高了9.51%的表现。因此，我们的模型为胰腺疾病的管理和更广泛的医疗应用提供了稳健且高效的基础设施技术支撑。代码将在此网址发布。
URL
https://arxiv.org/abs/2505.01644
PDF
-
Global Collinearity-aware Polygonizer for Polygonal Building Mapping in Remote Sensing
2025-05-02 16:49:07Fahong Zhang, Yilei Shi, Xiao Xiang ZhuAbstract
This paper addresses the challenge of mapping polygonal buildings from remote sensing images and introduces a novel algorithm, the Global Collinearity-aware Polygonizer (GCP). GCP, built upon an instance segmentation framework, processes binary masks produced by any instance segmentation model. The algorithm begins by collecting polylines sampled along the contours of the binary masks. These polylines undergo a refinement process using a transformer-based regression module to ensure they accurately fit the contours of the targeted building instances. Subsequently, a collinearity-aware polygon simplification module simplifies these refined polylines and generate the final polygon representation. This module employs dynamic programming technique to optimize an objective function that balances the simplicity and fidelity of the polygons, achieving globally optimal solutions. Furthermore, the optimized collinearity-aware objective is seamlessly integrated into network training, enhancing the cohesiveness of the entire pipeline. The effectiveness of GCP has been validated on two public benchmarks for polygonal building mapping. Further experiments reveal that applying the collinearity-aware polygon simplification module to arbitrary polylines, without prior knowledge, enhances accuracy over traditional methods such as the Douglas-Peucker algorithm. This finding underscores the broad applicability of GCP. The code for the proposed method will be made available at this https URL.
Abstract (translated)
本文探讨了从遥感图像中绘制多边形建筑物的挑战，并引入了一种新的算法——全局共线性感知多边形化器（Global Collinearity-aware Polygonizer，简称GCP）。GCP建立在实例分割框架之上，处理由任何实例分割模型生成的二值掩码。该算法首先收集沿二值掩码轮廓采样的折线，然后利用基于变压器的回归模块对这些折线进行细化处理，以确保它们能准确地贴合目标建筑物实例的轮廓。接下来，一个共线性感知多边形简化模块会对经过优化的折线进行简化，并生成最终的多边形表示。此模块采用动态规划技术来优化平衡多边形简洁性和精确性的目标函数，从而实现全局最优解。此外，该优化后的共线性感知目标被无缝集成到网络训练中，增强了整个流程的一致性。 GCP的有效性已在两个用于多边形建筑物映射的公共基准测试上得到了验证。进一步的实验表明，在没有先验知识的情况下，将共线性感知多边形简化模块应用于任意折线可以提高准确性，并优于传统的如道格拉斯-普克算法等方法。这一发现强调了GCP的广泛应用前景。 该方法的代码将在以下链接提供：[此URL]。
URL
https://arxiv.org/abs/2505.01385
PDF
-
AI-Assisted Decision-Making for Clinical Assessment of Auto-Segmented Contour Quality
2025-05-01 05:05:35Biling Wang, Austen Maniscalco, Ti Bai, Siqiu Wang, Michael Dohopolski, Mu-Han Lin, Chenyang Shen, Dan Nguyen, Junzhou Huang, Steve Jiang, Xinlei WangAbstract
Purpose: This study presents a Deep Learning (DL)-based quality assessment (QA) approach for evaluating auto-generated contours (auto-contours) in radiotherapy, with emphasis on Online Adaptive Radiotherapy (OART). Leveraging Bayesian Ordinal Classification (BOC) and calibrated uncertainty thresholds, the method enables confident QA predictions without relying on ground truth contours or extensive manual labeling. Methods: We developed a BOC model to classify auto-contour quality and quantify prediction uncertainty. A calibration step was used to optimize uncertainty thresholds that meet clinical accuracy needs. The method was validated under three data scenarios: no manual labels, limited labels, and extensive labels. For rectum contours in prostate cancer, we applied geometric surrogate labels when manual labels were absent, transfer learning when limited, and direct supervision when ample labels were available. Results: The BOC model delivered robust performance across all scenarios. Fine-tuning with just 30 manual labels and calibrating with 34 subjects yielded over 90% accuracy on test data. Using the calibrated threshold, over 93% of the auto-contours' qualities were accurately predicted in over 98% of cases, reducing unnecessary manual reviews and highlighting cases needing correction. Conclusion: The proposed QA model enhances contouring efficiency in OART by reducing manual workload and enabling fast, informed clinical decisions. Through uncertainty quantification, it ensures safer, more reliable radiotherapy workflows.
Abstract (translated)
研究目的：这项研究提出了一种基于深度学习（DL）的质量评估（QA）方法，用于评价放射治疗中自动生成的轮廓（auto-contours），特别强调了在线自适应放射治疗（OART）。该方法利用贝叶斯序值分类（BOC）和校准后的不确定性阈值，在没有依赖地面真实轮廓或大量手动标记的情况下，能够提供自信的质量评估预测。 研究方法：我们开发了一种BOC模型来对auto-contour质量进行分类，并量化预测的不确定性。通过优化符合临床精度需求的不确定性的阈值进行校正。该方法在三种数据场景下进行了验证：没有手动标签、有限的手动标签和大量的手动标签。对于前列腺癌患者的直肠轮廓，当缺少手动标签时我们应用几何替代标签；当标记数量有限时使用迁移学习；当有足够多的手动标签时则直接进行监督训练。 研究结果：BOC模型在所有场景中都表现出色。通过仅用30个手动标签微调，并对34个受试者的数据进行校准，测试数据的准确率超过90%。利用校正后的阈值，在超过98%的情况下能够准确预测超过93%的auto-contours的质量，这减少了不必要的手动审查并突出需要修正的情况。 结论：所提出的QA模型通过减少手工工作量并支持快速、有信息支撑的临床决策，提高了OART中轮廓绘制的效率。通过不确定性量化，它确保了更安全且可靠的放射治疗流程。
URL
https://arxiv.org/abs/2505.00308
PDF
-
GVPT -- A software for guided visual pitch tracking
2025-05-01 00:55:52Hyunjin Cho, Farhad Tabasi, Jeremy D. Greenlee, Rahul SinghAbstract
GVPT (Guided visual pitch tracking) is a publicly available, real-time pitch tracking software designed to guide and evaluate vocal pitch control using visual feedback. Developed for clinical and research applications, the system presents various visual target pitch contour and overlays the subject's pitch in real-time to promote accurate vocal reproduction. GVPT supports difficulty modification, session logging, and precise pitch tracking. The software enables voice pitch control exercise in both experimental and therapeutic settings.
Abstract (translated)
GVPT（引导式视觉音高跟踪）是一款公开可用的实时音高追踪软件，旨在通过视觉反馈指导和评估声乐音高的控制。该系统专为临床和研究应用而开发，它呈现各种视觉目标音高轮廓，并在实时中叠加受试者的音高，以促进准确的声音重现。GVPT支持难度调整、会话记录以及精确的音高追踪功能。这款软件能够在实验和治疗环境中进行声乐音高控制练习。
URL
https://arxiv.org/abs/2505.00750
PDF
-
Myocardial Region-guided Feature Aggregation Net for Automatic Coronary artery Segmentation and Stenosis Assessment using Coronary Computed Tomography Angiography
2025-04-27 16:43:52Ni Yao, Xiangyu Liu, Danyang Sun, Chuang Han, Yanting Li, Jiaofen Nan, Chengyang Li, Fubao Zhu, Weihua Zhou, Chen ZhaoAbstract
Coronary artery disease (CAD) remains a leading cause of mortality worldwide, requiring accurate segmentation and stenosis detection using Coronary Computed Tomography angiography (CCTA). Existing methods struggle with challenges such as low contrast, morphological variability and small vessel segmentation. To address these limitations, we propose the Myocardial Region-guided Feature Aggregation Net, a novel U-shaped dual-encoder architecture that integrates anatomical prior knowledge to enhance robustness in coronary artery segmentation. Our framework incorporates three key innovations: (1) a Myocardial Region-guided Module that directs attention to coronary regions via myocardial contour expansion and multi-scale feature fusion, (2) a Residual Feature Extraction Encoding Module that combines parallel spatial channel attention with residual blocks to enhance local-global feature discrimination, and (3) a Multi-scale Feature Fusion Module for adaptive aggregation of hierarchical vascular features. Additionally, Monte Carlo dropout f quantifies prediction uncertainty, supporting clinical interpretability. For stenosis detection, a morphology-based centerline extraction algorithm separates the vascular tree into anatomical branches, enabling cross-sectional area quantification and stenosis grading. The superiority of MGFA-Net was demonstrated by achieving an Dice score of 85.04%, an accuracy of 84.24%, an HD95 of 6.1294 mm, and an improvement of 5.46% in true positive rate for stenosis detection compared to3D U-Net. The integrated segmentation-to-stenosis pipeline provides automated, clinically interpretable CAD assessment, bridging deep learning with anatomical prior knowledge for precision medicine. Our code is publicly available at this http URL
Abstract (translated)
冠状动脉疾病（CAD）仍然是全球主要的死亡原因之一，需要通过冠状动脉计算机断层血管造影（CCTA）进行准确的分割和狭窄检测。现有方法在面对低对比度、形态变化及小血管分割等挑战时存在困难。为解决这些限制，我们提出了心肌区域引导特征聚合网络（Myocardial Region-guided Feature Aggregation Net, MGFA-Net），这是一种新颖的U形双编码器架构，通过整合解剖先验知识来增强冠状动脉分割的鲁棒性。 我们的框架包括三个关键创新： 1. 心肌区域导向模块：通过心肌轮廓扩展和多尺度特征融合将注意力集中在冠状动脉区域。 2. 残差特征提取编码模块：结合并行空间通道注意机制与残差块，增强局部-全局特征区分度。 3. 多尺度特征融合模块：用于自适应聚合分层血管特征。 此外，通过蒙特卡洛丢弃法量化预测不确定性，支持临床可解释性。对于狭窄检测，我们采用了一种基于形态学的中心线提取算法来将血管树划分为解剖分支，从而实现横截面面积定量和狭窄分级。MGFA-Net的优势在于其在Dice得分（85.04%）、准确率（84.24%）及HD95值（6.1294毫米）上的优异表现，并且在狭窄检测的真阳性率上相较于3D U-Net提升了5.46%，表明了其在冠状动脉疾病评估中的优越性。 该集成分割至狭窄检测管道提供了一种自动化的、临床可解释的CAD评估方法，将深度学习与解剖先验知识相结合，为精准医疗提供了支持。我们的代码公开可用，在此链接中可以找到：[此处插入URL]
URL
https://arxiv.org/abs/2504.19300
PDF
-
PixelWeb: The First Web GUI Dataset with Pixel-Wise Labels
2025-04-23 05:01:25Qi Yang, Weichen Bi, Haiyang Shen, Yaoqi Guo, Yun MaAbstract
Graphical User Interface (GUI) datasets are crucial for various downstream tasks. However, GUI datasets often generate annotation information through automatic labeling, which commonly results in inaccurate GUI element BBox annotations, including missing, duplicate, or meaningless BBoxes. These issues can degrade the performance of models trained on these datasets, limiting their effectiveness in real-world applications. Additionally, existing GUI datasets only provide BBox annotations visually, which restricts the development of visually related GUI downstream tasks. To address these issues, we introduce PixelWeb, a large-scale GUI dataset containing over 100,000 annotated web pages. PixelWeb is constructed using a novel automatic annotation approach that integrates visual feature extraction and Document Object Model (DOM) structure analysis through two core modules: channel derivation and layer analysis. Channel derivation ensures accurate localization of GUI elements in cases of occlusion and overlapping elements by extracting BGRA four-channel bitmap annotations. Layer analysis uses the DOM to determine the visibility and stacking order of elements, providing precise BBox annotations. Additionally, PixelWeb includes comprehensive metadata such as element images, contours, and mask annotations. Manual verification by three independent annotators confirms the high quality and accuracy of PixelWeb annotations. Experimental results on GUI element detection tasks show that PixelWeb achieves performance on the mAP95 metric that is 3-7 times better than existing datasets. We believe that PixelWeb has great potential for performance improvement in downstream tasks such as GUI generation and automated user interaction.
Abstract (translated)
图形用户界面（GUI）数据集对于各种下游任务至关重要。然而，这些数据集中生成注释信息通常通过自动标注完成，这常常导致不准确的GUI元素边界框（BBox）注释问题，包括遗漏、重复或无意义的BBoxes。这些问题会降低在这些数据集上训练的模型性能，在现实世界的应用中限制了它们的有效性。此外，现有的GUI数据集仅提供可视化的边界框标注信息，这限制了与视觉相关GUI下游任务的发展。 为了解决上述问题，我们引入了一个大型的GUI数据集PixelWeb，它包含了超过10万张注释网页。PixelWeb使用了一种新颖的自动标注方法来构建，该方法结合了视觉特征提取和文档对象模型（DOM）结构分析，并通过两个核心模块实现：通道导出和层级分析。通道导出通过提取BGRA四通道位图注释确保在遮挡和元素重叠的情况下准确定位GUI元素。层级分析利用DOM确定元素的可见性和堆叠顺序，从而提供精确的BBox注释。此外，PixelWeb还包含全面的元数据信息，如元素图像、轮廓及掩码标注。 通过三名独立标注者的手动验证确认了PixelWeb注释具有高质量和准确性。在GUI元素检测任务上的实验结果表明，在mAP95指标上，PixelWeb的表现比现有的数据集高出3-7倍。我们相信PixelWeb对于下游任务如GUI生成和自动化用户交互等领域的性能改进具有巨大的潜力。
URL
https://arxiv.org/abs/2504.16419
PDF
-
Contour Field based Elliptical Shape Prior for the Segment Anything Model
2025-04-17 01:08:24Xinyu Zhao, Jun Liu, Faqiang Wang, Li Cui, Yuping DuanAbstract
The elliptical shape prior information plays a vital role in improving the accuracy of image segmentation for specific tasks in medical and natural images. Existing deep learning-based segmentation methods, including the Segment Anything Model (SAM), often struggle to produce segmentation results with elliptical shapes efficiently. This paper proposes a new approach to integrate the prior of elliptical shapes into the deep learning-based SAM image segmentation techniques using variational methods. The proposed method establishes a parameterized elliptical contour field, which constrains the segmentation results to align with predefined elliptical contours. Utilizing the dual algorithm, the model seamlessly integrates image features with elliptical priors and spatial regularization priors, thereby greatly enhancing segmentation accuracy. By decomposing SAM into four mathematical sub-problems, we integrate the variational ellipse prior to design a new SAM network structure, ensuring that the segmentation output of SAM consists of elliptical regions. Experimental results on some specific image datasets demonstrate an improvement over the original SAM.
Abstract (translated)
椭圆形的先验信息在医学和自然图像特定任务中的图像分割准确性提升方面起着关键作用。现有的基于深度学习的分割方法，包括“Segment Anything Model（SAM）”，往往难以有效地生成具有椭圆形状的分割结果。本文提出了一种新方法，通过变分法将椭圆形的先验信息整合到基于深度学习的SAM图像分割技术中。所提出的方法建立了一个参数化的椭圆轮廓场，限制了分割结果与预定义的椭圆轮廓对齐。利用双算法，该模型可以无缝地融合图像特征和椭圆先验及空间正则化先验，从而大幅提高分割精度。通过将SAM分解为四个数学子问题，我们将变分椭圆先验整合到新的SAM网络结构设计中，确保了SAM的分割输出由椭圆形区域组成。在某些特定图像数据集上的实验结果表明，相较于原始的SAM方法，本研究的方法有所改进。
URL
https://arxiv.org/abs/2504.12556
PDF
-
MixSignGraph: A Sign Sequence is Worth Mixed Graphs of Nodes
2025-04-16 12:23:30Shiwei Gan, Yafeng Yin, Zhiwei Jiang, Hongkai Wen, Lei Xie, Sanglu LuAbstract
Recent advances in sign language research have benefited from CNN-based backbones, which are primarily transferred from traditional computer vision tasks (\eg object identification, image recognition). However, these CNN-based backbones usually excel at extracting features like contours and texture, but may struggle with capturing sign-related features. In fact, sign language tasks require focusing on sign-related regions, including the collaboration between different regions (\eg left hand region and right hand region) and the effective content in a single region. To capture such region-related features, we introduce MixSignGraph, which represents sign sequences as a group of mixed graphs and designs the following three graph modules for feature extraction, \ie Local Sign Graph (LSG) module, Temporal Sign Graph (TSG) module and Hierarchical Sign Graph (HSG) module. Specifically, the LSG module learns the correlation of intra-frame cross-region features within one frame, \ie focusing on spatial features. The TSG module tracks the interaction of inter-frame cross-region features among adjacent frames, \ie focusing on temporal features. The HSG module aggregates the same-region features from different-granularity feature maps of a frame, \ie focusing on hierarchical features. In addition, to further improve the performance of sign language tasks without gloss annotations, we propose a simple yet counter-intuitive Text-driven CTC Pre-training (TCP) method, which generates pseudo gloss labels from text labels for model pre-training. Extensive experiments conducted on current five public sign language datasets demonstrate the superior performance of the proposed model. Notably, our model surpasses the SOTA models on multiple sign language tasks across several datasets, without relying on any additional cues.
Abstract (translated)
最近的手语研究进展得益于基于卷积神经网络（CNN）的骨干架构，这些架构主要从传统的计算机视觉任务中转移而来（例如物体识别、图像识别）。然而，这类基于CNN的骨干通常擅长提取轮廓和纹理等特征，但可能难以捕捉与手语相关的特征。实际上，手语任务需要关注特定的手语相关区域，包括不同区域之间的协作（如左手区域和右手区域）以及单个区域内有效内容的理解。为了捕获这些区域相关的特征，我们引入了MixSignGraph模型，该模型将手势序列表示为混合图组，并设计了三个用于特征提取的图模块：即局部手语图（LSG）模块、时间手语图（TSG）模块和层次化手语图（HSG）模块。具体来说，LSG模块学习同一帧内跨区域特征之间的相关性，即关注空间特征；TSG模块跟踪相邻帧间跨区域特征的交互作用，即关注时间特征；而HSG模块聚合来自单个帧不同粒度特征图中的相同区域特征，即关注层次化特征。此外，为了进一步提高无音节注释的手语任务性能，我们提出了一种简单却反直觉的文本驱动CTC预训练（TCP）方法，该方法从文本标签中生成伪音节标签进行模型预训练。在目前五个公开的手语数据集上进行的广泛实验表明了所提模型的优越性。值得注意的是，在多个手语任务和几个数据集中，我们的模型无需任何额外线索即可超越最先进的模型。
URL
https://arxiv.org/abs/2504.12020
PDF
-
Towards an AI Observatory for the Nuclear Sector: A tool for anticipatory governance
Abstract
AI models are rapidly becoming embedded in all aspects of nuclear energy research and work but the safety, security, and safeguards consequences of this embedding are not well understood. In this paper, we call for the creation of an anticipatory system of governance for AI in the nuclear sector as well as the creation of a global AI observatory as a means for operationalizing anticipatory governance. The paper explores the contours of the nuclear AI observatory and an anticipatory system of governance by drawing on work in science and technology studies, public policy, and foresight studies.
Abstract (translated)
人工智能模型正在迅速渗透到核能研究和工作的各个方面，但这种嵌入对安全、保障以及保护措施的潜在影响尚未得到充分理解。本文呼吁在核行业建立一个针对AI的前瞻性治理体系，并提出创建全球AI观测站作为实现前瞻性治理的一种手段。该论文借鉴了科学与技术研究、公共政策及前景研究领域的成果，探讨了核能AI观测站和前瞻性治理体系的基本框架。
URL
https://arxiv.org/abs/2504.12358
PDF
-
Xpose: Bi-directional Engineering for Hidden Query Extraction
Abstract
Query reverse engineering (QRE) aims to synthesize a SQL query to connect a given database and result instance. A recent variation of QRE is where an additional input, an opaque executable containing a ground-truth query, is provided, and the goal is to non-invasively extract this specific query through only input-output examples. This variant, called Hidden Query Extraction (HQE), has a spectrum of industrial use-cases including query recovery, database security, and vendor migration. The reverse engineering (RE) tools developed for HQE, which are based on database mutation and generation techniques, can only extract flat queries with key-based equi joins and conjunctive arithmetic filter predicates, making them limited wrt both query structure and query operators. In this paper, we present Xpose, a HQE solution that elevates the extraction scope to realistic complex queries, such as those found in the TPCH benchmark. A two-pronged approach is taken: (1) The existing RE scope is substantially extended to incorporate union connectors, algebraic filter predicates, and disjunctions for both values and predicates. (2) The predictive power of LLMs is leveraged to convert business descriptions of the opaque application into extraction guidance, representing ``forward engineering" (FE). The FE module recognizes common constructs, such as nesting of sub-queries, outer joins, and scalar functions. In essence, FE establishes the broad query contours, while RE fleshes out the fine-grained details. We have evaluated Xpose on (a) E-TPCH, a query suite comprising the complete TPCH benchmark extended with queries featuring unions, diverse join types, and sub-queries; and (b) the real-world STACK benchmark. The experimental results demonstrate that its bi-directional engineering approach accurately extracts these complex queries, representing a significant step forward with regard to HQE coverage.
Abstract (translated)
查询逆向工程（Query Reverse Engineering，QRE）旨在合成一个SQL查询，以连接给定的数据库和结果实例。最近出现的一种QRE变体是提供了一个额外输入——包含真实查询的不透明可执行文件，并且目标是非侵入性地通过仅使用输入输出示例提取该特定查询。这种被称为隐藏查询提取（Hidden Query Extraction，HQE）的方法在包括查询恢复、数据库安全和供应商迁移等众多工业应用场景中都有广泛用途。然而，为HQE开发的逆向工程工具基于数据库变异和生成技术，只能提取具有键值相等连接且带有算术过滤谓词的平面查询，这使得它们在查询结构和操作符方面都受到限制。 本文介绍了一种名为Xpose的新解决方案，它扩展了HQE的提取范围至现实中的复杂查询，例如TPCH基准测试中发现的查询。Xpose采取了一个两管齐下的方法：（1）大幅扩大现有的逆向工程范围以包含联合连接器、代数过滤谓词以及值和谓词的析取；（2）利用大型语言模型（LLMs）的预测能力将不透明应用的商业描述转换为提取指导，这一过程代表了“正向工程”（Forward Engineering，FE）。FE模块可以识别诸如子查询嵌套、外连接和标量函数等常见构造。简而言之，FE确立了整个查询的大致轮廓，而逆向工程则填充详细的细节。 我们已在以下两个方面评估了Xpose：(a) E-TPCH——一个由完整的TPCH基准测试扩展而来的一组查询集，其中包含联合、多样化的连接类型和子查询；(b) 实际世界中的STACK基准。实验结果表明，Xpose的双向工程方法能够准确提取这些复杂查询，并在HQE覆盖范围方面取得了显著进展。
URL
https://arxiv.org/abs/2504.10898
PDF
