Abstract
Autonomous space operations such as on-orbit servicing and active debris removal demand robust part-level semantic understanding and precise relative navigation of target spacecraft, yet collecting large-scale real data in orbit remains impractical due to cost and access constraints. Existing synthetic datasets, moreover, suffer from limited target diversity, single-modality sensing, and incomplete ground-truth annotations. We present \textbf{SpaceSense-Bench}, a large-scale multi-modal benchmark for spacecraft perception encompassing 136~satellite models with approximately 70~GB of data. Each frame provides time-synchronized 1024$\times$1024 RGB images, millimeter-precision depth maps, and 256-beam LiDAR point clouds, together with dense 7-class part-level semantic labels at both the pixel and point level as well as accurate 6-DoF pose ground truth. The dataset is generated through a high-fidelity space simulation built in Unreal Engine~5 and a fully automated pipeline covering data acquisition, multi-stage quality control, and conversion to mainstream formats. We benchmark five representative tasks (object detection, 2D semantic segmentation, RGB--LiDAR fusion-based 3D point cloud segmentation, monocular depth estimation, and orientation estimation) and identify two key findings: (i)~perceiving small-scale components (\emph{e.g.}, thrusters and omni-antennas) and generalizing to entirely unseen spacecraft in a zero-shot setting remain critical bottlenecks for current methods, and (ii)~scaling up the number of training satellites yields substantial performance gains on novel targets, underscoring the value of large-scale, diverse datasets for space perception research. The dataset, code, and toolkit are publicly available at this https URL.
Abstract (translated)
自主空间操作，如轨道服务和主动碎片清除，需要强大的部件级语义理解和目标航天器的精准相对导航。然而，由于成本和访问限制，在轨收集大规模真实数据仍然不切实际。现有的合成数据集还存在目标多样性有限、单一模态传感以及地面实况标注不完整的问题。我们提出了**SpaceSense-Bench**，这是一个涵盖136颗卫星模型的大型多模态基准库，包含大约70GB的数据。每个帧提供时间同步的1024×1024 RGB图像、毫米精度深度图和256束LiDAR点云，并附有像素级和点级密集的7类部件级语义标签以及精确的6自由度姿态地面实况标注。 该数据集通过在Unreal Engine 5中构建的高度逼真的空间模拟生成，并采用完全自动化的管道，涵盖数据采集、多阶段质量控制及转换为主流格式。我们针对五个代表性任务（物体检测、2D语义分割、基于RGB-LiDAR融合的3D点云分割、单目深度估计和姿态估计）进行了基准测试，并识别出两个关键发现：(i) 当前方法在感知小规模组件（例如，推进器和全向天线）以及零样本设置下泛化到完全未知的航天器方面仍然面临重大瓶颈；(ii) 增加训练卫星的数量会显著提高对新目标的表现，这凸显了大规模、多样化数据集对于空间感知研究的价值。该数据集、代码及工具包可在[公开网址]获取。
URL
https://arxiv.org/abs/2603.09320
