Abstract
Human mesh reconstruction (HMR) provides direct insights into body-environment interaction, which enables various immersive applications. While existing large-scale HMR datasets rely heavily on line-of-sight RGB input, vision-based sensing is limited by occlusion, lighting variation, and privacy concerns. To overcome these limitations, recent efforts have explored radio-frequency (RF) mmWave radar for privacy-preserving indoor human sensing. However, current radar datasets are constrained by sparse skeleton labels, limited scale, and simple in-place actions. To advance the HMR research community, we introduce M4Human, the current largest-scale (661K-frame) ($9\times$ prior largest) multimodal benchmark, featuring high-resolution mmWave radar, RGB, and depth data. M4Human provides both raw radar tensors (RT) and processed radar point clouds (RPC) to enable research across different levels of RF signal granularity. M4Human includes high-quality motion capture (MoCap) annotations with 3D meshes and global trajectories, and spans 20 subjects and 50 diverse actions, including in-place, sit-in-place, and free-space sports or rehabilitation movements. We establish benchmarks on both RT and RPC modalities, as well as multimodal fusion with RGB-D modalities. Extensive results highlight the significance of M4Human for radar-based human modeling while revealing persistent challenges under fast, unconstrained motion. The dataset and code will be released after the paper publication.
Abstract (translated)
人体网格重建（HMR）提供了对人体与环境互动的直接洞察，这为各种沉浸式应用铺平了道路。尽管现有的大规模HMR数据集主要依赖于视线内的RGB输入，但基于视觉的感应技术受到遮挡、光照变化和隐私问题的限制。为了克服这些限制，最近的研究开始探索使用无线电频率（RF）毫米波雷达进行非接触式的室内人体感知。然而，目前的雷达数据集由于稀疏的骨骼标签、规模有限以及仅限于在地面上的动作而存在局限性。 为推进HMR研究社区的发展，我们引入了M4Human——当前规模最大（661K帧）、比现有最大数据集大9倍的多模态基准数据集。该数据集包括高分辨率毫米波雷达、RGB和深度图像数据，并提供原始雷达张量（RT）和处理过的雷达点云（RPC），以便在不同的RF信号粒度级别上进行研究。 M4Human包含高质量的动作捕捉（MoCap）注释，包括3D网格模型和全局轨迹，并涵盖了20个主体及50种多样化的动作，涵盖在地面上、就座状态下的动作以及自由空间内的运动或康复训练。我们分别针对RT和RPC模态建立了基准测试，并且还设置了多模态融合（RGB-D）的基准。 大量的实验结果突显了M4Human对于基于雷达的人体建模的重要性，同时也揭示了在快速、不受限制的动作下持续存在的挑战。该数据集及代码将在论文发表后公开发布。
URL
https://arxiv.org/abs/2512.12378
