Abstract
We present Lang2Motion, a framework for language-guided point trajectory generation by aligning motion manifolds with joint embedding spaces. Unlike prior work focusing on human motion or video synthesis, we generate explicit trajectories for arbitrary objects using motion extracted from real-world videos via point tracking. Our transformer-based auto-encoder learns trajectory representations through dual supervision: textual motion descriptions and rendered trajectory visualizations, both mapped through CLIP's frozen encoders. Lang2Motion achieves 34.2% Recall@1 on text-to-trajectory retrieval, outperforming video-based methods by 12.5 points, and improves motion accuracy by 33-52% (12.4 ADE vs 18.3-25.3) compared to video generation baselines. We demonstrate 88.3% Top-1 accuracy on human action recognition despite training only on diverse object motions, showing effective transfer across motion domains. Lang2Motion supports style transfer, semantic interpolation, and latent-space editing through CLIP-aligned trajectory representations.
Abstract (translated)
我们提出了Lang2Motion，这是一个通过将运动流形与联合嵌入空间对齐来生成语言引导的点轨迹的框架。不同于先前专注于人体动作或视频合成的工作，我们利用从现实世界视频中提取并通过点跟踪获得的运动数据，为任意对象生成显式的轨迹。我们的基于变压器的自编码器通过双重监督学习轨迹表示：文本运动描述和渲染后的轨迹可视化，两者都通过CLIP的冻结编码器进行映射。 Lang2Motion在文本到轨迹检索上达到了34.2%的Recall@1，比视频基线方法高出12.5个百分点，并将动作准确性提高了33-52%，与视频生成基准相比（ADE值为12.4 vs 18.3-25.3）。尽管训练数据仅为多样化对象的动作，但我们在人类动作识别上达到了88.3%的Top-1准确率，证明了在不同运动领域之间的有效迁移。Lang2Motion支持风格转移、语义插值以及通过与CLIP对齐的轨迹表示进行潜在空间编辑。
URL
https://arxiv.org/abs/2512.10617
