Abstract
Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.
Abstract (translated)
理解自由活动动物的行为是神经科学的核心，其中姿态估计和行为理解构成了将神经活动与自然动作联系起来的基础。然而，这两项任务仍然高度依赖于人工标注或不稳定的无监督流程，从而限制了其可扩展性和再现性。我们提出了BehaviorVLM，这是一种统一的视觉-语言框架，用于姿态估计和行为理解，无需特定任务微调，并通过详细的、明确且可验证的推理步骤引导预训练的视觉-语言模型（VLMs），大大减少了对人类标注的需求。 对于姿态估计，我们利用量子点接地的行为数据并提出了一种多阶段管道，该管道集成了时间、空间和跨视图推理。这种设计极大地减少了人工注释的工作量，并通过几何检查（如重新投影误差）来揭示低置信度标签，从而生成可后续过滤、修正或用于微调下游姿态模型的标签。 对于行为理解，我们提出了一种集成深度嵌入聚类以发现过度分割的行为、基于VLM的逐片段视频描述以及LLM推理合并和语义标记行为段的管道。这种行为分析流程可以直接从视觉信息中操作，并不需要关键点来划分行为。 这些组件共同使得多动物行为的大规模、可解释且标签轻量级的分析成为可能。
URL
https://arxiv.org/abs/2603.12176
