Abstract
We introduce a real-time, human-in-the-loop gesture control framework that can dynamically adapt audio and music based on human movement by analyzing live video input. By creating a responsive connection between visual and auditory stimuli, this system enables dancers and performers to not only respond to music but also influence it through their movements. Designed for live performances, interactive installations, and personal use, it offers an immersive experience where users can shape the music in real time. The framework integrates computer vision and machine learning techniques to track and interpret motion, allowing users to manipulate audio elements such as tempo, pitch, effects, and playback sequence. With ongoing training, it achieves user-independent functionality, requiring as few as 50 to 80 samples to label simple gestures. This framework combines gesture training, cue mapping, and audio manipulation to create a dynamic, interactive experience. Gestures are interpreted as input signals, mapped to sound control commands, and used to naturally adjust music elements, showcasing the seamless interplay between human interaction and machine response.
Abstract (translated)
我们介绍了一种实时的人机交互姿态控制框架，该框架能够通过分析实时视频输入动态地根据人体动作调整音频和音乐。通过建立视觉与听觉刺激之间的响应联系，此系统使舞者和表演者不仅能够回应音乐，还能通过他们的动作影响音乐。专为现场演出、互动装置和个人使用设计，它提供了一种沉浸式体验，在这种体验中用户可以实时塑造音乐。该框架整合了计算机视觉和机器学习技术来跟踪和解释运动，让用户能够操控音频元素如节奏、音高、效果以及播放顺序。 通过持续训练，此系统实现了独立于用户的功能，仅需50到80个样本就可以标记简单的手势。这个框架结合了姿态培训、提示映射和音频操作，以创建一种动态且互动的体验。姿势被解读为输入信号，并将其映射至声音控制命令，用于自然地调整音乐元素，展示了人机交互与机器响应之间的无缝融合。
URL
https://arxiv.org/abs/2504.19460
