Abstract
While music generative models have recently gained significant attention, how they can be effectively integrated into live music performances still requires further exploration. This paper presents Calliphony, a calligraphy-driven interface for real-time generative music performance. Specifically, we build a low-latency pipeline that captures brush motion with an attachable sensor and maps it to control signals for real-time symbolic music generation. Using a generative model, the system produces multi-track MIDI in performance settings, while brush-derived control signals constrain event timing and activate additional musical layers. The generated melody is then extended with real-time harmony and additional voices, and finally rendered through a DAW for live staging. Calliphony contributes: (1) a performance-oriented prototype that uses calligraphic motion as an external control layer for a real-time symbolic music generation model, controlling note density, pitch constraints, and accompaniment-layer activation; and (2) a cross-modal performance scenario that extends calligraphy beyond a primarily visual practice into an audiovisual, AI-assisted setting.
Abstract (translated)
URL
https://arxiv.org/abs/2608.03040
