Abstract
Spatial audio is crucial for immersive 360-degree video experiences, yet most 360-degree videos lack it due to the difficulty of capturing spatial audio during recording. Automatically generating spatial audio such as first-order ambisonics (FOA) from video therefore remains an important but challenging problem. In complex scenes, sound perception depends not only on sound source locations but also on scene geometry, materials, and dynamic interactions with the environment. However, existing approaches only rely on visual cues and fail to model dynamic sources and acoustic effects such as occlusion, reflections, and reverberation. To address these challenges, we propose DynFOA, a generative framework that synthesizes FOA from 360-degree videos by integrating dynamic scene reconstruction with conditional diffusion modeling. DynFOA analyzes the input video to detect and localize dynamic sound sources, estimate depth and semantics, and reconstruct scene geometry and materials using 3D Gaussian Splatting (3DGS). The reconstructed scene representation provides physically grounded features that capture acoustic interactions between sources, environment, and listener viewpoint. Conditioned on these features, a diffusion model generates spatial audio consistent with the scene dynamics and acoustic context. We introduce M2G-360, a dataset of 600 real-world clips divided into MoveSources, Multi-Source, and Geometry subsets for evaluating robustness under diverse conditions. Experiments show that DynFOA consistently outperforms existing methods in spatial accuracy, acoustic fidelity, distribution matching, and perceived immersive experience.
Abstract (translated)
空间音频对于沉浸式360度视频体验至关重要，但大多数360度视频因录制时难以捕捉空间音频而缺乏该要素。因此，从视频自动生成一阶全景声（FOA）等空间音频仍是一个重要且具有挑战性的问题。在复杂场景中，声音感知不仅取决于声源位置，还依赖于场景几何结构、材质以及与环境动态交互产生的声学效应。然而现有方法仅依赖视觉线索，无法对动态声源及遮挡、反射、混响等声学效应进行建模。为应对这些挑战，我们提出了DynFOA——一个通过融合动态场景重建与条件扩散建模来从360度视频合成FOA的生成框架。DynFOA分析输入视频以检测并定位动态声源、估算深度与语义信息，并利用三维高斯泼溅（3DGS）重建场景几何与材质。重建的场景表征提供了捕捉声源、环境与听者视角之间声学交互的物理基础特征。基于这些特征，扩散模型生成了与场景动态及声学环境一致的空间音频。我们构建了包含600个真实场景片段的M2G-360数据集，该数据集分为移动声源、多声源和几何结构三个子集，用于评估方法在不同条件下的鲁棒性。实验表明，DynFOA在空间准确性、声学保真度、分布匹配度及感知沉浸体验方面均持续优于现有方法。
URL
https://arxiv.org/abs/2604.02781
