Abstract
We propose VS-Singer, a vision-guided model that generate stereo singing voices with room reverberation from scene images. VS-Singer consists of a modal interaction network, a decoder based on consistency Schrödinger bridge (CSB) and a spatially-aware feature enhancement module (SFE). Specifically, the modal interaction network fuses spatial features into text encoding to produce a linguistic representation with spatial information. Subsequently, the decoder utilizes the CSB to establish a tractable diffusion bridge between the representation and binaural mel-spectrograms, enabling it to generate samples in one step. Additionally, we employ the SFE to accelerate model convergence. To the best of our knowledge, this work is the first to integrate stereo singing voice synthesis with visual acoustic matching into a unified framework. Experimental results show that VS-Singer can efficiently generate stereo singing voices that match the scene perspective in one step. Particularly, our model achieves 2x faster inference speed compared to cascaded systems.
