Sorry, I don't understand your search. ×
Back to Search Start Over

Accompanied Singing Voice Synthesis with Fully Text-controlled Melody

Authors :
Li, Ruiqi
Hong, Zhiqing
Wang, Yongqi
Zhang, Lichao
Huang, Rongjie
Zheng, Siqi
Zhao, Zhou
Publication Year :
2024

Abstract

Text-to-song (TTSong) is a music generation task that synthesizes accompanied singing voices. Current TTSong methods, inherited from singing voice synthesis (SVS), require melody-related information that can sometimes be impractical, such as music scores or MIDI sequences. We present MelodyLM, the first TTSong model that generates high-quality song pieces with fully text-controlled melodies, achieving minimal user requirements and maximum control flexibility. MelodyLM explicitly models MIDI as the intermediate melody-related feature and sequentially generates vocal tracks in a language model manner, conditioned on textual and vocal prompts. The accompaniment music is subsequently synthesized by a latent diffusion model with hybrid conditioning for temporal alignment. With minimal requirements, users only need to input lyrics and a reference voice to synthesize a song sample. For full control, just input textual prompts or even directly input MIDI. Experimental results indicate that MelodyLM achieves superior performance in terms of both objective and subjective metrics. Audio samples are available at https://melodylm666.github.io.<br />Comment: Working in progress

Details

Database :
arXiv
Publication Type :
Report
Accession number :
edsarx.2407.02049
Document Type :
Working Paper