-
- Google Research presents its AI SingSong in a scientific publication
- From the recording of a singer, it generates the music that accompanies it
- The demonstrations are impressive
An artificial intelligence that composes music automatically already exists. But researchers at Google Research went even further, creating an AI capable of generating this music to accompany a person singing (without music) in an audio recording. And one day, a budding artist might be able to write their song, record themselves on a smartphone, and get music from AI that matches that song. The technology is not yet accessible to the general public. But completely stunning demonstrations are already available online.
Transformers can map one mp3 to another mp3. But why do we want to do audio-space translation?
SingSong from @GoogleMagenta is a super clever application: singing voice -> instrument accompaniments, ie “Reverse Karaoke”!
Now everyone’s got their private, deluxe band ?
1/? pic.twitter.com/FXlAaTsT7r
—Jim Fan (@DrJimFan) February 1, 2023
Google Research recently presented this new AI in a scientific article. Called SingSong, it was created using a dataset of 1 million songs. Using an existing artificial intelligence, the researchers separated the music and the voice on these tracks. And it was these separate tracks that were used to drive their creation. To create SingSong, they also relied on AudioLM, another AI that generates music.
Generate audio with audio
To create SingSong, AudioLM was adapted so that it could generate audio from an audio file. And as mentioned above, the AI was then trained with a large database on which the singer’s voice and the instrumental are separated. It is, in essence, in this way that Google Research managed to create an AI capable of composing music for a singer who would not have musicians.
And if you’re wondering if these compositions are relevant to the vocal track, Google Research believes they are. Indeed, after creating the technology, he conducted a study. And according to the scientific publication: “In a pairwise comparison with the same vocal inputs, listeners expressed a significant preference for SingSong-generated instrumentals over those from a solid recovery base.” Many examples are available on the source of this article.
Ethical issues
“While music generation technology presents an opportunity to increase access to creative participation in music, it also presents significant risks”, reads the Google Research article. He is indeed aware of the fact that this technology could bring major upheavals in the industry. On the other hand, the researchers also mention the fact that currently, the user would have no control over the generated music. But these could improve the technology by ensuring that it takes into account other elements, such as gender, the style of the singer, or even the text.
And if you don’t aspire to be a professional singer, there’s still a chance that you could one day use this technology, so you can have fun. “We are also interested in the potential of SingSong to accompany and allow anyone to make music with their voice. We explore this here with singing samples from the Vocadito dataset which contains samples from novice singers recorded on consumer electronics.”, can we read on the Google site.
