Thanks to models like Sora, Dream Machine, Veo, and Kling, creating videos from text prompts is becoming more straightforward. However, many of these tools have a major drawback: they can’t generate sound, leaving us with silent videos. But Google DeepMind is stepping up to tackle this issue with its latest innovation: a new AI model that can create soundtracks and dialogue for videos.
In a recent blog post, Google DeepMind introduced V2A (Video-to-Audio), an exciting AI model that merges video visual cues with text prompts to create rich sound and audio. This new technology aims to transform how we produce and experience AI-generated videos by adding dramatic music, realistic sound effects, and matching dialogue.
V2A is meticulously designed to seamlessly integrate with Veo, Google’s text-to-video model that was unveiled at Google I/O 2024. This powerful combination empowers users to elevate their videos not only visually but also acoustically. V2A can breathe life into a wide range of content, from contemporary Veo videos to silent films and vintage archival footage.
One of the most impressive aspects of V2A is its versatility. It can generate an infinite number of soundtracks for any video, and users have the freedom to fine-tune the audio output with ‘positive prompts’ and ’negative prompts’ to achieve the perfect sound. Moreover, every piece of audio generated is uniquely watermarked with SynthID technology, ensuring its authenticity and originality.
V2A uses a diffusion model trained on a mix of sounds, dialogue transcripts, and videos. While the model is powerful, it wasn’t trained on many videos, so sometimes the audio might be slightly off. Because of this, and to prevent potential misuse, Google isn’t planning to release V2A to the public anytime soon.
Google DeepMind’s introduction of V2A is a significant step forward in video creation technology. V2A fills a crucial gap by adding sound and dialogue, making videos more immersive and engaging. Although it’s still in the works and not yet available for public use, V2A shows incredible promise for the future of video production.
