How to Use Generative Audio | Runway Academy
TLDRIn this Runway Academy tutorial, viewers learn to harness the power of generative audio, including text-to-speech, custom voice models, and lip sync videos. The process begins with writing text and converting it into spoken audio, choosing from a list of voices. Users can then train their own voice models with clean audio clips. Finally, the tutorial demonstrates creating lip sync videos with either generated or uploaded audio, offering tips to enhance the video workflow and suggesting joining the community on Discord for further exploration.
Takeaways
- 🎙️ Generative audio includes text-to-speech, custom voice models, and creating lip-sync videos.
- 💻 Access the generative audio tool from the Runway dashboard and type in your desired text to convert it into spoken audio.
- 🔊 Preview and select a voice from the default list before generating the audio file.
- ⏳ Audio generation time varies based on script length but is typically quick.
- 📂 Audio files are automatically saved in the 'generative audio' folder within the main assets folder.
- 👤 Train a custom voice model with a few minutes of clean audio, either imported or recorded in Runway.
- 📝 Ensure the audio for custom voice models is as clear as possible for optimal results.
- 📸 Create lip-sync videos using an image or video of a person with a full face visible.
- 🔄 Lip-sync can be applied to text-to-speech, recorded, or uploaded audio.
- 🎞️ If the audio is longer than the video, the video will loop back to the beginning once it ends.
- 🎨 For video workflows, avoid camera motion parameters and use subject motion with a motion brush to reduce the reversing effect.
- 📚 The tutorial concludes with a call to action to engage with the community on Discord for further resources and support.
Q & A
What is the main topic of the Runway Academy video?
-The main topic of the Runway Academy video is generative audio, which includes text to speech, custom voice models, and creating lip sync videos using Runway.
How do you access the generative audio tool in Runway?
-You can access the generative audio tool from your Runway dashboard by clicking on it at the top.
What is the process after typing in the text for text-to-speech conversion?
-After typing in the text, you can preview it, choose a voice from the default voice list, and then click on the generate button to convert the text into a spoken audio file.
How long does it usually take for the audio generation to complete?
-The generation times depend on the total script length, but they usually go pretty quickly.
Where are the audio generations saved by default in Runway?
-By default, audio generations are automatically saved to the generative audio folder inside your main assets folder in Runway.
What is required to train a custom voice model in Runway?
-To train a custom voice model, you need a few minutes of clean audio which can be imported into Runway or recorded directly within the generative audio tool.
What should be considered when recording audio for a custom voice model?
-The audio should be as clean as possible, meaning it should be free from background noise and clear in pronunciation.
How can you create a lip sync video in Runway?
-To create a lip sync video, you need an image or video of a person with their full face viewable. You can then add the generated or uploaded audio and adjust the lip sync accordingly.
What happens if the audio is longer than the video in a lip sync project?
-If the audio is longer than the video, once the video reaches the end of its duration, it will reverse and go back to the beginning for the duration of the audio.
What is a pro tip for using the video workflow in Runway to avoid noticeable reversing effects?
-A pro tip is to avoid using camera motion parameters and instead add subject motion with a motion brush, which makes the reversing effect much less noticeable.
How can viewers find more information or ask questions about using Runway?
-Viewers can join the community on Discord for more information and experimentation using Runway, or they can use the dashboard button at any time to find specific answers to their questions.
Outlines
🎙️ Introduction to Generative Audio in Runway Academy
This paragraph introduces the topic of the video, which is generative audio, encompassing text-to-speech, custom voice models, and creating lip-sync videos using Runway. The speaker guides viewers on how to access the generative audio tool from the Runway dashboard, input text to be converted into spoken audio, and choose from a list of default voices. The process of generating audio is described, including the saving of files to the generative audio folder within the main assets folder, and the option to save elsewhere if desired.
🎤 Training a Custom Voice Model in Runway
The speaker explains how to train a custom voice model using a few minutes of clean audio, which can be imported or recorded within the generative audio tool. The importance of clean audio for training is emphasized, and viewers are instructed to name their voice model once the recording is complete. The process is quick, and the custom voice model is ready to be used with text-to-speech functionality.
🎥 Creating Lip-sync Videos with Runway's Generative Audio Tool
This section covers the creation of lip-sync videos using an image or video of a person with a full face viewable in the frame. The speaker suggests using either custom media or preset characters provided by Runway. The process involves adding generated or recorded audio to the chosen media and selecting a voice for the lip-sync effect. A note on the video workflow is provided, advising against using camera motion parameters to avoid a noticeable reversing effect when the audio is longer than the video.
📚 Conclusion and Additional Resources for Generative Audio
The speaker concludes the video with a summary of the generative audio tool's capabilities and expresses appreciation for the viewers' time. They encourage viewers to join the community on Discord for more information and experimentation with Runway, or to use the dashboard button for specific questions. The video ends with a reminder of the work to be done and a thank you note to the audience.
Mindmap
Keywords
💡Generative Audio
💡Text-to-Speech
💡Custom Voice Models
💡Lip Sync Videos
💡Runway Dashboard
💡Generate Button
💡Assets Folder
💡Motion Brush
💡Discord Community
💡Gen 2
Highlights
Introduction to generative audio in Runway Academy.
Generative audio includes text to speech, custom voice models, and creating lip sync videos.
Access the generative audio tool from the Runway dashboard.
Type in text and convert it into a spoken audio file.
Preview and select a voice from the default voice list.
Generation times vary based on script length but are usually quick.
Audio generations are automatically saved to the generative audio folder.
Train a custom voice model with a few minutes of clean audio.
Use the generative audio tool to record or import clean audio for the custom voice model.
Name your voice model and it will be ready to use with text to speech.
Create a lip sync video with an image or video of a person.
Ensure the full face is viewable within the frame for lip sync.
Upload your own media or use preset characters for lip sync.
Lip sync can be used with text to speech, recorded, or uploaded audio.
Add text to speech and choose a voice for lip sync generation.
Convert an image into a video using Gen 2 for video lip sync.
Note the reversing effect if audio is longer than the video.
Avoid camera motion parameters for a less noticeable reversing effect.
Join the Runway community on Discord for more resources and experimentation.
Use the dashboard button for specific answers to questions about Runway.