Wild AI Video Workflow with Viggle, Kaiber, Leonardo, Midjourney, Gen-2, and MORE!
TLDRIn this video, Tim shares an innovative AI-based film-making workflow that spans from pre-production to short film generation. Inspired by the 2016 film Rogue One, Tim discusses the use of AI tools to create a hybrid storyboard animatic animation. He demonstrates the process using references from Gladiator, John Carter of Mars, and Warhammer 40K, and highlights the capabilities and limitations of tools like Vigle, Midjourney, and Kyber. The result is a visually engaging, if somewhat imperfect, AI-augmented short film that offers potential for filmmakers, especially in pre-production stages.
Takeaways
- 🎬 The speaker shares an AI filmmaking workflow with potential applications from pre-production to generating short films.
- 🚀 The inspiration comes from the 2016 film Rogue One, specifically an interview with the editor about creating a feature-length story reel without a finished script.
- 🧠 The idea is to use AI tools to create a hybrid storyboard, animatic, and animation, as demonstrated with a scene from the movie Gladiator.
- 📏 The process starts by clipping reference footage and using Vigle's 2.0 update for initial video creation, which includes dancing and other features.
- 🌟 The character model is created using Midjourney, with a focus on full-body images to ensure accurate representation.
- 📹 Vigle's output may require refinement; it doesn't respond well to camera movement and can produce shaky results.
- 👤 Leonardo is used to enhance character shots with additional details, such as hand gestures, by using image-to-image references.
- 🎥 Kyber is introduced as a powerful AI video generator, especially with its new Motion 3.0 feature for stylized and unique outputs.
- 🖼️ Backgrounds are added using a combination of Gen 2 and Kyber, aiming for a cohesive look with the character.
- 🎨 Video editing software like Premiere or DaVinci is used for compositing character and background, with adjustments for a more cinematic feel.
- 🎶 Audio elements, such as crowd chanting, are generated using AI tools like audiogen, and soundtracks are composed using software like Ableton.
Q & A
What is the main focus of the video?
-The main focus of the video is to share an AI film making workflow that covers pre-production to generating short films, using a combination of various AI tools.
What historical significance does the speaker mention about the 2016 film Rogue One?
-The speaker mentions that Rogue One holds historical significance as it features the first major film with a fully deep faked character.
How did Colin Ghoul contribute to the editing of Rogue One?
-Colin Ghoul created a feature-length story reel before the script was finished, using scenes from hundreds of movies to help determine the dialogue needed in the film.
What AI tool does the speaker use to generate a model for the main character?
-The speaker uses Midjourney to create a model for the main character.
What issues did the speaker encounter with Vigle's output?
-The speaker found that Vigle's output was a bit stuttery and messy, particularly with camera movement, and it didn't respond well to certain prompts like the 'God Emperor' shot.
How does the speaker address the issues found with Vigle's output?
-The speaker addresses the issues by using additional AI tools like Leonardo and Kyber, and by refining the prompts and image references to achieve better results.
What is the role of Kyber in the workflow?
-Kyber is used to further refine the output from Vigle, adding a layer of stylization and consistency to the character's appearance, and to generate a cohesive look for the background.
How does the speaker handle the background and character compositing?
-The speaker composites the background and character by using a video editor like Premiere or DaVinci, adjusting settings like chroma key, choke, soften, and contrast to blend them together seamlessly.
What tools does the speaker use to generate the crowd chanting audio?
-The speaker uses a free site called audiogen to generate the crowd chanting audio with the prompt 'crowd chanting in an arena'.
What was the final result of the AI film making workflow?
-The final result was a short film that, while not perfect, demonstrated the potential of using AI tools in film making for pre-production and generating watchable short films.
What additional advice does the speaker give for those interested in this workflow?
-The speaker advises that this method is more useful and productive for pre-production or small indie films and encourages those interested to explore incorporating other tools into the kit-bashing technique, with more workflow videos coming soon.
Outlines
🎬 AI Filmmaking Workflow Introduction
The speaker introduces an AI-based filmmaking workflow that has potential from pre-production to generating short films. Inspired by Gareth Edwards' 2016 film 'Rogue One', the speaker discusses the use of AI tools to create a hybrid storyboard animatic animation. The process involves using AI to synthesize scenes, as demonstrated by the 'Are you not entertained?' scene from 'Gladiator', with elements from 'John Carter' and 'Warhammer 40K'. The speaker plans to share their learnings and experiences to potentially save time for others interested in trying out this workflow.
🌟 Utilizing AI Tools for Character Creation and Scene Development
The speaker details the process of using AI tools like Vigle, Midjourney, and Leonardo for character creation and scene development. They discuss the challenges of using AI for full-body images and the importance of having a green screen background. The speaker also shares their experience with Vigle's 2.0 update and its limitations, particularly with camera movement. They describe the process of refining the AI-generated footage by using additional AI tools like Kyber, which offers a unique video generation capability, and the steps taken to improve the character's appearance and animation quality.
🎞️ Enhancing AI-Generated Content with Backgrounds and Special Effects
The speaker explains the process of enhancing the AI-generated content with dynamic backgrounds and special effects. They describe using Gen 2 to create movement in the background and Kyber to give the scene a cohesive look. The speaker emphasizes the importance of using a video editor like Premiere for compositing the character and background, and the techniques used to blend them seamlessly. They also discuss the addition of cinematic elements like black bars and crowd chanting to enhance the overall atmosphere of the scene.
🎧 Audio and Narration for AI-Generated Films
The speaker addresses the challenges of adding dialogue and sound to AI-generated films. They share their experience with text-to-speech tools and the process of finding suitable audio sources for the project. The speaker discusses using 'Typcast' and the 'Frankenstein' model to generate dialogue that fits the scene, despite the limitations of the AI-generated voice. They also mention creating a soundtrack using Ableton and loops, highlighting the importance of audio in completing the film's atmosphere.
Mindmap
Keywords
💡AI film making workflow
💡Deepfake
💡Hybrid storyboard animatic animation
💡Vigle
💡Mid journey
💡Leonardo
💡Kyber
💡Background comp
💡Audio generation
💡Text-to-speech
Highlights
AI film making workflow shared, covering pre-production to generating short films.
Inspiration from 2016 film Rogue One, using AI tools to create a hybrid storyboard animatic animation.
Colin Ghoul's 2017 interview about creating a feature-length story reel before Rogue One's script was finished.
Using AI to synthesize scenes similar to the 'Are you not entertained?' moment from Gladiator.
Clipping out reference footage and using Vigle 2.0 for initial video generation.
Vigle's new 2.0 update and its capabilities for dancing and other movements.
Creating a model for the main character using Midjourney and incorporating elements from Star Wars and other films.
Challenges with camera movement and stuttery footage in Vigle's output.
Utilizing Leonardo for image to image reference and improving the character's appearance.
Using Kyber's motion 3.0 feature for a more stylized and cohesive character animation.
Comping character and background in a video editor like Premiere or DaVinci for final touches.
Adding crowd chanting audio from a free site like audiogenen to enhance the atmosphere.
Generating dialogue using text-to-speech models, with the Frankenstein model from typcast.
Creating a soundtrack with Ableton using loops for a quick 20-second cue.
The method's potential for pre-production and its usefulness for both large-scale and indie filmmakers.
The workflow's ability to save time and offer an engaging, though not perfect, short film outcome.