Instructions for creating an inspiring AI music video with Stable Diffusion

In this tutorial, I will show you how to create an impressive AI music video using Stable Diffusion and Deforum. The tutorial will guide you through all the necessary steps, from selecting a music piece to creating the final video. With these tools and a little creativity, you can create your own artworks that are both artistically and technically appealing.

Main Insights

• You need Stable Diffusion and an Audio to Keyframe String Generator to play your music synchronously with the visual content.
• The selection of the music piece and extracting the relevant audio tracks are crucial to control the visual representation.
• Adjusting the parameters in the Colab notebook enables you to control and refine the artistic aspects of your animations.

Step-by-Step Guide

To create your AI music video, follow these steps:

Step 1: Select Music Start by selecting a suitable music piece. I chose the song "As You Fade Away" by Nefex, which is available in the YouTube Audio Library. Download the song to start the creation process.

Step 2: Edit Audio After downloading the song, you can go to Phonic Mind. There, you have the option to extract audio tracks. In my case, I extract the drums as they are best suited for visual synchronization. Upload the song to Phonic Mind and select the tracks you want to use.

Step 3: Create Keyframe Once you have extracted the relevant drums, you need to upload the audio file to an Audio to Keyframe String Generator. This allows you to generate a keyframe that we will later use in Stable Diffusion. Be sure to upload only the drum sounds.

Step 4: Set Keyframe Parameters Now you need to set the framerate. I recommend a framerate of 12 frames per second as this simplifies animating the frames. Also, consider which mathematical formulas you want to use. Zoom is a particularly important factor here. I start with a zoom of 1.04, which works well for the drum beats.

Step 5: Copy Keyframe String Copy the entire keyframe string as it is needed in Stable Diffusion to create the animation. Paste the string into the Colab notebook of Stable Diffusion.

Step 6: Configure Colab Notebook In the Colab notebook, you can leave the default settings but you should adjust the Translation value for the animation. Ensure that the zoom ratio is correct and the animation looks as desired.

Step 7: Prompt for Animation The choice of prompts is crucial for the visual theme of your animation. I opted for a Cyberpunk-style theme. Search for suitable prompts in Lexica, copy them, and paste them into your Colab notebook.

Step 8: Video Output Settings Ensure that the frames per second are set to 12 as this should match with the previous settings. Now you can run all cells to generate the animation.

Step 9: Generate Animation Let the Colab notebook run all cells. It will take about 20 minutes to create the animation. Make sure to grant access to your Google Drive as all frame files will be stored there.

Step 10: Video Creation Once all frames have been created, you can convert the animation into a video. Uncheck the "Skip video for run all" box and click Play to merge the frames together.

Step 11: Edit the video Download the video and open it in your preferred editing program. Here you can cut the video together with the original audio file to obtain the finalized music video.

Summary

In this guide, I have explained step by step how to create an AI music video with Stable Diffusion. Starting from music selection, editing the audio tracks, up to generating and editing the video. With these tools and techniques, you can create impressive visual and auditory experiences that give your creative expression a new dimension.