About

In late 2022, I had the pleasure of working on a visualizer video for the rock band ANTITYPE.
Besides aesthetic drone shots, clips of the band members and stock footage, due to my great interest in AI technologies, I also had the unique opportunity to produce one of the first (if not the very first) music videos with AI-VFX (see sidebar).

How I created the AI-Effect:

For the lyric “With a keepsake in my hand To remind me what I’d had […] When all I depend on has faded away” I really wanted to have a video clip that clarified what the “lyrical I” had lost.

Since on stock footage sites nothing really fit 100% going out to film myself was out of scope, I decided to use Cutting Edge AI effects to make non-fitting clips fit.

Since summer 2022 I’ve already been experimenting with Stable Diffusion as well as custom trained AI models.

After a short period of experimentation with Dall-E 2 and Midjourney, I decided to go this route because the open source nature of this AI allows for much more customization and creative expression.

For this video, I used my own Stable Diffusion Instance on my own computer.

I searched stock video sites for a basic clip that roughly fit the text passage.
It showed a Young Woman happily holding her partner’s hand, symbolizing the Happy Times in the past.
However, since those times were obviously long gone and the memory of them was slowly fading, I wanted to show this in my video as well.

I wanted the woman’s face to slowly change and then take on many different identities before she quickly begins to age until only a skeleton remains and it all fades away.

To achieve this, I exported the individual frames of the stock footage clip and fed them into my Stable Diffusion DeForum script, which allows AI input prompts to be set for individual frame checkpoints in the video and interpolates between them using custom mathematical formulas.

Here is an example how such a prompt matrix could look like:

"0": "(happy young asian woman holding her lover's hand :1.0), movie, jj abrams, dramatic, cinematic, film, movie, hollywood, detailed, textured --neg ((outdoors)), male, illustration, drawing, painting, 3d, cgi",
"16": "caucasian woman extremely old, movie, jj abrams, dramatic, cinematic, film, movie, detailed, textured - --neg ((outdoors)),,male, illustration, drawing, painting, 3d, cgi",
"45": "heavy creepy (((glitch))) in hear face, movie, jj abrams, dramatic, cinematic, film, movie, detailed, textured - --neg ((outdoors)), male, illustration, drawing, painting, 3d, cgi",
"60": "her skin tears apart, movie, jj abrams, dramatic, cinematic, film, movie, detailed, textured - --neg ((outdoors)),, male, illustration, drawing, painting, 3d, cgi",
"70": "sadness, face melting off, revealing bones of her body, creepy, scary, jj abrams, dramatic, cinematic, film, movie, detailed, textured, ((normal background)) - --neg ((outdoors)), male, illustration, drawing, painting, 3d, cgi",
"80": "grotesque skeleton woman holding hands of a man, jj abrams, dramatic, cinematic, film, movie, detailed, textured - --neg ((outdoors)), male, illustration, drawing, painting, 3d, cgi"

I experimented with it for a few days, tried many different prompts and formulas to make the flow as clean as possible and created a total of 35 different versions of the clip.

I imported these into After effects for my final version of the effect, combined them with a variety of keys and masks to pick out the best frames from each version, and finally color graded them to match the rest of the video.

Other than the Intro/outro scene, this was the most complex part of the video, which took a ton of hours (also because this is a veeerrryyy resource-heavy process that takes a lot of time to process – even on modern GPUs), but which I think is completely justified by the result.

And I learned a lot too!

Used Software/Tech

• Stable Diffusion
• AI Video Upscaling
• Premiere Pro
• After Effects
• Photoshop
• Illustrator
• Cinema 4D
• Octane
• Mocha Pro

[More] Photos