Cultural Connections
Released 2021
Using the latest tools in AI workflows, this project aims to rethink what is possible with animation. Using a team of AI artists, this project was shot traditionally, but integrated with AI style transfer.
2 minutes | AI Short Form
Directed By: Quinn Halleck
Executive Producer: Dustin Callif
Production Company: Tool
Story By: Shuang Hu
Starring: Shuang Hu
Young Shuang: Achsah Mimi Xu
Teen Shuang: Alaina Adachi
VFX Supervisor: Gavin Bedford
AI Artist: CoffeeVectors
Deforum Artist: Ariel Klevecz
Cinematography: Viken Kazandjian
Production Designer: Noah Dains
Gaffer: Dessie Coale
1st Assistant Camera: Max Tsarev
Line Producer: Matt Wassong
Production Assistant: Hayden Bushfield
Dog Wrangler: Adam Halleck
BTS Videography: Matt Block
The Concept
Director Quinn Halleck was asked to create a 1–2-minute demonstration featuring a story that leverages a pipeline blending AI and traditional filmmaking. The story was devised to feature the newest AI tools, focusing on style transfers and unconventional visuals. This required the development, shooting, and delivery of a complete animated project within a timeline of less than 8 weeks.
Why AI?
While the majority of this project was filmed in a very standard timeline, tools such as AnimatedDiff, Runway, and Midjourney allowed the team to expand the visual style further than any traditional production. Creatively, to demonstrate the "same life but different" split screen, these tools allowed for a unique animated pipeline that could style transfer matching footage, emphasizing the core story of the project.
Workflow
Pre-Production
The project began outlined into 6 principal scenes/sequences all associated with a specific age of the main character. After each scene was outlined, transitions were designed to take the audience from one scene to the next.
With the film conceptualized, Quinn and VFX Supervisor Gavin Bedford outlined early-on how they anticipated being able to capture each moment. With that, they would design a pipeline to use the latest AI tools available at the time. Based on the shot list, they created a grid of style references for both sides of the split screen (modern vs traditional Chinese) and different styles of artwork associated with each era as Shuang grows older in both worlds.
Principle Photography
All the photography was shot practically on a green screen with tracking markers. Both sides of Shuang were shot independently of each other and later composited together to give the illusion of a twin living on the opposite side of the screen. It was important to choose sharp and undistorted lenses to allow for the AI to interpret the footage as cleanly as possible.
Based on the references, scenes were lit as closely to the storyboards as possible (e.g., hard backlight if a window was anticipated). AI interpreted the lighting we provided for our actor physically and helped
further marry it to our world.
This was particularly interesting when Shu reaches across the kitchen counter to steal the salt/soy sauce as a physical lighting gag had to be created as she entered the other world. Later, the two worlds were
composited together, giving the illusion of her reaching across the split screen.
Early on, Quinn and Gaving decided that even with style transfers, they could not change the shape of the actor to be "young Shu," so the team casted two younger actors (6 and 13 year olds).
Wardrobe was purchased for the main actor to differentiate between the different worlds. It was predicted that background actors' wardrobe could be changed within a style-transfer workflow; however, it was underestimated how much R&D would be required to create a convincing/non-artifact result.
VFX
The green screen footage was keyed, rotoscoped, and separated into layers. Each layer was then composited into a 3D environment based on the storyboards. Initially, this "slap comp" was to suffice for running a style transfer – but due to the split-screen setup and differing worlds, achieving consistent style transfer for the Midjourney background necessitated compositing both 2D and 3D elements.
The Style Transfer process was executed using a combination of ControlNets, Text Prompts, and AI Checkpoints & LoRAs. Experimentation with a wide range of settings and combinations was required for each shot.
After creating the image sequences, additional post-processing steps such as AI upscaling and color grading were applied to add the finishing touches.