Martin Scorsese directed the film adaption of the book "I Heard You Paint Houses" by Charles Brandt. Netflix released The Irishman in November. It powered out of the gate with film festival awards and a 96 percent approval rating from critics compiled by Rotten Tomatoes.
Three septuagenarian megastars led the cast and played characters at various younger ages. Robert DeNiro, who was 76 when filming, plays Sheeran from ages 20 to 80. Al Pacino, 78, plays Jimmy Hoffa from 37 to his disappearance in 1975; and Joe Pesci, 76, played Bufalino from ages 47 to 72.
ILM created the youthful characters. Pablo Helman was visual effects supervisor, with Leandro Estebecorena, Nelson Sepulveda, and Ivan Busquets as associate visual effects supervisors. Artists in San Francisco and Vancouver worked on the show.
This is not the first time ILM has created a digital lead actor - the studio won an Oscar for turning Bill Nighy into a half-dead pirate for Pirates of the Caribbean: Dead Man's Chest in 2006. Ten years later
, they brought actor Peter Cushing back to life through a digital character that plays Tarkin for a few shots in
Rogue One: A Star Wars Story
.
Now, for The Irishman, the crew created younger versions of the characters played by DeNiro, Pacino, and Pesci who appear throughout the entire film. To do so, the studio developed a new system called Flux. Stephane Grabli led the R&D Flux team.
"We could have captured them using head cameras and dots on their faces," Helman says. "But when I met with Martin, he said, 'No head cams. No volume. I want them to be on set with theatrical lighting. You figure it out.' "
They did just that. And more. No head cams. No volume. No special lighting. And, no keyframe animation.
No Head Cams, No Markers
"The idea was to capture the most amount of information we could without markers," Helman says. "And, if there were no markers, the software we would develop would need to derive everything from the light and textures captured on set. So, we came up with a rig that used infrared cameras and didn't stop Marty [Scorsese] from doing anything. We worked closely with Director of Photography Rodrigo Prieto and Arri Los Angeles."
The rig has three cameras placed side by side on a 30-inch bar, narrow enough to fit through a door. In the center is the RGB camera, and on either side are two infrared film-grade Arri Alexa Minis.
"We needed to neutralize the light without changing the lighting on the set," Helman says, explaining the need for the infrared cameras. "In effect, rather than taking the actor into a controlled environment, we created a controlled environment on the set."
No other "witness" cameras were needed. The infrared light didn't interfere with the theatrical lighting on set and produced images without shadows. The actors and director didn't see it.
The actors could sit at a table in a crowded, busy restaurant and lean toward each other to talk. Scorsese could film them in close-ups, and as he did, ILM captured their facial expressions using the three cameras on that one rig.
Two camera operators controlled the cameras remotely. One operator managed the main camera. Another operator controlled the infrared cameras, which have a different depth of field.
No Keyframe Animation
Then, the magic happened. Helman describes the process used to create DeNiro's more youthful Sheeran from footage and data captured from the three cameras.
"Once I got the take, I brought the footage from the three cameras here to ILM," Helman says. "We also had the data gathered on set: HDRIs for light and density, and Lidar data to know where all the lights were. The footage went through layout to solve the camera [determine the camera view], and we did matchimation for the bodies and heads. All that data - the layout, roto, HDRI, Lidar - went into Flux with information from the three cameras. The software made a cocktail of it. Flux figures out where the actor is in 3D space and derives geometry from the three cameras to create a digital double of the actor."
Flux produces an albedo model showing a representation of light and textures and a plastic shaded render. The software then compares its digital double to a model built of the actor and deforms the model on a per-frame basis to match the actor's performance.
"Then, we retarget this performance to a younger version of the model and render it through lighting and texture," Helman says. "We had no keyframe animation in this project at all. We didn't want to change the performance."
Models and Textures
A team led by Digital Model Supervisor Paul Giacoppo sculpted contemporary and youthful models of each actor, changing the geometry in the chin and neck as needed.
"Each model started as an accurate scan of the actor at his current age using a combination of [Disney Research's] Medusa for likenesses and facial expressions and Otoy for facial detail," Giacoppo says. "Then, by looking at past films, we sculpted younger faces. We had a slider that could take us from current ages to previous ages."
The models provided the form and larger--scale bumps and pores. Texture artists led by supervisor Jean Bolte added the finer details, working from the Otoy scans and photographic reference.
"We de-aged them in stages," Bolte says. "Each stage had to have wrinkles and age spots painted out judiciously as we figured out how much to take away. I'd look pixel by pixel, zooming in to make sure of the integrity. We didn't want them to be too pretty. We wanted to keep things a makeup artist might have taken out. I was well aware that we could have ruined the movie if we didn't nail it."
She smiles and says, "I think we pretty much nailed it."
For what Helman calls a "sanity check," the crew spent two years gathering a library of performances for the three actors at the targeted ages from different movies. An AI-based program they've dubbed Face Finder found frames to match rendered frames in terms of age, expression, pose, camera angle, and lighting.
But, they weren't aiming to exactly match the actors at those ages.
"Martin said he didn't want us to take DeNiro from Taxi," Giacoppo says. "He had to be the younger self of the character he was playing in this movie. A young Frank Sheeran."
Adds Bolte, "We didn't have a clear goal. Not only do these actors appear different from one film to another, they're different even from one shot to another. It was a matter of, who is this character Frank Sheeran? We had to find that. I spent days studying images of DeNiro. He can change his expression with the raise of an eyebrow. He is a master at being a chameleon."
All told, the crew spent nine post-production months working on the 1,750 shots, but that was after they'd spent four years making the post-production possible.
"We knew the risk we were taking," Helman says. "So, we invested those four years of development to have something completely performance-driven. If you follow the natural progression of visual effects, you find that the next thing is markerless performance capture.
"I remember Ewan McGregor's reaction when he walked onto a bluescreen stage," Helman continues. "He said, 'What the [expletive] is this?' I thought an actor could imagine the set. But, after working with these actors on The Irishman, I realized what he meant. It's crucial for actors to be where they're meant to be."