Matter of Time

By Karen Moltenbrey

If we have learned anything over the past decade since the characters from the Marvel Cinematic Universe (MCU) began gracing the big screen, it's that each Avenger can easily carry his or her own film all the way to resounding box-office success. But, when they band together, it's a total game changer - and none more so than the recent Avengers: Endgame, which toppled box-office records even before it was released. As of press time, its success continues, poised to surpass Avatar as the top-grossing movie of all time - this after just two months' time.

In fact, of the top 10 highest-grossing movies, half are Avenger-related films - and only one of those five centers on a single Avenger (Black Panther), while the other four movies feature a team - proving their power in numbers.

With more Avengers on screen come more visual effects scenes. And in Endgame, the 22nd film in the MCU, there are many - nearly 2,500 out of roughly 2,700 shots in the film contain VFX. In comparison, Avengers: Infinity War had approximately 2,700, and since the two films were shot back-to-back, many of the studios on Infinity War continued their work on Endgame, a direct sequel, as did Directors Joe and Anthony Russo. According to Dan DeLeeuw, Endgame's visual effects supervisor, 13 vendors worked on this show, with Weta Digital and Industrial Light & Magic generating the larger share (494 and 550, respectively), along with Digital Domain and Framestore assuming a number of shots as well, while DNeg, Cinesite, Cantina Creative, RISE, Lola, ScanlineVFX, Capital T, Exceptional Minds, and Perception rounded out the list.

(Images ©2019 Marvel Studios.)

Despite Endgame being a continuation of Infinity War, the facilities did not rest on their laurels, with a number of advancements for the latest film, especially concerning work on Thanos, Smart Hulk, and throughout the final battle.

"Interestingly, Endgame has about 200 fewer VFX shots than Infinity War, but the actual complexity and length of the shots far exceed anything we've ever done," says DeLeeuw, who was also visual effects supervisor for Infinity War as well as Captain America: Civil War and Winter Soldier, and second-unit VFX supe on Iron Man 3. "A film like [ Endgame] is kind of your doctorate thesis. I couldn't have done Civil War without Winter Soldier. I couldn't have done Infinity War without Civil War. And, I couldn't have done Endgame without Infinity War. It was definitely a progression in terms of understanding the size, scope, and especially the density of the effects in Endgame."

Endgame picks up following the devastating events of Infinity War, with the universe destroyed after an injured Thanos manages to activate the Infinity Gauntlet, and half of all life across the universe disintegrates, including T'Challa, Groot, Mantis, Strange, and others. However, surviving Avengers unite to take back the Infinity Stones in order to reverse Thanos's destruction, but soon learn that Thanos has destroyed the stones. Five years pass, and Scott Lang (Ant-Man) escapes from the Quantum Realm, prompting the Avengers to ponder whether time travel is possible and leading Tony Stark (Iron Man) and Bruce Banner (Hulk) to successfully build such a machine in an attempt to resurrect those who Thanos had disintegrated.

ILM PUSHED THE ENVELOPE FOR THE DIGITAL CHARACTER SMART HULK, A COMBINATION OF BRAINS AND BRAWN.

Smart Hulk

 Avenger fans are used to seeing two sides of Hulk (Mark Ruffalo): the angry, green, muscular version and the scientist Bruce Banner. In Endgame, they see a whole new side - Smart Hulk, with the stature of the big green guy but with the intellect and demeanor of the scientist. "I think we pushed Smart Hulk further in terms of what we could do with a digital character," says DeLeeuw of the work, which was handled by ILM.

ILM has worked with Hulk a few times in the past, such as in Thor: Ragnarok. But this time, the single character had both brains and brawn. "We spent a lot of time trying to find the right balance of Ruffalo, but not too much, and just enough Hulk," says Russell Earl, ILM's VFX supervisor. "Our creature team of Lana Lan, model supervisor, and Smart Hulk lead modeler Sven Jensen worked to sculpt the perfect blend of Banner and Hulk." This required a new approach, giving the CG character more human-like qualities. So, between Infinity War and Endgame, both ILM and Disney Research Studio rewrote many of the programs the teams of artists used for creating the CG movie character from the actor's performance.

ILM began with scans from Disney Research Studio's Medusa Performance Capture System, a mobile rig of eight cameras and lights coupled with proprietary software for reconstructing a high-resolution version of an actor's face in full motion, without the use of traditional motion-capture dots. Ruffalo (and Josh Brolin, as well, for the character Thanos) sat in front of the cameras and practiced facial shapes and performed dialog. That information was captured and used as the basis for building the underlying Banner mesh as well as for the basis of ILM's Hulk facial shape library.

Medusa tracks the pores of an actor's face, from which a photoreal 3D model is derived. However, the eight-camera solution requires the actor to sit in a studio environment. "We wanted to put Mark Ruffalo on the set with the other actors for a more intimate performance so they'd be able to act off each other," DeLeeuw says. "Sometimes when you have your actor in a motion-capture environment, it's very sterile and you don't get the same performance."

So, ILM began the solve using SNAP, what was then its current facial solver, which comprises two head-mounted cameras positioned in front of the actor's face, where tracking markers are placed, enabling the facial animation to be acquired on set. The trade-off is that a low-resolution mesh is used, which ILM then reapplied to the high-resolution mesh from the Medusa scan to drive the facial shapes. 

"We started there but felt like we needed a higher level of fidelity given Ruffalo's performance, the nuance of his face, and wanted to be sure we got as good a representation of that as we could," says Earl. "So, we started to improve the SNAP solve system by taking the meshes we were solving, and then basically comparing them back to the shapes we had captured in that initial Medusa session."

The results were better in this newer system, known as Xweave, but ILM learned about Disney Research Studio's Anyma, which up to this point was used as an ADR-style booth with three stationary cameras for recording performance dialog after the fact. "We had asked the guys at Disney Research if they were able to adapt the Anyma solver to work with our head-mounted camera footage that we had already shot," says Earl. It took some time, and it was already in the latter part of production, but they were able to make that happen.

Anyma doesn't just rely on the low-resolution mesh generated from the points; rather, it generates a mesh per frame and does a photometric solve based on the footage from the head-mounted cameras. "So, it looks at the actual pixels and the images to give you a much higher-fidelity solve," Earl explains.

While the performance solve was being completed with the new system, ILM was simultaneously rebuilding the re-targeting aspect, whereby Ruffalo's performance would be applied onto Smart Hulk. Spearheading this was Abs Jahromi, ILM's Endgame facial technology supervisor, with Owen Calouro, layout supervisor, initiating the process from the production shot side.

Once ILM reviewed the solves on a Banner mesh, the crew compared them back to plates of Ruffalo, making sure they had a one-to-one match. Then, using the system, called Blink, they re-targeted the Banner solve onto the Hulk model using new code that Jahromi wrote. At the same time, that re-target broke the per-mesh solve into Hulk facial shapes and provided animators a much more user-friendly version to work with.

"Typically, you would get that data and the solve, but it would be very difficult to handle or change the data," explains Earl. "In this instance, it was all translated into the underlying Hulk facial shapes, which had very friendly animation controls that our animators here in San Francisco, led by Kevin Martel, could then use to compare Ruffalo's performance to the performance on Smart Hulk and adjust it accordingly, dialing it up or down in intensity once it was on the target model."

When it came to animation, ILM used new deformers that were also more user friendly for dialing up or down the different aspects of Ruffalo's performance. Additionally, ILM provided its Hulk model, shader information, and base rigging to Framestore, which also worked on some Smart Hulk shots.

THANOS WAS UPDATED, REFLECTING MORE OF JOSH BROLIN’S PERFORMANCE.

Thanos

As DeLeeuw points out, Thanos, like Smart Hulk, was another character who was pushed further for this film, particularly for the end battle. "We pushed the level of detail in his face and the level of detail in his movements, capturing even more of Josh Brolin's performance," he says.

Whereas Hulk's performance is very broad and his face very elastic, Thanos's face is intense yet subdued. "Here you've got opposite ends of the spectrum, and you're dealing with a really precise performance," says DeLeeuw, comparing Thanos and Smart Hulk. "There's not a lot of movement in [Thanos's] face."

Nevertheless, the character's performance is crucial to the storytelling. "We need to understand his decision-making at an emotional level because it's his decision to come into the future and destroy the universe, and rebuild it from scratch, which motivates the entire third act," says Matt Aitken, VFX supervisor at Weta. "He had to work at an emotional level, so we did a lot of very finely crafted performance work to make that happen."

In the final shots, Thanos does not speak, but it is clear what he is thinking based on his computer-generated body language and facial expressions.

Early in the film, Digital Domain again handled shots of Thanos, as it had for Infinity War, with Weta taking control of the character when he attacks the Avengers' compound well into the film and creating a few hundred Thanos shots for Endgame, building on the work it also had done for Infinity War. "When we were working on him for Infinity War, we hit a little bit of a wall with some of the fine detail on his face, particularly around the corners of his mouth. We felt we didn't have the complete range of expression that we needed, so we had to patch some of those shots by hand," says Aitken.

In the short time between the two films, Weta continued to work on the character. According to Aitken, the facility's facial modeling team created some new target shapes around the edges of the mouth. The team took advantage of new developments in Weta's facial animation pipeline, using Deep Shapes, which enable the animators to procedurally add more fine-level detail to the facial performance.

"When Thanos's facial performance changes from one expression to the next, we're not changing either of those end points of that transition. They're still staying exactly the same. "That's important because we like our facial animators to have complete control over the shape of Thanos's face so we can control the facial performance at a very high level," says Aitken. "But, we wanted to add some complexity to that transition itself, so we're modeling a little bit of inertia so that where the tissue sits on top of the bone in the head, depending on the depth of that tissue, there's maybe some inertia, a little bit of fine-level detail, shape. It's not something you would necessarily consciously be aware of when you're looking at Thanos, but it's something we feel adds more natural movement to his facial performance and, therefore, makes him more believable."

The Thanos model, however, is pretty much the same as it was for Infinity War, though he is dressed differently, this time in armor. In addition, he is a bit younger, having come from four to five years in the past. "He's younger, more agile, and more powerful," Aitken says, requiring keyframe animators to dial in his movements on top of the motion captured from Brolin and the motion captured from a stunt performer whose actions are driving the character in the fight scenes during the end battle. However, Thanos is eight feet tall, and keyframe animators had to reflect that scale and mass within the captured motion.

"We wanted people to immediately recognize that this is the same Thanos from Infinity War, but also that he's subtly different," explains Aitken of the age differential.

ILM GAVE THE CG SMART HULK MORE HUMAN-LIKE QUALITIES THAN EVER BEFORE.

Past Selves

After Scott Lang (Ant-Man) escapes the Quantum Realm, where he was stuck for five hours, he learns that five years have actually passed, leading Smart Hulk and Tony Stark to uncover a way for the Avengers to time travel. Three teams are sent back to various time points to retrieve the Infinity Stones before Thanos grabs them and unleashes his destruction with his snap. For fans, the sequence re-introduces scenes from past films and memorable characters from the first Avengers to Guardians of the Galaxy Vol. 1.

"We went back and unarchived the original shots, but because we're at [a different aspect ratio now] compared to the first Avengers, we had to recompose those shots for Endgame," explains DeLeeuw. "Some of the work was seven years old and used older technology, so we had to take the original plates prior to the DI and convert them into our code base."

In one scene, Nebula and Rhodes travel back to Morag 2014 and steal the Power Stone before Peter Quill does so in Guardians; the current visual effects team re-created the set. During the scene with Quill dancing, the team began to intercut the shots with those from Guardians, timing it to the beats of the song to the point where they are back in their present day. A similar technique was used for the sequence in New York City, where Bruce Banner and his team head to collect their stone.

In another time-heist scene, Banner, Lang, Rogers, and Stark travel to NYC 2012. Banner retrieves the Time Stone, and Rogers the Mind Stone, but Stark and Lang have difficulty as Loki makes off with the Space Stone, which they later obtain by traveling back further in time. ILM was able to reuse a lot of the assets for Stark Tower and the penthouse from the first Avengers. The sequence in NYC is especially entertaining, as new Hulk meets his old self, thanks to the artists at ILM. "It was a combination of old and new, which was fun," says Earl. For the most part, the moments before and after the flashback in the sequence were reshot, and then the artists restored and upgraded the original assets.

Staging the Final Battle

Endgame is filled with many impressive visual effects sequences, none more than the end battle, which brings back a plethora of characters who join in the fight to defeat Thanos. After the Avengers return to present day with the Infinity Stones, the stones are inserted into a gauntlet created by Tony Stark, and Bruce Banner uses it to bring back all those who had disintegrated from Thanos's snap. But, the past version of Thanos arrives and attacks the compound with his warship. Soon the restored Avengers and other heroes arrive, and a battle of epic proportions ensues. Various heroes step in for the relay to carry the gauntlet to safety, with Thanos gaining control at one point, but not before Stark snatches the stones and uses them to disintegrate Thanos.

ENDGAME IS ALL ABOUT THE INFINITY STONES.

"Going back to Civil War, we had about a dozen heroes running at each other and fighting in the big battle scene. In this one, we had hundreds of heroes and villains, and every Avenger using his or her powers at the same time. The complexity of the shots went through the roof!" says DeLeeuw. "When we started the show and read the script, we thought, OK, there are a lot more character moments in the film and we wouldn't have quite as much to do in the early part. But, there were visual effects running through the entire beginning of the film. And then once we got to the final battle, there were showcase effects - and lots of them. It was kind of the kitchen sink, with everything in the kitchen sink."

The big challenge in the sequence comes from the sheer number of digital characters and blending them with the live-action characters. "It's something that really should have broken the back of the entire studio in terms of getting it out, but Weta Digital and ILM really pulled it together and got it done," says DeLeeuw.

The battle starts when the Avengers' compound is blown up. Weta built, and then destroyed, the CG environment, making sure set pieces could be moved to accommodate various camera angles. Weta also built digital doubles of all the characters and all the creatures - essentially a kit they used to create the battle. Then, the fighting breaks out within the crater that's now where the compound used to be.

A lot of the sequence was shot in Atlanta on a soundstage. "One of the biggest challenges was just trying to get the lighting to work in a way that was believable," DeLeeuw says. "And, no matter how big the stage was, it was never big enough to photograph all the action. Whenever we tried to put set dressing out, it would interfere with the sheer number of stunt characters we had to photograph. Imagine trying to have maybe 40 stunt people out there, spread out. There are sorcerers and Asgardians, all in costume, and between them all, you have to have your CG characters interacting as well."

As DeLeeuw points out, the Sakaarans from Guardians were actors, dressed in black armor. The Chitauri were CG, as were the Outriders from the first Avengers and Infinity War. "You are on the stage, and there's dirt on the floor, and set dressing, with stunt people swinging through the air. That's where we started the battle," he says. "Then the CG has to carry that all the way to what you actually see in the film, where the battlefield is populated and extends for miles beyond what you're actually photographing. You've got the fog of war and the light effects, and everything else, so you're starting with very little and adding to it. In some cases, making the shots all-CG might have been easier. You don't get the nice interactivity that you do with stunt people out there, but just being able to control the scene from the beginning sometimes would have been a more expedient route."

The final battle is filled with explosions, starting with the compound and continuing as Captain Marvel (ILM, to be exact) destroys Thanos's ship - all calling for large-scale simulations. "We wanted to push the envelope on that type of shot. In Winter Soldier, when the Helicarrier falls into the river - we wanted to push that look and those effects further," says DeLeeuw, noting the battle scene in Endgame was an extension of that as well as the airport battle in Civil War. "The water sims and explosion sims [in Endgame] are light-years ahead of where we were just six, seven years ago in terms of what we were able to do."

This time, however, there was little to no pre-production time to plan out the sequence, since this film was shot back-to-back with Infinity War. "Our pre-production time was occurring while we were shooting and also posting Infinity War, so we had a previs team going for Endgame and a postvis team going on Infinity War - around 60 people from The Third Floor working on both films at the same time," says DeLeeuw. "So, we took the final battle and just ran with it, always trying to give a little bit more."

Make no mistake, visual effects carried this major battle. And in the thick of it all were two vendors in particular: Weta and ILM. Weta's first shot in the film is in the third act when Thanos destroys the compound, and culminates when Stark succumbs to his injuries.

While the set was extensively dressed with a mixture of building rubble and burned-out tree stumps (the compound exists on the edge of the Hudson River surrounded by forest), when the sequences were cut together, the tree stumps played dominant, making it feel as if the battle was taking place in a bombed-out forest. So, Weta artists roto'd all the characters off the plates and replaced the set environment with a CG version that had more rubble and chunks of concrete and building, and "checked" the forest a bit.

"Predominantly, the environment is CG throughout the sequence, and that turned out to be a blessing. It's a little extra work to get to that point, but once we were in that space, we could relight the environment and are not constrained by the lighting they got on set," says Aitken. "We can make it feel more like we are outside, with low lighting from a distant sun, and we can change the look of the environmental lighting from action beat to action beat. So, it's sunny at the start of the sequence, and then after the compound is destroyed, it's overcast; there's a huge pall of smoke and dust hanging over the area. And when the portals open up, it becomes brighter again because we're playing the environment lighting to match the mood of the film."

THE BATTLE IS FILLED WITH DIGITAL CHARACTERS AND COMPLEX VISUAL EFFECTS.

For the explosions, Weta extended the work it had done in War for the Planet of the Apes to create the destruction of the base at the end of the film, incorporating volumetric pyro physics and modeling the transfer of heat into its simulations for a more physically correct and realistic result.

Weta also employed its crowd simulation software, Massive, developed originally for The Lord of the Rings, to populate battle scenes with tens of thousands of soldiers. "There were a lot of CG shots that required camera animation," says Aitken. "There's a shot when the two armies clash with each other at the start of the battle. It's a big, complex shot, and it lasts for quite a while. It's one of the longest shots we produced. There are plate elements in the shots, but essentially it's a CG shot that easily could have gotten muddled. Instead, our animation team on that shot produced something that is spectacular and also very easy to watch. You know where you're supposed to be looking at any one time. That's a real art."

Characters to the Rescue

In addition to Thanos, Weta also crafted Iron Man's Nano suit, having a history with the character all the way back to the first Avengers film. "It's interesting to look at the way he has progressed over time. We've been doing different suits for him each time," says Aitken. "He's got a new suit in Endgame, which includes some of the same Nano tech he was deploying in Infinity War, but he took a beating, so he built this new suit that has far more armor plating, more like the solid-steel plate suits of the original Iron Man."

Weta also did some Hulk work for the first time and shots of Thanos battling Cap, Thor, Iron Man, and Captain Marvel at the end of the movie. And, the artists did the Scarlet Witch, who amped up her power in this film. DeLeeuw provided some comic-book art frames that showed what he wanted her magic to look like, and then Weta interpreted them into CG simulations, in essence taking reference material that was very graphical in style and adding detail along with a physicality and volumetric look to it.

Weta also opened portals. In Endgame, just about every hero and villain from the MCU emerge and partake in the battle as they unleash their full potential. In a sequence that drove fans to scream with excitement, and visual effects artists to nod their head in approval, the characters materialize through Dr. Strange's portals, which Weta reworked at an immense scale. As Aitken explains, the artists started with the portal technology they had devised for Infinity War, the more human-size Dr. Strange portals, and repurposed them to be much larger. Also, a significant number of them had to fit in a single shot - and still be recognized as Dr. Strange's portals.

"It was important that when the portals start to appear, the audience is able to identify what they are and then realize these characters are now alive after having turned to dust at the end of Infinity War," Aitken says. "We didn't want bad effects to detract from the power of that sequence, so we spent a lot of time on it. It was very challenging, making sure that it played well and was spectacular and beautiful, and that the storytelling was clear."

AMONG ITS WORK IN THE FILM, WETA OPENED THE PORTALS THAT ENABLED THE HEROES TO EMERGE.

Moreover, the environments inside the portals were full-3D environments built by Weta: Wakanda, New Asgard, Contraxia, Kamar-Taj, and Titan. At the start of the sequence, it's very nuanced the way the characters appear out of the portals. The first one to open is into Wakanda, and it's very bright and hazy, before the characters are revealed.

"On the days when all the characters were there, were some of the most amazing days on set because you are looking at every character who has ever existed in the first 10 years of the Marvel Universe together at the same time," DeLeeuw says. "We had to make sure they all got their moments as they came back from the snap."

ILM also had a big role in the end battle, creating Strange's magic (water tornado, Winds of Watoomb). The studio opened the battle after the attack first occurs and as the heroes fall to the lower levels of the compound, animating Hulk and Rocket. ILM artists continued the action during the battle after Hawkeye hands off the gauntlet, with a shot of Hulk and Cap together, followed by shots once Black Panther picks it up and takes out Chitauri and Outriders, before he is stopped by Thanos's blade. Spider-Man flies in to assist. Strange uses the Winds of Watoomb to form a water tornado, then we're back to Spidey, before Captain Marvel swoops in and takes out Thanos's ship.

To ensure that each studio's battle shots were visually consistent and in the "same world," Weta and ILM regularly communicated, with DeLeeuw keeping them on the same page in terms of aesthetic, as each worked on their shots.

Having worked on the prior films with Marvel (including the Helicarrier destruction mentioned earlier) was a good launching pad for this film. "The battle of Wakanda was difficult, and we had put a lot of tools into play to build those environments, and destroy those environments, and inserted thousands of characters, which certainly helped us on the end battle here, where we were adding even more characters and more destruction," says Earl.

All Good Things…

Emotions ran high and low during the battle and in the aftermath. Indeed, the portal openings became a key turning point of the film - in addition to eliciting a good deal of emotional response from audiences. DeLeeuw witnessed the full effect that the scene had on viewers as he watched the film in a theater. "It's one thing to think [the scene] is going to work, then you see it with an audience and realize how much it really works and affects them."

Without question, Endgame weaves the entire MCU saga together into the biggest superhero movie of all time. And like in the film's end battle when all the characters unite in their ambitious common goal, so too did the visual effects artists from many top studios as they overcame daunting challenges to achieve a level of success never before reached in a cinematic production. Ah, the power of teamwork.