Power Players

It wasn’t that long ago when Marvel decided to launch its own movie studio and turbo-boost the Avengers comic-book heroes by first releasing the hugely successful Iron Man film (starring Robert Downey Jr.) in 2008. Grossing more than $318 million, it was the birth of a franchise that then led to the release of The Incredible Hulk, Thor, Captain America: The First Avenger, and several sequels, before culminating in Marvel’s The Avengers in 2012, which grossed more than $623 million. 

With numbers like those and groundbreaking special effects work from some of the industry’s leading studios, why wouldn’t Marvel give the Avengers another go ’round? 

In its latest offering, Avengers: Age of Ultron, Marvel and Director Joss Whedon captivate audiences once again, leaning heavily on visual effects by studios that include ILM and Trixter, among others, as well as previs and postvis work from The Third Floor. In addition to creating a captivating cast of characters, visual effects artists built stunning digital environments that play an equally important role in the story.

ILM: Hulk, Iron Man, Ultron Prime, Destruction

ILM’s San Francisco, Vancouver, London, and Singapore studios contributed 800-plus shots featured in Avengers: Age of Ultron, making the facility one of the film’s largest VFX vendors.

ILM built the Hulk, Hulkbuster suit, and Ultron Prime for the latest Avengers film.

The film challenged ILM with further developing the Hulk and Iron Man characters they had worked on for past releases, as well as imposed brand-new challenges, including the creation of Ultron and helping to carry out the villain’s grand plan to destroy a European city by ripping it from the ground.

According to ILM VFX Supervisor Ben Snow, the Hulk appears 50 percent more in Age of Ultron than in the prior Avengers film, and he is more agitated. “[He’s] not really himself,” says Snow. “He’s an amped-up version, so one of our first tasks was to make him a more extreme and crazed-looking version. Joss likened him to someone strung out on drugs.”

ILM, says Snow, set the bar high, working to improve on an already well-received character. “It was like, how do we push this even further to make it more believable and a more realistic performance? We thought a lot about doing that and worked with Mark Ruffalo so he could help us on set, performing the character for some of these quieter moments. Also, we rebuilt his technology. The design of Hulk was largely the same, albeit angrier and more strung out. We essentially doubled the resolution and rebuilt him from the inside out. We made a full skeleton and skull, which he hadn’t had in the past. We rebuilt the muscle system to more correctly drive his surface, rather than be a simulation tool. We took the sculpting we did in the past and rebuilt his muscles, so once we layered the flesh and skin simulation on top, he would look the same, but you see his muscles move under his skin much more convincingly.”

While Snow isn’t sure of the poly count, Hulk represents the studio’s largest human asset to date. ILM employs a mixture of off-the-shelf and proprietary tools in its feature workflow. Most of the studio’s animation and base rigging is performed using Autodesk Maya. Simulations are created in Maya and the studio’s proprietary tools, which provide soft-body dynamics and solid-body dynamics, says Snow.

“For a lot of our simulation and for flesh and muscle systems, we were using our proprietary solvers. The skin-sliding simulation or muscle-driving simulations were implemented mostly with internal stuff,” Snow says.

The group uses Pixar’s RenderMan for rendering creatures, and Chaos Group’s V-Ray is used for rendering environments.

Even though ILM was able to use some of the assets from Iron Man’s Mark 43 suit, which was introduced at the end of Iron Man 3 and built by another facility, the artists nevertheless were faced with making the “Hulkbuster” Iron Man suit and the new Mark 45, Iron Man’s latest creation.

“It’s a little bit of a different direction,” says Snow of the Mark 45 suit. “It’s a much more streamlined suit, with rounded forms. It still feels like Iron Man, but it presented some particular challenges because of the streamlined roundness. As soon as he started moving, he got these weird gaps that would open up — more than we had experienced with the more traditional design.”

But even Iron Man’s impressive suit pales in comparison to the work ILM put into creating Ultron Prime.

Ultron, says Snow, “was probably the most elaborate hard-surface] hero character that we’ve created. He was a full robot and had to feel like he was made of rigid material, but we really wanted the essence of the James Spader performance to come through and have a nuance you would get from having an actor like James Spader.”

ILM received an initial Ultron design from Marvel. The studio then built on top of that, making his body and face even more complicated. 

“We used a proprietary tool for rigging,” recalls Snow. “Basically, it was 10 times as complicated as the rigs we did for Transformers. It had around 2,000 individual nodes in the rig, with 600 in the face alone. This is because we couldn’t really get the face to squash and stretch. You didn’t want it to deform like a creature would, you wanted to see the plates sliding below one another. It was an extremely elaborate rig and a collaboration among the rigger, animator, and modeler, adding more cuts and breaking it up.”

Snow continues: “The ultimate goal was to give the animators the sort of control they’d have on a normal face. We gave them the tools they would have on a normal face, and we gave them a version of James Spader they could animate as a digital double. It was driving this elaborate, robotic face that rode on top of that.”

ILM employed its proprietary Muse motion-capture system on set to allow the actors to see how the Hulk and Ultron’s expressions would appear, and to serve as a reference for the artists creating the final animation.

The studio also contributed to a big, timed sequence in the trailer, where all the Avengers are shown coming together and in action. The Hulk and Hulkbuster sequences were among the first scenes shot, since they appear in multiple trailers and in time for last summer’s Comic--Con convention in San Diego.

“It was the first sequence that was shot,” says Snow of the one featuring the Hulkbuster. “We shot it in South Africa and planned it well. There was previs, and it was a second-unit shoot, though Joss was there with us for part of the shoot. It was, of course, featuring the Hulk and the Hulkbuster exosuit that Tony [Stark] creates around his Iron Man suit.”

In the film, a city is ripped from the ground, leading ILM to build one of its most complex digital environments yet.

Beyond their contributions to the film’s superheroes, the ILM team also handled a major VFX sequence in the third act.  

“Ultron engineers this crazy scheme to basically lift a small European city from the ground it’s sitting on and raise it up into the sky, with the idea of dropping and crashing it into the earth and causing a huge cataclysmic event,” Snow explains. “We did a lot of the work on the creation of this European city, and the liftoff and the ground effects/earthquake effects, ripping itself out of the earth, and the flying city.”

The sequence, says Snow, represents ILM’s most elaborate digital environment asset. “We had to create this city, which you have to see from all sorts of different distances. It had to break apart.”

The studio took digital matte assets — generated in Auto-desk’s 3ds Max and rendered in V-Ray — and ran them through ILM’s proprietary rigid-body destruction tools, finally going back into V-Ray for rendering. 

“In the wider shots of the city lifting up, you have whole avenues of buildings breaking apart, the ground ripping apart,” Snow explains. “In the past, when we’ve wrecked buildings, it was a single skyscraper or a few skyscrapers falling into one another. We had to really step outside of this — it’s a whole city this time — and change our approach a little bit. We wanted the detail and realism, and hand-drawn artwork that you get from the digital matte artists, and combined that with our rigid-body destruction so that we could put interiors into these buildings and not have it seem like they were eggshell buildings cracking.”

The studio employed its Plume proprietary effects simulation software to add smoke and dust in the debris.

VFX work on Avengers: Age of Ultron spanned ILM facilities around the world. The Singapore studio has been contributing to films since the first Iron Man release, but this film marked the debut of ILM’s UK office and also took advantage of ILM’s resources in Vancouver, which were able to receive dailies at the same time as the San Francisco location.

“It was a terrific experience because we were able to hire talented artists in those locations,” says Snow. “The team in Singapore has become an advanced team and does work at the levels of anywhere else in the world. And because we share a pipeline, we are able to split sequences among companies.”

Trixter: Quicksilver, Ultron Mark I

Trixter, which has German offices in Munich and Berlin, as well as Los Angeles, was responsible for more than 300 VFX shots in Age of Ultron. The studio’s work includes effects for Quicksilver and Scarlet Witch, and the first time audiences see Ultron (Mark I). According to VFX Supervisor Alessandro Cioffi, the studio got involved in early 2014, collaborating during preproduction with the film’s visual effects supervisor, Christopher Townsend.

“We investigated a look for Quicksilver and his colleague,” says Cioffi of the studio’s initial work. “This started during the time of Captain America: The Winter Soldier. There’s a sequence at the end of the show where these characters were introduced. [We did] the concept for the visual effects for these two characters and then applied them.”

Quicksilver has the ability to run or move extremely fast. “He has a peculiar and specific look in the movie,” Cioffi explains. “Scarlet Witch has telekinetic powers and mind-control powers, and also needed some enhancements and CG effects to be explanatory in the live--action performance.”

All the work was performed at Trixter’s Munich facility, which has a core team of 40-plus artists, but scaled up to approximately 100 during crunch time. This included delivering 80 final shots in time for the film’s trailer debut at Comic-Con last summer.

“We worked massively on the project. In proportion to our scale, we are a fairly small facility. Working on over 300 shots was quite a challenge,” Cioffi notes.

The visual effects that support Quicksilver’s motion went through several stages of development. “We did some preproduction work with Chris, trying to define the look and where the possibilities were, and what has been seen before and what we wanted to avoid,” says Cioffi. “The first phase was trying to figure out what we could have done in-camera.”

Trixter gives quicksilver’s high-speed action a visual boost.

Cioffi says the team looked at a number of ways to visualize Quicksilver’s high-speed motion. One option was to use real time, where the character is shown moving extremely fast in-frame. Another was to show him moving at normal speed but with the world around him appearing to slow down. And the third was Quicksilver moving at normal speed but with the world around almost frozen in time. 

“All of these approaches have a different look and require different shooting techniques,” Cioffi explains.

Ultimately, Townsend and Whedon opted for a look that resembled shooting things “at 6 fps, with a pretty open shutter angle,” recalls Cioffi. “You get a photographic, smeary effect — not just the direction of blur, but a very dynamic and curvy effect behind things that are moving fast. In order to achieve that with our compositing tool, we had to shoot pretty much everything in high-speed — at least the foreground, namely Quicksilver — with an Arri Alexa. We were oscillating between 120 fps and 72 fps, depending on the shots, and always trying to shoot a clean plate or backgrounds.”

Trixter uses a combination of off-the-shelf software within its proprietary pipeline. For modeling, animation, and rigging, the studio employs Maya. Lighting is achieved using The Foundry’s Katana, and rendering is done using a combination of RenderMan and Solid Angle’s Arnold. Side Effects’ Houdini is used for effects, and The Foundry’s Nuke handles compositing.

“They are all standards but are joined together in our pipeline, which is a living creature and has to adapt in specific cases,” Cioffi notes. 

In the comic, Quicksilver emits silver particles that sometimes freeze in position. Subliminal images also appear along his motion path. This was achieved by freezing random frames here and there, and compositing them in a subtle way.

“Then there was the whole part of creating the CG trail, which makes the effect look interesting,” Cioffi adds. The CG trail has a blueish, cold tonality, with silvery, shiny reflections. This was created entirely in Houdini. “The process was not that straightforward, as we had to create rotomation of the character. We had to rotomate a dummy on him to re-create his motion. This dummy was the base for releasing the particles behind him.”

The studio had as many as 100 licenses of RenderMan. Some of the rendering resources were allocated to characters and effects, while others were reserved for Houdini and compositing operations.

Trixter also produced the assets for the Ultron Mark I. “It’s the first time Ultron appears in the movie, and he appears in a temporary shape, as a self--assembled robot,” says Cioffi. “He has an extraordinary look — very creepy and mysterious. This CG character has been enriched greatly by James Spader’s performance.”

The studio also performed a fight sequence at the Avengers’ headquarters. “This sequence, which we worked on entirely from A to Z, was, for me, the most interesting thing we worked on. In terms of visuals, animation, rendering, integration, and compositing especially, what happens at Avengers Tower during this fight is something that made me excited and willing to work on this show,” says Cioffi. 

THE THIRD FLOOR: Previs/Postvis

As previs/postvis supervisor at The Third Floor working on such Hollywood blockbusters as The Avengers, Iron Man 3, Thor: The Dark World, and Thor, Gerardo Ramirez has plenty to say about the important role that visualization played in the newest Avengers offering.  

At top is a Previs frame from The third floor of the early tie-in shot. at bottom is the final frame from the fight. The scene required coordination through Previs, camera mapping, detailed set planning, and Postvis to help execute the shoot and effects.

“We knew early on this was going to be a big film with lots of visual effects and a need to figure out how the story would develop or how to shoot certain scenes,” Ramirez says. “Joss wanted to utilize all the visualization tools we had — previs, mocap, virtual production, techvis — so we knew we were going to touch multiple aspects of this film and help multiple departments.”

According to Ramirez, previs was a tool used to help the director and other departments — stunts, second unit production, visual effects, and so forth — realize Whedon’s vision. 

“When they can actually see something in motion, see this idea and the way Joss actually is envisioning the concept, it’s a lot easier for them to figure out what they’re going to need to build, what kind of props they’re going to need, what lenses, what angles. It’s a very informative tool.”

Beginning in September 2013 at its LA studio and then moving to its UK offices for the majority of the work, Ramirez says The Third Floor wrapped up its Age of Ultron contributions earlier this year. Shot predominantly on Arri Alexa cameras, with the Vision Research Phantom camera for the high-speed 

Pietro Maximoff shots, The Third Floor relied on its Xsens MVN system for motion-capture and on Maya.

“We visualized a lot of the major CG scenes; the action scenes,” Ramirez says, “such as the Hulkbuster sequence, the freighter scene, the South Korean chase scene, the final battle, the party fight, and the early tie-in shot.”

Ramirez explains that the studio was involved in so many areas of the film, “we had part of our team designing one of the action sequences, like the Hulkbuster sequence, with a few artists animating that, and we also had a virtual production team, led by Casey Schatz, that helped scout and rapidly prototype scenes using previs, motion capture, and techvis viewed within a virtual camera system. 

Once the filmmakers found a location or had a design they wanted to construct, the previs team created that location in CG. Then, at their convenience, the director, cinematographer, or any department was able to come in and hold the virtual camera, and basically do a virtual scout and walk around the sets and explore camera angles.”

According to Ramirez, Second Unit Director John Mahaffie used the virtual set heavily, spending hours with the virtual camera to explore different parts of the set, rehearse scenes and stunts, and view them from different camera angles. “Once he was on location, he knew that set like the back of his hand,” says Ramirez. 

As soon The Third Floor team received the plates for the Hulkbuster sequence, they began integrating previs CG character versions. “It was important for that sequence because it’s all CG action,” he explains. “After they shot empty background plates based off the previs, we would track the plates and then try to use the animation they liked from the previs and place it onto those plates,” explains Ramirez. “And that’s kind of the postvis process we did throughout the entire film.”

Ramirez points out that for a scene such as the “party fight” in the Avengers Tower, where the characters are running away from Iron Guard legionnaires firing at them, “we comp’d in our CG Iron Guards in the background, and this process helped Joss take a look at the sequence, edit it, and tighten it up. And the end result was that anyone in production could see the sequence and understand where it was going. It was helpful for the editor and helpful for the VFX vendors.”

The Third Floor’s biggest challenge, Ramirez says, was the film’s early tie-in shot, where, “in one continuous camera move, we go from each character. We spent extensive time on this shot. We worked on the previs, and then we broke down the shot and took it into techvis and calculated the camera moves. Once they shot the plates, the next step for us was the postvis, which was also challenging because we had to take all these plates and try to combine them into one continuous shot. They really finessed the shot until they got it to exactly where they wanted it, so when we passed this off to ILM, they knew almost exactly what Joss was looking for.”

With a film like Avengers, “previs becomes so powerful because we’re helping to make a lot of the decisions. And I mean, Joss, the editors, and the VFX supervisors used previs for showing all the departments, including the visual effects studios, this is a vision we like,” explains Ramirez. “With such a tight schedule and a film of this magnitude, there’s not a lot of time for the visual effects studios to get these shots done. It was important that when we passed off [the work] to the visual effects studios, it was close to what everyone was looking for and expecting.” 

In Age of Ultron, it took a team of superheroes to overcome the odds and get the job done. The same can be said of the teams that worked on the feature film.