They dance like bulldozers and hit like jackhammers, as far from the “float like a butterfly, sting like a bee” boxer Muhammad Ali as possible. And yet, the powerful punches packed by the one-ton robotic boxers send crowds in the film
Real Steel cheering as much as the flesh-and-blood knockouts did before them.
“It’s very much a movie grounded in plausible reality,” says Digital Domain’s Erik Nash, overall visual effects supervisor. “It takes place in the near future. There’s nothing fantastic about anything in the movie.”
Nothing fantastic, that is, except the full-metal digital robots. Directed by Shawn Levy, the Walt Disney Pictures film stars Hugh Jackman as Charlie Kenton, a has-been boxer replaced by the robots. Charlie, now a low-level fight promoter, builds his ’bots from scrap metal. But, he might have found a spoiler. With help from his young son, Max, he believes he can build and train a high-tech fighting machine, a robot Rocky Balboa.
As might be imagined, much of the action in the film takes place in a boxing ring surrounded by crowds of avid fans. To create the illusion that giant robots fought in the ring, Digital Domain was part of a unique and exceptionally streamlined process that included Giant Studios in partnership with Glenn Derry from Video Hawks, the latter two groups bringing expertise and a Simulcam system honed while working on
Avatar. While Levy filmed Jackman and the other live-action actors on location, he and his camera operator could see a representation of what would become final shots in the viewfinder: boxing CG robots composited into the video stream of the real ring, with cheering crowds outside the ring reacting to the action.
The process began in late 2009, several months before filming, with Nash and Digital Domain digital effects supervisor Swen Gillberg working with Levy to plan motion-capture sessions that would take place at Giant Studios—sessions the team called “performance capture” and “camera capture.”
Digital Domain used a clever new 2D card-based system to populate the stands in the arena
with multiple copies of live-action extras.
The goal was to give Levy, first, the ability to direct stunt actors on the motion-capture stage at Giant and see, in real time, the boxing CG robots; and second, the opportunity to frame shots of the robots fighting in environments matching the locations where he would film the live action. The capture sessions also provided the in-ring fighting robots seen through the Simulcam system while the filming occurred on location, as well as the performance data for Digital Domain’s final animation work later.
“We called it ‘pre-capture,’ ” says Matt Madden, vice president of Giant Studios and the studio’s performance-capture supervisor. “We couldn’t do the motion capture on location because the robots needed to be one and a half times larger than a human. If you do a straight map of data from a six-foot human onto a nine-foot robot, it doesn’t look right, so we couldn’t have the robot actors in the same ring as a [human] actor. We had to put the robots in smaller rings.”
There was a second reason for creating the fights early, one perhaps equally important. “Shawn [Levy] wanted to educate the actors about the intent of the fights. He wanted to go on location with a solid understanding of what each fight was about,” Madden says. “Not a typical previs. He wanted a scene cut with the editor based on the actual performance of the robots. This turned out to be tremendously valuable during the shoot.”
Setting the Stage
Prior to the capture sessions, Digital Domain created environments and CG robots that would work with the Autodesk Motion- Builder-based pipeline used at Giant. Robot modeling and rigging for the eight main robots began in December 2009. Environment capture took place in February 2010.
“We had a two-pronged task for the robots,” Nash says. “One was to start the full development on the high-res robots, but also quickly create lower-resolution versions that Motion-Builder could handle.” Legacy Effects had begun work on practical models for three of the robots —Ambush, Atom, and Noisy Boy—that the artists at Digital Domain referenced, and the client provided concept art for the other robots.
“We photographed the robots at Legacy for modeling and texture reference, in stages,” says Paul George Palop, CG supervisor. “We didn’t want to wait for them to finish before we began. With the concept art, we had to grab the main idea of the robot without changing the look and make all the mechanics function. That took a lot of back and forth from the art director, director, and modeling department. Sometimes concept art looks good on paper, but the models can’t move.”
For the motion-capture sessions, the crew created low-res models of the robots fitted with simple rigs. “We only used the low-proxy models for the motion-capture sessions and playblasts,” Palop says. “The final models were a hundred times more complex, with moving parts.” Although each robot had its own look and personality, none of the robots talked. There was no facial capture, and no need for modelers and riggers to add elaborate facial systems later.
While the modelers worked at the studio, a separate crew from Digital Domain traveled to Detroit where filming would take place, to scout the physical locations and do photo surveys. “We wanted to create a fairly accurate but not labor-intensive virtual environment for each of the fights,” Nash says. They then gave the geometric framework for each location they surveyed to Giant.
Performance Capture
At Giant, the stunt actors worked in rings that were roughly 75 percent of the actual ring sizes, which varied from 18 feet to more than 25 feet. Giant turned the rings into capture volumes by surrounding them with an 80-camera optical system. Outside the ring, stand-ins played the roles of Charlie (Jackman), his son Max (Dakota Goyo), and a few crowd extras, which would help Levy choose camera angles later. To adjust for the scaleddown ring, Giant placed the stand-ins on a lower level. “If Atom [the robot] had to look down, we got a close approximation of where he would need to look in the final shots,” Madden explains.
During the performance-capture sessions, Levy referenced storyboards, communicated what he wanted to the stunt coordinator and the fighters, and directed the fights as he would have directed actors on location. Although Giant could have given him real-time robots to view during the fights, it didn’t. “We’ve found that when we’re capturing the performers, the directors want to concentrate on the action,” says Casey Schatz, previsualization supervisor at Giant. “And, it’s crucial to get the motion capture perfect. We didn’t want Shawn [Levy] worrying about camera angles.” Camera angles would happen later.
After the capture sessions, Giant applied the data to Digital Domain’s robots, and Levy picked the performances he liked. “Sometimes he liked the performance from one take,” Schatz says. “But sometimes he liked the first half of one fight and the second half of another, so we would stitch the parts together.” Giant also adjusted the motion as needed to give Levy the performances he wanted.
“The ’bots are superhuman, and sometimes the stunt actors weren’t capable of doing the actions Shawn [Levy] wanted,” Madden says. “We might speed up a punch cycle to give the robot a hyper-drive move. Or, layer additional motion when a robot jumped across the ring so it didn’t look like he was on a pulley system. This allowed Shawn to bring the actions closer to final performances so he could explore the camera angles he wanted for final shots.”
Above left, Giant captured stunt actors in motion-capture suits at its studio for the virtual
boxers who performed in the rings. Above right, stunt actors wearing IBC (image-based capture)
“jammies” performed the robots’ actions outside the ring while standing on stilts to give actor Hugh
Jackman a proper eye line.
Camera Capture
Once the crew at Giant knew which performances Levy wanted, they applied the data to Digital Domain’s low-res robots, placed the robots into environments created by Digital Domain and at Giant from art department plans. Then, they handed Levy a virtual camera. The virtual camera, a tablet-like device, had motion-capture markers on it so the crew at Giant could track it in much the same way they had tracked the stunt actors’ performances. Levy held the “camera” on the motion-capture stage. On the display, he could see the CG robots—boxing with the performances he had selected and properly scaled in CG environments that matched the locations and sets in which he would later film.
“We even put crowds in the arena,” Madden says. “They were static, but it gave the director a real-time version of the environment. The structures were accurate, and we lit it close to the final lighting. He wanted it to be as seamless as it could be on location.”
Exactly what Levy saw on the Giant stage through his virtual camera depended on the camera lens and camera angles he chose. Schatz had made sure that the virtual camera had the same lens kit that the cinematographer would use later.
“Shawn [Levy] would move around the stage holding the device, and he could see what I streamed from MotionBuilder, the robots, and the environments through the [virtual] lens,” Schatz explains. “He might say, ‘Put on a 35mm camera,’ and I’d change the lens. He drove the camera. If he did a take and didn’t like it, we wouldn’t save the data. When he got a take he loved, we’d dump it into the cut.”
Giant had an Avid system on site, and Levy’s editor was there on the days they captured the camera to assemble into a cut the shots Levy liked. “The cut would grow organically,” Schatz says. “We started with the first shot, put it in the Avid, and then the second shot.” At lunchtime, Levy would look at the assembled edit to see whether the progression worked in the way he had in mind. If not, he could go back to the stage to, perhaps, change a camera angle.
“Covering a fight can be difficult,” Madden says. “It’s so fast paced. There is frenetic movement with the camera. This session allowed us to create a realistic cut of the fights with gamelevel CG graphics and resolution. The timing of the action and the timing of the camera movement were close to the final result.”
Extreme Previs, Extreme Shoot
It was, in fact, an extreme form of previs. Directed by Levy, the previs included sequences representing all the robot fights. “The edit had good mocap motion, accurate backgrounds, and the director had the opportunity to do multiple revisions in real time,” says Digital Domain’s Gillberg. “It was a refined, advanced previs that we used as a very appropriate guide for principal photography.” At the end of a week, the team had the shots locked down.
On location, everyone knew which lens Levy wanted for each frame and every fight beat. In addition to the previs, Schatz created a PDF for every shot that showed the camera’s placement, field of view, how fast it was moving, and a time-code range. “The PDF also had the subject distance and height,” Schatz says. “It even showed the seating area to help Nash and the assistant director move the extras around and schedule the shots.”
With all these fight specifics in a database, the crew could sort and organize shots based on the type of camera, where the camera was looking, and how many extras were visible. “We could be extremely efficient,” Nash says. “By organizing the shots in groups by lens and camera rather than in continuity, it made shooting the plates incredibly time efficient.”
The previs also helped the actors and extras during principal photography, which took place from June through September 2010. Although the camera operator would see the CG robots during filming, the actors and extras needed to react to fights they couldn’t see. “We cued up the fight on the day so the actors could see it, and we also had the robot performers there on stilts as reference for the actors,” Madden says. “The performers knew the moves and re-created them as best they could as a run-through so the crowd knew what would happen. They would understand what it meant later when someone held out a tennis ball.”
When filming began, the crew would use a Simulcam process for the real-time compositing, with Giant and Glenn Derry’s Video Hawks installing an evolution of the system used for
Avatar. To make this possible, before principal photography, Digital Domain and Giant crews surveyed the set and registered it to the environment in MotionBuilder, to align the two boxing rings—real and virtual—exactly. This helped Giant determine origin points for the live-action camera and the robots so that the CG robots’ feet, for example, would look like they were on the floor of the real ring. To capture the movement of the liveaction camera, Giant placed and calibrated motion-capture cameras around the stadium to create a usable capture volume.
“We had to make sure the calibration was exact between the virtual fight performed at Giant and the real set,” Schatz says. “Sometimes that meant offsetting the real-time stream from the live-action camera.”
Prior to filming, Giant had rendered the robots in a void, with an alpha channel for compositing in the Simulcam system. So, during filming, wherever the camera moved, the operator would see an appropriate view of the CG boxers in the camera’s viewfinder overlaid on the live-action stream coming into the camera. In other words, the camera operator would see the robots as if they were in the ring fighting, and at the same time, see everything around them in the location—the actors, the crowds, the ring, full-scale mock-ups of the robots from Legacy outside the ring, stunt actors on stilts wearing image-based capture (IBC) “jammies,” everything. It was a real-time composite against a live-action background.
“The goal was to make the experience as intuitive for the first unit as we could,” Madden says. “Because, think about the alternative.What would the camera operator do? Whip the camera around and then in postvis hope you could put robots in the right place?”
The robots that were fed into the camera could box in fights that extended beyond the performances selected for the previs. “If we had a 20-second take, we’d keep the whole 20 seconds, even if we had used only two three second pieces for the previs,” Madden says. “That way [the camera operator] could shoot the action as if the robots were there.”
Thus, although the operator shot with the previs in mind knowing the angles Levy had chosen and how to get from one shot to the next, because he had the entire take, his choices on the day of the shoot could be intuitive. As he moved the camera, he could see the robots as if they were really on stage, and he could respond to the action authentically. “He could walk to the corner of the ring and what he saw would be correct,” Schatz explains.
It wasn’t so obvious for the audience. During the rehearsal and later, when filming began, Levy gave feedback to the crowd, as if he were a radio announcer, to help them understand what was happening in the fight. “We had ideas about sophisticated tricks that would help the 100-plus people around the ring feel connected to the fight, but what it came down to was having the robot performers who did the fights repeat them in the ring, and have Shawn doing voice-over of the fight like a sportscaster,” Madden says. Even though they couldn’t see anything in the ring, Levy’s narrative would remind them of the rehearsal. If the action moved to one corner, they’d know to look there. They’d know when to cheer and when to boo.
“We had never done Simulcam for movement this frenetic, for such a fast-moving camera,” Madden says. “In the beginning, we were nervous. We couldn’t have any significant delay in the CG robots displayed against the live action because that would be too distracting for the camera operator.”
Madden describes the first take: “Dave Emmerichs, the camera operator, was on a Steadicam rig for our very first take. We had the virtual robots composited into his viewfinder, so he watched the beat we wanted to shoot. He talked with Shawn. He went into the ring. Shawn called ‘Action.’ There were hundreds of extras jumping up and down, beer flying, people bumping the other camera we used to cover the shot. It was a classic, hardcore, liveaction scenario. And, Dave covered it beautifully. We knew then that it would pay off like we had hoped.”
Each time a robot landed a punch on another robot, Digital Domain needed to change the model.
Sometimes, the artists applied a new texture or displacement map. Other times, they deformed
the model. A database system tracked the damage.
The resulting plate became the first temp version of the shot. “As soon as we shot a plate, we had robots in it,” Nash says. “And the editor had that to cut with.”
Outside the ring, robots came and went, and waited to fight, and sometimes these robots interacted with the actors and extras. “There’s a shot with Noisy Boy walking through the crowd with Charlie and Max that wouldn’t have been practical to do at the studio because the actors drove the shot, the way they walked through the crowd and navigated the robot,” Madden says. “So we did the shot with the robot performer on stilts walking beside Charlie and Max.”
Similarly, stunt actors on stilts performed other robots, each performer wearing an IBC suit, so-called “jammies” that matched their character’s personality. “We put Noisy Boy in purple, Midas in yellow, and Atom in gray,” Madden says. “Each suit had markers in colors that would stand out in particular lighting conditions.”
Giant extrapolated motion data from the images captured by the witness cameras, applied it to character meshes that moved against the background, and handed the result to Digital Domain. Both studios, Giant and Digital Domain, tracked the camera. At Digital Domain, artists removed the actors on stilts from the plates, re-created the backgrounds, and refined the performances.
As for the boxing robots, after filming, Levy once again evaluated their performances, this
time in the context of the live-action shoot. Giant then implemented any performance changes before sending Digital Domain the clean motion-captured data. “We also cleaned up the Simulcam,” Madden says. “The Simulcam gave us a decent level of camera tracking, so once we got the approved shots from production, we sent a clean track to Digital Domain. They could put robots into the plate using those camera tracks and get quick temps not long after the live action wrapped.”
Crowd Control
Extras on location filled only the first few rows of the arenas in which the robots boxed. So, to fill the empty seats, digital effects supervisor Swen Gillberg at Digital Domain led a team that created an innovative crowd-replication system using 2D cards. The crew calls the system “Swen’s Kids.”
Gillberg had 80 extras to work with, so he shot five positions for each using three Sony EX3 HD, 24-frame progressive cameras: front on, quarter front, quarter profile, three-quarters back, and back. He shot them from camera level, from on high looking down, looking up, looking severely up, and in a medium shot. “Then for each of the five positions, we had them do four different emotions for five seconds without cutting the camera,” Gillberg says. “We ended up with 8tb
of extra data, but we covered all the angles you’d want in an arena.”
Gillberg and his crew then put each of the extras on individual cards. “We could have a crowd of 40,000 people, or we could push in to four people,” Gillberg says. “We had enough information to do three extras edge-to-edge on frame, and they would hold up at 2k.”
Digital environments supervisor Geoffrey Baumann and digital environments lead Justin van der Lek created the system that organized and oriented all the cards. “It would have all the cards facing the camera in [The Foundry’s] Nuke,” Gillberg says. “We would bring in the shot camera, and the system would calculate how many cards would be in the frustum of the camera. It would figure out which cards to grab based on where the camera was—the back of people’s heads, their faces, whatever. Then, it would calculate out how to render all those people in manageable chunks, render them out, and then combine them as a post process. The genius part was giving non-technical artists sliders they could use to get emotional beats.” –Barbara Robertson |
Post Production
At Digital Domain, in addition to the Simulcam track, artists received plates with the corresponding motion-captured data from Giant that matched the frame range in the plates. “Giant would give us the [Autodesk] Maya scene with a camera, a proxy plate, and the motion-captured data on our original skeletons with the proxy geometry,” Gillberg says. “We had the camera track from the Simulcam, the plate from the motion-picture camera, and the motion capture from the virtual camera. So first, we’d render the Giant scene to double check. Then we’d ingest that data onto our high-resolution robot rigs and test that. And afterward, we’d send the shots through our pipeline and lighting packages, with default lighting using [The Chaos Group’s] V-Ray. That gave us temps right away. We didn’t have to wait for a tracking team, or for animators to bring in the motion capture. So it was a great head start. We’d know what worked.”
There was cleanup, of course, of the camera track, capture data, and animation. But, the temps let the artists see how the robots looked in the various environments.
“Giant got us 80 to 90 percent there,” Nash says. “It was the foundation. It was the heaviest use of motion capture I’ve been involved with. And, we did a lot of work on top, especially for the robot contact. There was no shot with straight motion capture. We made the robots’ punches appear harder. We did keyframe animation for nuance and subtlety, to finesse and fine-tune the performances. And, we spent a lot of time in R&D on robot damage.”
As the robots moved, pistons and gears inside pumped and turned automatically via expressions built into the Maya rig, and jiggle deformers tied to the animation curves caused the heavy plating to react slightly, adding a subtle secondary motion. “If the robot turned to the left, an inner gear would turn clockwise at the same ratio,” Nash says. “It gave the impression that the internal movement was driving the robot, when, in fact, it was the opposite. The motion capture was driving what happened inside; the internal mechanics were derived procedurally from the motion capture.”
The amount of visible damage increased as post production progressed, with two of the robots suffering severe damage. To bang-up and batter the robots, the artists at Digital Domain needed to create new surfaces and textures, and sometimes change the models. Each time a punch landed, metal had to deform or the surface had to change. “Tracking the damage was quite a logistical challenge,” Nash says. “We had to be very precise about what level was appropriate for each shot so that damage didn’t suddenly disappear.”
No Bull
Oddly, one of the biggest challenges for the crew at Digital Domain was a bull, not a robot, and chances are many people won’t realize the bull is a CG creature.
“They shot most of the plates with a real bull running around, but they couldn’t punch a real bull,” says Paul George Palop, CG supervisor. “So we had to add a CG bull for key moments, and that single asset was more complex than we expected. We had to deal with fur, muscle simulation, skin simulation, and photorealistic rendering of organic skin. It required a lot of work for the animators also, because there’s heavy interaction between the bull and a robot. And, they’re
both kicking an incredible amount of dust and dirt, so some of those shots are quite complex. The bull is in only approximately 15 shots, but those were some of the toughest shots in the film.” –Barbara Robertson |
Damage Control
The Crash Palace sequence had the most gruesome fight—one between Noisy Boy and Midas—but the hard-metal boxers damaged each other throughout the film. The artists had to add hoses that dripped fluids, sparks, and smoke. “We had to choreograph a sequence of events for where the punches landed and what the effect was,” Palop says. “Creating the damage involved several departments—we were doing everything: texture maps, displacement maps, modeling. We had to crush, bend, deform, and break pieces of the robots. So, we developed a system that knew what damage level to apply to which shot. We had a database with all the information regarding damage for each sequence and each robot. We could partially automate the way we could switch on levels of damage through a sequence.”
To create the robot surfaces, the artists designed more than 30 different materials, some based on photographs of real objects, some imaginary. “For the black robot Zeus, we wanted something that looked slick and cool, but couldn’t find photographic reference,” Palop says. “So, we created a few layers of clear coat and carbon fiber, and put a bumpy surface beneath with a specular component. It was really cool, but not necessarily anything that’s really out there.”
Camera operators on location could see CG robots in their viewfinder overlaid on the live-action
stream in order to accurately frame close-ups and medium shots. Digital Domain created the
robots in these final images.
The effects department added the fluids and sparks using Side Effects Software’s Houdini for hydraulics and other fluids, and Maya fluids to create dust and dirt. “We baked out geometry for some of the effects,” Palop says. “We would render fluids out of Maya using [Mental Images’] Mental Ray, but if we had geometry—say, Noisy Boy’s arm—we’d use Houdini and V-Ray.”
V-Ray, however, did the lion’s share of the rendering. “We set up a test between [Pixar’s] RenderMan and V-Ray, and ended up going with V-Ray,” Palop says. “It was giving us a better look for the robot. We probably could have gotten there with RenderMan, but it would have taken more time, and we had another show going on that needed the Render-Man TD support more than we did. So V-Ray made sense, and it turned out to be a great experience. The crew on
Tron had done a lot of the work for us. We basically grabbed the best features of the
Tron lighting pipeline and used those on our show.”
The survey crew that had been on location brought back photographs and HDR probes that gave the lighting crew information about every main source of light on set that they needed to match. “We had geometry based on the survey data of the set, and we could project photographs taken on set onto the geometry to have an accurate representation of the set,” Palop says. “We gave that to the lighters. They’d bring the robots into this virtual environment and have V-Ray start rendering. We had photographed every light source using a fish-eye lens with dynamic range. We knew its location and how intense it was. Then we put the HDR onto cards; each light source was a card that we put in the set.”
The lighters would choose the number of rays to bounce inside the set based on the materials in each robot, optimizing the lighting and rendering based on, particularly, glossiness. “The glossier the material, the fewer rays that we needed,” Palop says. “If you have a mirror, you only need one to have a perfect reflection. When you have more diffuse surfaces, you want to sample more to reduce your noise.”
The crew rendered each robot separately to have absolute control over the rendering. When two robots fought, they’d render the two together to have accurate reflections and shadows from one onto the other, but with only one visible at a time.
“One of my favorite memories involved our lighting and surfacing teams, which were really special,” Nash says. “We were in Detroit. I think it was in August, about nine weeks after shooting our first robot sequence. The team at Digital Domain sent me a shot, a first pass out of lighting that blew me away. I looked at the shot and said to Swen [Gillberg], ‘What do you mean first pass? I don’t have any notes.’ Then I walked up to Shawn [Levy]. He smiled. He could tell I had something to show. When he saw the robots, he went bananas. He showed the shot to everyone on set, and it was a boost for the whole crew. We had the practical robots Legacy had built. And, we had the guys in green spandex with tracking markers. But for the most part, we had Simulcam with nothing to look at. So it was good for the crew to see what the robots would look like. We finaled some of our early shots while we were still in Detroit.”
Despite his excitement and pride in the look of the robots, though, Nash believes the coolest thing about this show was the process. “The whole process worked beautifully,” he says. “Because we were involved from the beginning, we had ownership and full membership in the main unit team. It was an incredibly smooth production, a very tight ship, and hugely collaborative.”
These teams from Digital Domain, Giant, and Video Hawk moved the
Avatar process out of Pandora and into the real world. By inserting CG creatures into the middle of a liveaction shoot, the director and camera operators could work as if the CG creatures were there on location outdoors and indoors. And that means visual effects had taken one more step out of post production and into production.
Barbara Robertson is an award-winning writer and a contributing editor for Computer Graphics World.