Battle Plan, Part II: Making the Matrix

A range of popular movie franchises has released sequels and prequels during the past few years. Yet none have been as ambitious as The Matrix Reloaded and Revolutions, which were created in parallel—in essence, as one huge movie—with only months separating their screen debuts. For Reloaded, digital artists focused on creating superhuman stunts, while for the more recent Revolutions, they created even "bigger" CG effects to portray the immense conflicts between man and the machines.

So how does a person plan and execute such groundbreaking shots? There's only one way, says visual effects supervisor John Gaeta, and that's by utilizing a wide range of previsualization techniques. "For films that are as technically complex as The Matrix sequels are, it was mandatory to develop a method of orienting all types of film craftspeople toward a common objective," he says. "As such, previz was used extensively to create a centerline of intent and objectives for the sequels."

Previz was used to develop numerous scenes in Revolutions, yet the technique was never more imperative than when visualizing the action of the siege, including the motion of the humans and the APUs. Previz images courtesy Pixel Liberation Front.

Gaeta's previz group used these techniques extensively in many ways across many scenes in the films, from conceptual scene design to highly accurate stage planning and even more accurate camera-path planning, motion-base exporting, and rig-photography management. The visualization process—which began in mid-2000 and didn't end until mid-2002 when filming stopped—was the result of a long premeditated strategy by the directors, Andy and Larry Wachowski, and Gaeta, who introduced the brothers to the idea of using visualization to plan their complex Bullet Time shots for the first Matrix film.

"Those experiments weren't just for conceptualizing camera paths, but they also were used for every aspect of the [sequence's] setup and execution," Gaeta notes. "That experience empowered us to make two sequels that were extensively visualized."

For Reloaded and Revolutions, Gaeta and the Wachowskis previsualized as much of the effects content as they could and then provided that information to the vendor studios hired to work on the shots, all the while involving those facilities in the process as much as possible. "If you are going to get very literal with the viz, then the vendors need to be invested in the shots you have created to determine the downstream feasibility of the event you created in previz," explains Gaeta.

In many instances, the studios executed the visualization alongside the centralized previz group formed by Gaeta under EON Entertainment's banner. The backbone of this composite group was Pixel Liberation Front (PLF), helmed by Colin Green, PLF founder and previz supervisor for Reloaded and Revolutions. Assisting PLF and acting as Gaeta's logistics supervisors were visual effects supervisors John DesJardin and Dan Glass from EON.

The viz team's mandate was to work with the vendors as well as the departments within the principal photography group, including camera, lighting, and design, who would benefit greatly from the previz. "Our extensive use of previz required the support of the vendor visualization supervisors, who took the material and attempted to evolve it into a functional scene template," says Gaeta. "From that, they could retrofit next-generation components, whether it was viz-guided photography or computer-generated characters."

In fact, utilizing previz "for just about everything" was a novel concept for the Matrix producers, who had to be sold on the idea. "This is the largest visualization project in movie history," notes Gaeta. "Building a CG department within the production group was a different method for organizing crews and broke the tradition of limited use for limited purposes."

In all, Gaeta estimates the group created more than two hours of previz content—basically an animated movie. Although not every scene in both films was previsualized, the majority of them were, falling into two main categories—photography management and CG development.

The 270-day film shoot involved countless scenes and shots with extensive greenscreen and bluescreen setups, CGI set extensions, and integrated CG characters. Having previsualized these shots, Gaeta could determine, for instance, where the action was taking place, what objects would be digitally created, and what types of camera perspectives were needed. Looser, or more basic, previz versions provided a general idea of where to point cameras, while more precise renditions allowed for the exportation of camera paths into robotic camera systems for scenes whose filming required the use of a motion-control base.

On the flip side, the previz was used for shots that were 100 percent computer-generated. For these, the scenes were logistically immense and extremely costly. Yet by creating entire scenes and editorializing the previz shots, Gaeta and the directors were able to determine, for instance, how many objects were needed in each frame for budgetary purposes. It also enabled vendors to calculate their own costs, and it eliminated a lot of unnecessary work.

In a more complex application, Gaeta used previz to find unique camera and lens styles for capturing thrilling shots, including Bullet Time shots, those involving vertigo-like lens effects that don't necessarily follow straight paths, and complicated curving camera moves with floating interest points. "We did a number of virtual photography and Bullet Time shots that would have been impossible to do to their full potential without visualizing them ahead of time in 3D," he points out.

In fact, one of the most elaborate shots in Revolutions didn't make it out of the previz stage. This scene had Neo flying through the mega city in a curved, zigzag pattern while the camera orbited his body and sliding lenses created a feeling of vertigo. The shot was nixed because of its steep $1 million production price tag. "It was super killer and my favorite," says Gaeta.

Aside from that scene, the group spent a year conceptualizing and designing the visual effects for the freeway chase in Reloaded. That work, in turn, lead to the Trinity Falls and Trinity Saves scenes, and the siege on Zion in Revolutions. As the filming began, the visualization became the backbone of the stage planning and technical orientation for a variety of the film departments at EON, as well as vendor templates from which they could build scenes and shots.

"We used previz for all the flying design—which was a high-concept study area—including the scene when Neo flies out of the flaming building and down the street at 2000 miles per hour," notes Gaeta. The tool also was used to plan the unique camera perspectives in the super brawl between Agent Smith and Neo, and for constructing the digital face and determining the camera proximity for the ultra-realistic Smith model used for the super punch (see "The Matrix Resolution," December 2003, pg. 22).

"People may be curious as to why certain shots have a unique layout or wonder how a person arrives at a certain perspective," Gaeta says. "Well, the idea is an amalgamation of the creative minds of the director and concept artists, and a by-product of the creative 3D exploration process."

Revolutions, in fact, includes some of the most highly choreographed combat scenes to play out on the big screen, including the siege, a 12-minute sequence depicting the battle between the humans and the squid-like sentinels. With 90 percent of the scene created digitally, it's the largest CG segment in the movie and a climax in the Matrix saga. "A scene like that costs tens of millions of dollars," says Gaeta, "so you can't afford to make any mistakes while shooting the live-action pieces."

Executing this detailed, extremely complex showdown at Zion turned out to be a monumental battle in itself, as the previz group spent more than a year planning just that one sequence. In the end, their effort provided the effects team at ESC Entertainment with a tangible blueprint to use while realizing the directors' elaborate vision on film.

The directors used previz (top) to define sentinel behaviors for the final shots (bottom), such as how fast the sentinels would move and how big the swarms should be.

While the finished sequence is impressive, DesJardin notes that the conceptualized scene was far grander in scale. However, during the previz process, it became apparent that the brothers had to pare it down to clarify the action and story points for audiences and to enable ESC—which was being formed as a studio at the time of the previz—to finish the sequence on time. As a result, the effects supervisors and, in turn, PLF spent a tremendous amount of time reacting to the changes.

The directors wanted the siege to be emotive as well as illustrate the dynamic struggle between the humans and the machines. And conceptualizing a battle sequence between the armored defenses of an underground city and hundreds of thousands of flying sentinel robots is really beyond the imagination of anyone but the directors, contends DesJardin. Thus, it had to be defined visually as a sequence in order to be fully and accurately understood by the hundreds of artists creating it.

Moreover, the sequence had to address numerous plot points—for instance, what kind of battle tactics do the humans and machines employ, how do the humans initially fend off the machines, and how do the machines eventually outsmart and conquer the humans—all of which dictated how the battle had to evolve. "They had to show a reason why the sentinels are attacking in schools and groups," says Green. "They also had to show the cause and effect of the background animation, such as what happens to all the dead sentinels—an issue worked into the story line when the humans have difficulty reloading the APUs because the debris is piling up on the catwalks. All of this had to fit within a coherent battle plan."

Even the directors evolved their vision for the siege, as well as many other scenes, as the previz group planned the scenes visually. Working from storyboards and concept drawings, the team created an extensive animation of the siege using low-resolution models of all the objects in the scene, which were created in Alias Systems' Maya. Breaking the shot-by-shot paradigm of traditional animation, the previz animations were created in 15-second chunks that recorded an event from a number of camera angles.

While nearly all of the previsualization for Reloaded and Revolutions was done in Softimage|XSI running on Hewlett-Packard machines and specially built PCs, PLF chose Maya for the siege work. Creating the lower-resolution images in the same software package that ESC planned to use for constructing the final complex imagery in the CG-heavy sequence enabled ESC to work directly from the previz components.

While planning the siege, the team also used previz onset to generate the hybrid human/robotic motion of the Armored Personnel Units (APUs), massive machines that mimic the movements of their human operators. In essence, the robot is an extension of the human piloting it, so as the person lifts an arm and presses a trigger, for example, the APU's huge robotic arm lifts and fires a weapon. Having a visual connection between the robot and the powerful forces it was applying to the pilot was vital to selling this concept.

Previz imagery (such as above) was used to plan many final film scenes (such as shot below).

For this segment, Gaeta wanted shots that started wide, where you see the APU doing its various moves, and then push in close to show that a real human is driving the robot. This was done by importing the CG motion to a motion-control simulator, thereby synchronizing the CG and the live action and mimicking motion that under any other condition would have been impossible to replicate.

To accomplish this, PLF acquired the fully fleshed-out APU animation files from ESC, which created the robotic models and their walk libraries. Under Gaeta's direction, Green then devised a setup that would take the APU animation and the camera animation from ESC and turn it into motion-controlled camera-move files and motion-base files.

According to DesJardin, this sequence was the most difficult to previz because of the importation and exportation of position data to achieve the precise moves that Gaeta and the directors wanted. "We were photographing a live element (the actor) that had to be accurate to the animation in order to achieve all the bumps, sways, and inertia on the human," DesJardin says, "so when the CG character was added, it all made sense and the animation lined up perfectly."

To forge an intimate relationship between the APUs and their human drivers, PLF set up a complex pipeline that converted the APU animations into motion-controlled camera and motion-base files (top) for matching the live action to the CG (middle) for the f

Despite using state-of-the-art camera equipment and a high-end motion base, the original motion still had to be smoothed and scaled, and the camera and APU motion had to be translated from Maya (which ESC used) into Softimage|XSI, where a virtual Milo motion-control camera and virtual motion base were constructed.

"Highly conceptual and highly accurate camera-path planning and perspective control via motion-base exports can be refined to an exact science," Gaeta says. "It's when to use it and apply it accurately that ultimately matters." While previz has assumed an increasingly important role in film production, Gaeta believes that supervisors from both the facilities and the production visualization department must forge a closer relationship before new ideas in photography will be unlocked.

While studios and directors implement previsualization for different reasons, some—like the Wachowskis—see it as a way to make a better movie. "Many people use it as a thought process to arrive at their visual objective," says Gaeta. "For me, it unlocks my imagination. I see far beyond the simplistic cartoons that are the result of some of our visualization sessions. It's my job to evolve these impressionistic blocking animations into something that has all the matter of nuance and action. I see it as completely vital to getting great shots."

According to Gaeta, if you want to acquire the type of film shots that seem impossible to accomplish, then you need to create and apply an entire palette of previz applications, including design and directorial, originating visual effects concept, sophisticated and general stage planning, and highly accurate camera-control and motion-base exports.

"Bullet Time, as we've approached it in the first Matrix, was basically undoable without visualized strategies," Gaeta notes. "In these sequels, we've turned the previz process into an entirely centralized production nerve center accessible to all facets, from the physical to the digital departments.

"Would I consider making a film like this without using visualization? Absolutely not," Gaeta adds.

Karen Moltenbrey is a senior technical editor forComputer Graphics World.

Photos courtesy Warner Bros. ©2003 Warner Bros.-US-Canada-Bahamas-Bermuda; Village Roadshow Films (BZI), Ltd.