Author JRR Tolkien filled his beloved fairy tale The Hobbit with memorable, mythical characters and creatures. Director Peter Jackson and the crew at Weta Digital, under the leadership of Senior Vi-sual Effects Supervisor Joe Letteri, gave those creatures life—in three-dimensional stereo on movie screens. Big. Small. Funny. Frightening. Some resemble animals. Others re-semble humans. And at least one, Gollum, is unforgettable.
In Peter Jackson’s The Hobbit: An Unexpected Journey, the first of a three-part series, all those characters and creatures are CG, created by artists at Weta Digital. Indeed, in many scenes, only the actors are real, and sometimes they are digital doubles. So many digital characters run, hop, gallop, fight, and they that the studio split the work of animating them.
Head of creatures, Simon Clutterbuck, ticks them off: Tree trolls, giant eagles, spiders, wolf-like Wargs, Gollum, a glimpse of the dragon Smaug, hundreds of goblins, Orcs, hedgehogs, rabbits, horses, elves, bats, birds, insects…. “There’s a lot,” he says. “I’ve lost track. We built a huge amount of stuff.”
Dave Clayton’s animation team focused on the goblins, trolls, stone giants, and eagles in sequences supervised by Jeff Capogreco (Gollum, trolls), Chris White (goblins, stone giants), and Kevin Smith (eagles).
Eric Reynolds’ team performed the Wargs, Orcs, dwarf doubles, eagles, spiders (which will appear in the second film), and the wizard Radagast’s rabbits, in sequences supervised by Matt Aitken (rabbits) and Smith (Wargs, Orcs, eagles). Visual effects supervisor Eric Saindon pro-vided overall guidance. All the animators worked with digital doubles.
Whether eventually made of stone, skin, or feathers, the characters all start as puppets cre-ated in the modeling department. Approximately 70 artists led by supervisor Marco Revelant sculpted some 1400 individual objects for the film, many of which were fully costumed digital doubles. Modelers at the studio work in Autodesk’s Mudbox, which originated at Weta Digi-tal, and Autodesk’s Maya. They create topology and extract displacement maps with Headus’s CySlice, and fashion costumes with Marvelous Designer from CLO Virtual Fashion.
Supporting and extending this software are proprietary tools and techniques for creating faces, muscles and skin, fur, feathers, and hair, and for quickly producing a variety of creatures from one framework. Weta Digital’s researchers and technical team developed software specifically for this film and updated tools and techniques used for previous films.
In The Lord of the Rings trilogy, Gollum’s face had 5,000 polygons and displacement maps. The Hobbit’s Gollum, pictured at top and bottom, has 48,000 polygonal faces, which gave the animators more control. Inside the CG creature is a deformation system that produces high-resolution muscle and skin simulations.
Gollum Anew
The character Gollum provides a good road map for describing how character creation and animation has evolved at Weta Digital. He appears in one sequence in the film, a sequence that many critics have called the best moment in the movie. Some lament that Andy Serkis, who plays the CG character, did not receive a best supporting actor nomination.
In the sequence, while trying to escape from the goblins, Bilbo [Martin Freeman] has stumbled and fallen into a cave. There, he finds Gollum’s “precious” ring, the powerful “One Ring” in The Lord of the Rings, although Bilbo doesn’t know this yet.
Before the ring, Gollum was Smeagol, a hobbit. Under the ring’s influence, he has become an emaciated, slimy creature who lives on cavefish and small goblins. His two personalities, past and present, are in constant conflict as we see in the sequence: Gollum threatens Bilbo— he would probably like to eat him. But, Bilbo intrigues the Smeagol side of the creature by suggesting a game of riddles. If Bilbo wins, Gollum will show him the way out of the cave. If Gollum wins, “We eats him,” the creature hisses at Bilbo.
Weta Digital created, and Serkis first performed, Gollum for the second and third films in The Lord of the Rings trilogy. At that time, Serkis performed the creature during filming on location, and then later, on a motion-capture stage, had to duplicate that performance. Now, for this film, the studio captured Serkis’s performance during filming using the same tools and techniques with which they had captured Serkis’s performance as Caesar in
Rise of the Planet of the Apes.
“We wanted to capture every nuance Andy [Serkis] had,” Capogreco says. “The LED camera mounted on the front captured his face, and the remotely synced cameras tracked all his physical movements, his fingers, hands, feet. Literally every nuance. The animators then pushed Gollum’s face into contortions no human could quite do. They also worked on his eyes, to give them movement of some sort even when nothing else was moving. And, especially because of the stereo, they made sure Gollum’s toes were on the ground and his hands wrapped around the right things.”
Data captured from Serkis’s performance on set moved into Weta Digital’s motion-edit department. There, technical animators translated the clouds of data points into animation curves that move the digital puppet. “When all the individual controls on the puppet are mapped to corresponding bits of motion data, the puppet comes to life,” Capogreco says.
For the modelers, the challenge was creating a familiar face and body for fans of The Lord of the Rings trilogy, even though Gollum was 60 years younger and the artists could take advantage of improved technology and tools.
“It’s been a painstaking process,” Revelant says. “Ten years ago, we created Gollum using geometry and painted displacement maps. We reworked him completely, using real geometry rather than displacement. To do that, we looked at how he was then, rebuilt on top of that model, subdivided the geometry, and applied the displacement as geometry. For his face, we went from 5,000 polygons and displacement to 48,000 polygonal faces to give the animators more control. We wanted to bring him up to speed with our new, improved facial tools.”
Capogreco and Animation Director Dave Clayton started working on a test shot with Gollum in September 2011, and continued for another two or three months.
“When the first renders came out, we knew we needed to do some work on his facial rig and muscles,” Capogreco says. “Gollum set a huge benchmark 10 years ago, but we needed to bring him up to today’s standards and perhaps push past [that]. I also spent a good month or two with Joe [Letteri] looking at shots. We tried making him sweaty. We made his teeth grosser. Then we circled back, but not completely. In The Lord of the Rings, he had been tortured and beaten, and we didn’t want to make him that messed up. But, we poured in subtle details like age spots.”
To tweak the motion-captured data and add to Serkis’s performance, animators manipulate a simple puppet that has rigid pieces of geometry parented to the bones. “The animators get what most people think of as a previs puppet,” Clutterbuck says, “a simple puppet, not even skinned. We plug the output from animation, the curves, into our deformation system.”
The technical staff at Weta Digital had developed the deformation system to do highresolution muscle and skin simulations for Apes; particularly for Caesar, the chimpanzee star. The creatures group on Hobbit used an updated version of that system.
With rare exceptions, the crew doesn’t create rigs with skin clusters attached to joints. Instead, the creature group maps the animation curves onto a high-fidelity skeleton, builds and attaches muscles to these highly-detailed bones, adds a fat layer and skin, and runs the in-house finite-element simulation. The animation curves move the bones, and that movement triggers the muscle simulation. The muscles drive the fascia (the fat layer) and the skin, and thus, the simulation system deforms the shapes. Weta Digital’s Clutterbuck, James Jacobs, and Dr. Richard Dorling received scientific and engineering awards from the Academy for developing the “Tissue Physically-Based Character Simulation Framework” (see Here).
“People typically think of rigging as skin clusters and muscle primitives,” Clutterbuck says. “We don’t do that. We no longer have separate simulation and rigging groups. These days, everything—all the muscles, all the skin—is simulated. The muscles are elastic— big finite-element models with collision detection—so they tend to perform well, even when the animators go crazy. It’s a very anatomically- driven framework; very real world.”
Animators performed all the bipedal characters and some animals using data captured from actors. Top, Actor Barry Humphries wears a facial-capture rig and marker dots to perform the Great Goblin, as did Andy Serkis to perform Gollum. Humphries’ facial expressions move the Great Goblin’s digital face, shown in a pre-rendered form (right of top image) and fully rendered at bottom.
Anatomically-driven even for such an odd, scrawny creature as Gollum. “You actually see finer details on Gollum than on the big, bulky guys, like the trolls, which have no muscle definition,” Capogreco says. “We made sure that when this little, skinny guy stretched his arms, the right muscles, the tendons in his neck, moved accordingly. He’s lean and cut, but underlying everything, you need muscles. He demonstrated some beautiful skin sliding and little subtle nuances. I think the biggest thing we improved upon, though, were his facial expressions to map Andy’s face onto Gollum. Andy really pushed himself.”
For faces, animators use a system based on blendshapes. The creature group then adds simulation elements on top. “The facial system is not broken down into types of expressions, just muscle groups,” Clayton says. “We can pull on certain muscle groups to get expressions, and it all combines nicely on the face. We could observe all of Andy’s details and replicate them.”
To give Gollum these extreme expressions and take advantage of the simulation tools, the crew rebuilt the facial puppet used a decade ago. The new system provided extra controls for additional detail, particularly around the creature’s cheeks, eyes, and lips, to take advantage of the fidelity now available in motion-captured data and to replicate Serkis’s performance.
“Ten years ago, the movement we could capture was more linear,” Capogreco says, “not as fluid. The capture is so much more advanced now. And, I think because of the technology and because he has gotten more comfortable with it, Andy [Serkis] knows we can capture a wider range of expressions and more subtlety. He gave the modelers and animators a run for their money. On one shot, we spent two or three weeks modeling corrective face shapes to capture what he was doing—closing his eyes, sputtering his lips.”
Animators also made sure Gollum’s big feet landed on the ground and didn’t go through the geometry. And, sometimes, they took Gollum beyond what Serkis’s supple face could do. “There’s one shot where Gollum realizes that Bilbo has the ring,” Capogreco says. “He turns around, shaking. He’s enraged. There was so much minute detail in Andy’s performance, but Peter wanted it amped up. He wanted to see the tension build in Gollum’s eyes and a shudder across his face. So that was hand animated.”
In the original version of Gollum, the crew had created his eyes using a sphere with textures and fake caustics. Later, for Planet of the Apes’ Caesar, they created eyes using a different method. “We give the eyes a lot of depth using a multitude of layers,” Capogreco says. “We layer together the major areas with the right shading.” For Gollum’s skin, Weta Digital’s Textures Supervisor and Creative Art Director Gino Acevedo, who was one of the painters for the first Gollum, led teams that used scans from various types of real-world skin to create the look of Gollum’s surface (see “Animation Evolution,” December 2011/January 2012).
“We have fine pore detail that wasn’t in the original Gollum,” Capogreco says. “And his shoulders by his neck almost look like turkey skin. Like a really senior person, but pushed beyond. There was a great deal of re-painting to add little moles and skin defects. And, we gave him peach fuzz. On some close-ups, you get kicks of hair on the tips of his ears.”
Gollum also got a haircut. “In the original version, his hair was down his back,” Capogreco says. “Now it’s to his shoulders.”
For the first Gollum, the hair team animated cloth strips to create his hair. For this film, his hair is dynamic; that is, simulation systems driven by animation curves move his hair. “We used a mix of tools for hair simulation,” Clutterbuck says. “We used an elastic rod system for the hedgehog spines and for the Warg fur because these big, crazy, dog-like things have hackles like furry manes. We also wrote tools in Maya that allowed us to represent hair as nCloth volumes or as individual strands. Then, we chuck everything into our in-house system called Barbershop, which allows us to plug in any dynamics we like.”
At top, groomers used Weta Digital’s custom Barbershop system to style the Wargs’ hairy coat. Animators moved the creatures using data captured from dogs. At bottom, modelers and creature developers built all bipedal characters, including this troll, from one generic template.
Wargs
Barbershop had a particularly large role to play in creating the rough coat for the huge, wolflike creatures called Wargs, a more hyena-like version of which had appeared in The Two Towers, which was the second film in The Lord of the Rings trilogy. In Hobbit, the Wargs attack the dwarves in a fast-paced action sequence.
“Peter [Jackson] had an idea in his head for the Wargs in Rings, but he was never quite able to achieve it,” Saindon says. “Our technology for hair and fur wasn’t up to par at the time. We stepped it up a notch and got to the point where they are what Peter was thinking they should be: a combination of a wolf and a lion, with thick hair and a bit of a mane. Creepy, scary creatures with retro-reflective eyes.”
To help the groomers achieve the look Jackson wanted, the technical crew upgraded Barbershop, the hair and fur system used for Planet of the Apes. “Instead of the old-fashioned way of drawing curves and putting guide curves in the direction you want them to work, modelers have the ability to groom the curves,” Saindon explains. “They push the curves around with tools similar to blow-dryers, brushes, combs, and scissors, in the same way a hairdresser would work. Barbershop even has gel.”
As they had for Gollum and would for the digital doubles, other creatures, and animals, the animators created the performances using motion-captured data. For the Wargs, they captured data from Alsatian dogs brought into the studio for that purpose and from a dog that often visited the stage.
“We completely covered the dogs in Lycra suits to do the data capture,” Reynolds says. “We captured some nice motion, like how they turn and shift weight, that’s hard to do in keyframing, and we used a lot of it. When you have animators who need to perform 20 photoreal dogs in a scene, they instantly want to find some motion capture that would help.” Because the Wargs were five times larger than the dogs, the animators slowed down the captured motion data by 15 to 20 percent to add mass and weight, and had the Wargs turn more slowly. They also adjusted the postures in the data captured from dogs, which were uncomfortable wearing the Lycra suits.
“The tough thing with the Wargs was that there was always a heavy amount of keyframing rather than motion capture,” Reynolds says. “It didn’t help that most of our shots had 20 Wargs in them, and that every fourth Warg had an Orc rider. It was like a British foxhunting group, but with all Wargs rather than dogs and horses. The foreground ones would run to attack.”
The biggest Warg, a white creature that stands six feet tall at its head, carries the biggest Orc. Azog is a seven-foot-tall white Orc, and the main villain in the film.
Bipedal Dudes
On location, an actor in prosthetics played the Azog character, but later, Jackson wanted a different look, so the character became digital. “We painted out the first actor and replaced him,” Saindon says. “Azog’s performance is all based on motion capture that we did later on our motion-capture stage with another actor. The Orc is really creepy-looking, not quite human. He has no hair whatsoever. Big, sharp teeth. Spread-apart eyes set back in his head. Self-inflicted scarring that covers his body. Because he’s a white Orc, we played the subsurface scattering a little deeper than normal so he looks like maybe he’s underground a lot.”
Weta Digital built the bodies for bipedal characters, including the Orcs, digital doubles, goblins, and trolls in this film, from one template called “genman.” Genman is an ongoing project, a template for a male human, that the modelers and creature group base on data collected from a gymnastics instructor. Using realworld data from motion capture, various types of measurements, and scans, the crew builds and simulates the motion of bones, muscles, and fat, and validates the system they build digitally.
“We’ve done everything to this guy,” Clutterbuck says. “We’ve done full-body MRIs. We had his hands MRI’d in different poses to validate that our hand rigs match real-world poses. We’ve cyberscanned him, measured him, put him in a suit tailored for him and then taken the pattern apart to make sure our cloth simulation works. Genman is our template for bipedal dudes.”
To create a digital double, then, or a bipedal synthetic creature, the crew manipulates genman until it fits the required shape.
“For digital doubles, we have a model made with photo reference that has a polygonal skeleton modeled from a cyberscan,” Clutterbuck explains. “Not Maya joints. An actual anatomical skeleton. Using the outer skin and inner skeleton, we approximate the fit inside, warping everything, the way muscles attach to bones, the tendons, the entire setup. And, we do a bit of would be. Because we know genman works, we know we’ll be in a good ballpark whether we’re making Captain America or a dwarf. We don’t have to start from scratch. We get genman, do the warp, and we’re up and running.”
Because Weta Digital builds all creatures great and small with a muscle-simulation system, it also has a “genhorse,” created in collaboration with a nearby veterinary college, and a “gendog,” first created for the character Snowy in The Adventures of Tintin. “We didn’t do an MRI for the eagles,” Clutterbuck says, “but we found lots of repositories with CT and MRI data, and our system mapped almost one to one. We use MRI data as the basis for all muscles— real-world data that we can almost plug straight in.”
Gobs of Goblins
The generic templates help the crew more easily set up a creature. Other tools help them duplicate the result with variations. “We wrote new tools to populate the film with characters at the same quality as the ‘hero,’ ” Clutterbuck says. “Although we generated all the Orc and goblin variants from one ‘hero’ Orc and one ‘hero’ goblin, we don’t delineate between ‘hero’ and ‘background’ any more. We build one guy and use a tool set to create others procedurally. What’s cool about this is that if we change the ‘hero,’ the changes propagate out to all the others.”
The largest goblin and the one with the starring role was the toad-like but nine-foot-tall goblin king. “He had amazingly gross features, like massive wattles, that made him a lot of fun to work with and animate,” Clayton says. “We polished off the rough edges from the captured body movements and made him feel big and heavy, and we added extra ideas with keyframe animation to give him a stylized motion and identity.” Actor Barry Humphries provided the Great Goblin’s voice; Terry Notary, the motion-capture data.
Surrounding the Great Goblin, which was a unique model, is a horde of equally disgusting but smaller goblins that were surfaced, like the king, with pus and boils. To create these creatures, the crew produced 13 variants from one version fitted with a skeleton and muscles. “We do that with a script,” Clutterbuck says. “We transfer the basic setup to other characters of various sizes, and warp all the muscles and skin to fit. We don’t make 13 unique character rigs. In fact, when we make the ‘hero’ guy, we only make the right-hand side and then generate the left procedurally. We build an arm, a leg, and the right side, hook it up, and then asymmetrically warp the whole thing. There are 130 goblins in the film, and the whole build was run by two people.”
Similarly, the R&D department developed software for defining and creating a variety of rigged faces for characters built with a common topology.The FaceMixer program, developed by Alex Ma, Marco Barbati, and JP Lewis, improves upon simple blending techniques for crowd generation.(See Here)
“The system is like a mixer,” Revelant says. “You can specify a number of variations and it will automatically create that number of new characters. Or, you can drive the variations through sliders. For example, you can use 25 percent of the nose and the left eye of one, and 25 percent of the nose of another.”
This gave modelers the ability to design and create variations quickly. Even though the faces looked radically different, they all included complex facial rigs that incorporated 100 connected blendshapes or more. Some goblins, for example, have no nose or lips.
“Other systems based on blendshapes that work this way cut from one area of influence to another,” Revelant says. “But this system takes into account the surrounding area, so you never break the model. There is never an area of discontinuity. You end up with a new model, organically solved, with a new facial rig; it creates the complete blendshape tree.” And, because a hero character is the starting point for the variations, each variant has enough resolution in the body and face to step into a starring role.
At top, Weta Digital’s award-winning “Face Mixer” software helped the crew quickly set up infinite face variations for the hordes of goblins. At bottom, the trolls use the same muscle and skin simulation system as Gollum.
Fat Trolls, Stone Giants
Facial animation, it turns out, was the biggest challenge for the three giant, flabby, CG trolls: William, Bert, and Tom, who appear in Hobbit. “Before now, they just grunted and roared,” Clayton says. “But, in Hobbit, they converse with one another, have arguments, and confront Bilbo. And, of course, there’s a big fight scene with the dwarves in which they roared and screamed, so we needed good facial animation with convincing dialog.”
Oddly, three of the actors who played the dwarves also doubled as the trolls, providing performances for the director and actors during the live-action shoot. “Parallel to that, on the motion-capture stage, Peter [Jackson] captured the trolls and recorded their voices,” Clayton says. “Peter went through the performances he liked and handed us the selects. I was able to use the motion they captured as the basis for their animation. The trolls are four meters high (approximately 13 feet), though, so we needed to add a lot of weight.”
The trolls were small, however, compared to the stone giants. In the film, the dwarves encounter the stone giants during a rainstorm while travelling through Misty Mountain. They emerge from what looked like a solid mountain and attack each other by throwing boulders, catching the unsuspecting dwarves in the middle.
“In a way, they were simple in terms of animation,” Clayton says of the giants. “They had no facial expressions, and they were slow. The challenge was to give something made of stone enough dynamic motion to make the sequence exciting but still make them feel heavy. We had to articulate the movement of their arms and shoulders without causing intersections.”
Eagles
Intersections were also a problem for the eagles, which, although large for their species, sit at the opposite end of any measurable scale from the stone giants. Heeding Gandalf’s call, the eagles fly into a sequence of approximately 120 shots to rescue Bilbo and the dwarves from an attack by the Orcs and Wargs; they snatch the wizard and his intrepid trekkers from trees they climbed to escape. Some eagles carry the rescued dwarves in their talons. Bilbo and Gandalf ride on the eagles’ backs as they swoop over the beautiful New Zealand landscape. Digital doubles and live-action actors filmed on a motion base helped the animators create the sequence.
Reference footage provided clues to the eagles’ performances, as did the eagles in the Rings trilogy. “Because the birds are so big, we had to make sure the poses were really strong,” Reynolds says. “The bigger challenge was the feathers, which are always a science experiment. The way they collide always affects us. We wanted to add a gentle flutter through the wings, and even in the main wing, the feathers bend a bit. So it doesn’t take much to end up with intersections. We animated the shapes and the flutter, and then the creature group simulated more flutter and ‘un-intersected’ everything.”
To bring the eagles up to date, the creatures group re-engineered the rig as well as the feathers. Although the 13 eagles were basically the same, the crew differentiated them with random feather colors and by changing the look of each eagle’s face.
Kevin Smith, who had worked on the eagle shots in Two Towers, supervised sequences with the eagles for The Hobbit. “We redid everything from The Lord of the Rings,” he says. “Chucked it out. They’re new from the ground up. For the feathers, we kind of based what we did on the falcon we created for Tintin. Previously, every feather on our birds was polygons with texture and opacity maps. This time, we grew each feather as physically correct as possible. We made the feathers with hair; they had a spine with hairs growing out the side.”
To groom the feathers, the creatures group used new, custom software called Plumage. “At render time, Plumage gives you any kind of features you want,” Smith says. “For far-away birds, we could use polygonal feathers if we saw no benefit in rendering a million hairs.”
The eagles were one of the few characters that didn’t start with motion-capture data; the sequence in which they soar was one of the last to wind its way through the pipeline. “It was a nice way to finish the animation on this movie,” Clayton says, “just before we had all these high-adrenaline shots when the dwarves are chased. And then these beautiful wide shots of the eagles. It was lots of fun to animate.”
At top, the wizard Radagast cures a sick little digital hedgehog, created by the artists at Weta Digital and keyframed by animators. At bottom, a digital bird flies toward Radagast, looking for the nest hidden in the wizard’s hair under his cap.
Teasers
Animators also keyframed the performances of the rabbits, hedgehog, spiders, and the dragon Smaug, characters that played relatively minor roles in this film. The wizard Radagast [Sylvester McCoy] treats a sweet, but sick little hedgehog and cures the digital creature. Spiders climb over his house. And, the rabbits pull Radagast’s sled as he races across the country to draw the Wargs away from the dwarves.
“We had a slightly feral, giant Belgian rabbit that we used to model the look of our rabbits,” Aitken says. “We didn’t want them to look beautifully groomed or pampered. We wanted them to have a bit of character. We have great animation of them running, jumping, and pulling the sled, all keyframed. In the second film, the spiders will take over the forest, and the dwarves will have a big encounter with them. We’re paving the way for the detail that will be in films two and three.”
So, too, with the dragon Smaug, whose appearances bookend the first film. “In the animation takes, you could see Smaug clearly, but we used lighting, smoke from his breath, to tease out only parts of him. His tail snaking out through a destroyed façade, a foot here, a bit of belly there. We will probably do an extensive rebuild for film two.”
And again, no doubt for film three. The journey taken by scientists and artists at Weta Digital to make creatures that look and behave like those in the real world (whether the creatures are figments of someone’s imagination or not) is not unexpected.
Barbara Robertson is an award-winning writer and a contributing editor for Computer Graphics World. She can be reached at BarbaraRR@comcast.net.