The Hobbit Habit

After five films, one might wonder if Middle-earth has become as precious to Director Peter Jackson and his crews at Weta Workshop and Weta Digital as a certain gold ring is to Gollum and, sometimes, Bilbo. It’s been 12 years now since the first Lord of the Rings film immersed audiences in author JRR Tolkien’s fantasy world. With Warner Bros.’ The Hobbit: Desolation of Smaug, the New Zealanders create Tolkien’s Middle-earth for the fifth time.

 

This film, the second in Jackson’s Hobbit trilogy, continues to follow Bilbo and the 13 dwarves led by Thorin Oakenshield on their dangerous trek to the Lonely Mountain. Also known as Erebor in the language of the elves, the Lonely Mountain is the dwarves’ lost homeland. They lost the land when the gargantuan dragon Smaug claimed the mountain and the dwarves’ forges, set the nearby villages on fire, and used the clan’s immense treasure horde inside the mountain as a bed.

 

As in the first film, actor Martin Freeman plays Bilbo, Ian McKellen returns as Gandalf, and Richard Armitage reprises his role as Thorin. New to the film is Benedict Cumberbatch who voices the dragon Smaug.

At Weta Digital, Eric Saindon was the show’s overall supervisor, with help from Visual Effects Supervisors Chris White, Jeff Capogreco, Kevin Smith, and Matt Aitken. Dave Clayton and Eric Reynolds were animation supervisors. Mark Gee was a visual effects sequence supervisor. Charlie Tait, the head of compositing. And, Dejan Momcilovic, the motion-capture supervisor. As always, overseeing it all was Joe Letteri, senior visual effects supervisor.

Filmed in stereo at 48 fps on Red Epic cameras, the two-hour-and-41-minute film had approximately 2,000 visual effects shots ranging from set extensions and environments to creatures and digital doubles. Notable among the CG environments are Mirkwood, Lake-Town, Thanduil’s realm, and the treasure trove and forges inside the Lonely Mountain. All the actors had digital doubles. Notable creatures include the bear Beorn, the Orcs and the Wargs they ride on, the spiders in Mirkwood, and, most of all, the fire-breathing dragon Smaug.

“Smaug is the main event of this film, and that’s what we were working on all year,” Clayton says.

Designing Smaug

The dragon had made a brief, cameo appearance in the first Hobbit film, but in this one, he commands the center of the entire third act once inside Lonely Mountain. Bilbo is the first to encounter him after the dwarves send the Hobbit “thief” to look for a large, white, glowing gem called an “Arkenstone.” Bilbo finds himself in an immense cavern filled with a sea of gold coins and other treasures. He spots the Arkenstone, but the coins move like sand on a beach when he tries to reach it. As the coins slide, they reveal Smaug’s eye. It’s nearly as large as Bilbo.

Modelers at Weta Digital began work on Smaug before the first Hobbit, referencing illustrations by John Howe who, with Alan Lee, were chief conceptual designers for The Lord of the Rings trilogy and the first Hobbit film. For this feature, Howe and Lee, who had illustrated many of Tolkien’s books, joined the Weta Digital crew.

“John Howe did intricately detailed drawings,” Saindon says. “The dragon’s claws, teeth, tongue, every bit he could.” 

Initially, the dragon had four legs and wings, much like a typical fairy-tale dragon. But, that design evolved into a creature with two legs and membraned wings with bat-like features. “He has to grip things, so we put a thumb and forefingers on his wing that he can use like a hand,” Clayton says. “When he moves around in Erebor, he uses his wing hands on the ground like a bat would, but he can stand up on two back legs if he needs to, as well.

 

When the dragon is angry, he glows from within thanks to a sphere inside his belly. “We light it up to the temperature of fire and it glows through the bones, muscles, subsurface, and scales,” Saindon says. “As he becomes angry, he gets that fire going in his belly.”

ARTISTS IN A newly formed environments group at Weta Digital created CG backgrounds for wide shots such as this and extended sets in many others.

A new liquid-fuel solver sent the fire shooting out the dragon’s mouth. “He sprays fuel using a fluid simulation,” Saindon says. “We plugged that into a fire simulation and ignited it. That gave the fire a sort of napalm look. We could ignite the fire with intricate detail from his mouth all the way out to the point where it hits something.”

Like all digital characters at Weta, the dragon’s internal muscle simulation moved the surface of the creature’s skin. In addition, the creature department applied a separate simulation to each scale as it slid on the skin surface.

Modelers created the overall sculpture within Autodesk’s Maya and then used Pixologic’s ZBrush for the scales. “We had close to a million individual scales on the dragon,” Saindon says. “We tried to build as many as we could. Sometimes in geometry. Sometimes in displacement. When he bends, the scales fold over and slide on top of one another so he doesn’t look like a big rubber thing. Then, we sent him to Gino [Acevedo] for textures.”

Acevedo is Weta Digital’s creative art director and the head of the textures department, a role he has played since the third Lord of the Rings film. On the first two Lord of the Rings, he was a prosthetics supervisor at Weta Workshop.

“We had a team of five texture artists led by [Senior Texture Artist] Myriam Catrin working on Smaug for a good year,” Acevedo says. “We looked at a lot of reptile research from alligators to lizards. We had input from John Howe. Since he and Alan [Lee] have been drawing Smaug for so many years, having his input was exciting. We had snake and lizard skins here and got some silicon molds that we used as reference and for actual textures. We wanted the imperfections. We didn’t want to do anything procedurally. And, Guilluame Francois [senior shader writer] wanted detailed maps to do tricks in the shader. So, a lot had to be painfully hand-drawn. Myriam and the team were hand-drawing scales for months and months.”

 

Each week, the texture artists and shader writers would meet, look at the renderings, and make adjustments. “We put scratches into the scales. We had patterns that ran from the tail to the nose, and on top of those, had scales within scales,” Acevedo says. “Because we could isolate every scale, we had an overall map that changed the specular on each. The map almost looks like a piece of stained glass: We didn’t have one specular sheen across all the scales. It’s broken up; each scale has a different color. Guillaume [Francois] could take that into the shader and apply a different specular response to each.”

Because the team couldn't predict how closely the camera would follow the dragon, artists working in The Foundry’s Mari and Adobe’s Photoshop also painted complex, microscopic details into the skin textures that underlie the scales and added believable complexity even to the beast’s claws and toes.

“He’s an ancient dragon, really, really old,” Acevedo says. “He has big patches of skin about to peel off like a snake, and lots of scars, ferocious scars on his face, because Peter wanted him to feel like he had been battling hundreds of years ago. So, like in real life, when a patch of dry skin is peeled away, that new skin is more saturated with color.”

As a result, the artists painted hundreds of texture maps for the dragon alone, with some as large as 8k resolution. “He’s the biggest creature we've ever done, and the heaviest with all the geometry,” Acevedo says. “But, the best way to work on a Peter Jackson film is to detail the hell out of everything. That way you’re covered no matter what.”

Dragon Moves

To create the dragon’s performance, animators keyframed the digital character. Actor Benedict Cumberbatch provided Smaug’s voice.

“We couldn’t do performance capture because, obviously, Benedict is not a dragon and the motion wouldn’t apply,” Saindon says. “The dragon was keyframed by animators who had video of Benedict they could reference. When he was on the ADR stage [soundstage], we used little Sony handheld cameras to take videos while we recorded the dialog. He got down on all fours and really got into the motion.”

Later, the animators who created Smaug’s performance found that the look in Cumberbatch’s eyes and the way he moved his head was particularly helpful reference.

“Obviously, we don’t have a one-to-one match with a dragon, but we tried to put Benedict’s head nuances and his presence into the scene when we could,” Clayton says. “Without Benedict’s voice and presence, Smaug wouldn’t feel as cool as he does now, but Smaug is an animated creature. We used Benedict’s performance only as a reference for personality; we didn’t apply it directly. Smaug has a crocodilian snout and loose skin around his muzzle and eyes. His head is massive – the size of a van or bigger. You have to articulate it slowly or it breaks the illusion.”

At first, the team didn’t know whether the dragon would talk telepathically, with a jaw that opened for a magical voice, or with full articulation. By applying early dialog recordings of Cumberbatch onto a digital model of Smaug articulated with a simple facial puppet, the animation team created a test of the dragon moving its mouth and lips as it talked.

Bear of a Man

We first see the character Beorn (actor Mikael Persbrandt) in his bear form, a fierce, almost werewolf type of bear with no cuddly teddy-bear appeal.

“He was a big design challenge,” says Dave Clayton, animation supervisor. “Peter [Jackson] wanted the facial features of a bear, but didn’t want him to feel exactly like a bear. We wanted him to be a fearsome creature, not cute and huggy. He’s very muscular. He, Smaug, and the spiders were our main keyframe challenges on this film.”

In the film, Beorn guards his house and protects the dwarves from a band of Orcs, then transforms into his human form. “We spent a long time getting his look right,” says Visual Effects Supervisor Matt Aitken. “We like to reference nature, but as we worked on the shots, Peter felt we were straying into having him look too appealing. Even a grizzly bear has a bit of a teddy bear about him if it isn’t roaring, which is the last thing we wanted. So we made the button on our bear’s nose smaller, made his eyes fiercer, his teeth fiercer, and his fur darker and scruffier. The clincher was not having a plush look to him at all.”

The transformation from bear to man happens quickly in the film, at night, in the moonlight. “We were nervous about how it would play,” Aitken says. “So we initially staged it as a series of glimpses between the trees. Peter told us to strip all the trees back to see it clearly. He said we could add trees later. But the creature department did such a fantastic job, we didn’t add the trees.”

To accomplish the shape change, the rigging crew created a bear rig, a digital-double rig that referenced the actor, and a third rig that could blend between the two. “The rig had to access the transformation across multiple channels,” Aitken says. “There’s the shape of him, and also his textures. His fur has to shrink away. His human hair grows out. His teeth change. His claws become fingernails. What really clinched it, though, was the keyframe animation. You get the sense that he struggles through the process. You see the man appear in the bear early on, and at the end, see a hint of the bear dying out in the man.”

To facilitate the transformation’s timing within the shot, animators could control the change with eight sliders that affected specific regions. The movement of hip from bear to human, for example, might take place a few seconds after another part of the body had already transformed.

Typically, Weta Digital’s creatures with human forms have a common UV layout to easily apply textures from one creature to another. But in this case, the shapes and sizes were different enough that the crew needed to do texture migration. “We set up a version of the transforming creature to have a common place where we could migrate the human textures into a bear-texture space,” Aitken says. “Our fur system already supported the ability to animate the length of fur with texture maps. So, we used the maps to shrink his bear fur and grow his human hair.”
It might seem like a lot of time and energy for a quick shot that occurs in moonlight, but a quick check of the cast list for film three suggests that Mikael Persbrandt might be back.

“I don’t think we’ve seen the last of the bear,” Aitken hints. – Barbara Robertson

All through the sequence, Smaug and Bilbo engage in a testy conversation, and the camera frequently focuses on Smaug’s face. “Even though a crocodile has rigid scales and skin, we decided to move Smaug’s lips,” Clayton says. As is typical for creatures that talk, the rigging team at Weta created an articulated model based on the Facial Action Coding System (FACS), which breaks expressions down into individual muscle movements.

 

“We tried to channel Smaug’s face into our FACS system to keep as many of those controls as we could – lip stretches, brow control, and so forth, so we could animate his face in the same way we could animate Gollum’s face using a similar control set. On Gollum, though, we have upper-lip left and right controls. On Smaug, we have upper-lip mid, back, left, and right to have more fidelity and accommodate his long snout.”

Smaug’s body presented additional challenges. He crawls through the shifting heap of gold coins, articulates with his hands, and presses his hands into the pile causing avalanches of gold coins. “The control system worked for his flight mode, as well,” Clayton says. “We tucked the membrane part of his wings behind his elbow. The challenge was in finding great poses that made him feel predatory and really arrogant. He’s confident, but also suspicious, paranoid. It was an animator’s dream to perform such an awesome character voiced by such a great actor.”

Shifting Treasure

During one shot in the treasure hall sequence with Bilbo, Smaug bursts out from beneath the massive pile of gold coins and slithers around a column. As he moves, coins shift throughout the cavern and fly into the air around the characters.

“In that shot, we simulated 18 million coins at once,” Saindon explains. “To get that scale, we had to write a new rigid-body solver that could move millions of coins quickly. The new solver allowed us to fill the huge spaces with a 

volume of RBD (rigid-body dynamics) coins, not just a texture of coins, and get that sense of movement.”

The need for a new solver wasn’t due solely to the number of coins – the simulation engine needed to move the coins in ways most rigid-body dynamics solvers don’t address.

(TOP) BILBO SEARCHES for a gem among 10 million CG gold coins with rigid-body dynamics. (Bottom) Weta Digital raytraced the dwarves’ gold forge using PR RenderMan.

“Rigid-body solvers are great for destruction, for smashing things to bits,” Saindon says. “But, they don’t work well for things that don’t move. They like to be always moving. We needed to have the coins sit still and not jitter. Rigid-body solvers don’t like to do that.”

Even with the new solver, the team discovered that creating believable motion was a surprising challenge. They put their digital coins on a 30- to 40-degree slope, sent them sliding down, and thought they had created a good simulation.

“Peter [Jackson] said, ‘No, this isn’t what the gold coins would do,’” Saindon says.

 

The team tried again, and again. Jackson wasn’t happy with the result. So, they brought fake gold coins from the set onto the motion-capture stage and videoed them. The coins were plastic coated with metal to look like gold.

“We dumped five tons of this fake gold on a 30-degree ramp,” Saindon says. “It didn’t move. We dumped only a bucket. The coins sat there. We kicked them and only a few moved. It didn’t do at all what we expected. So we went back to the drawing board.”

By increasing the resolution of the simulation result and tweaking the weight and the friction, they persuaded the pile of digital coins to move as gold coins really would. “It’s like sand on a beach,” Saindon says. “You knock off a little and a little moves, but it isn’t a landslide like snow on a mountain. It’s heavy. It has more friction.”

To render the sequence, the team used Pixar’s PR RenderMan, the rendering engine used throughout the film. “It was all raytraced,” Saindon says. “A dragon frame probably took on the order of 12 hours each. The environment, another 12 hours. So, it was close to 24 hours per frame, 48 frames per second. Stereo. Luckily, we have a big render wall.”

The render wall also came into play during a sequence that featured a river of gold. “We created that with our new fluid solver,” Saindon says. “It was a water simulation but with different properties that represent the viscous quality of gold.”

Escape from Thranduil 

Beyond Smaug, Visual Effects Supervisor Matt Aitkin singles out two areas in which the Weta Digital team working on Hobbit 2 advanced the state of the art: water simulation with the new fluid solver and digital doubles.

“The water simulation work is the best we’ve done,” Aitkin says. “It’s the culmination of development work we instigated around the time of Tintin. You see it in the water around [Elvenking] Thranduil’s realm in the northern part of Mirkwood, but the advances really become obvious in the sluice sequence.” That sequence puts the dwarves and Bilbo inside barrels tumbling down a sluice.

As the sequence begins, Thranduil [actor Lee Pace] has imprisoned the dwarves in barred caves within the Wood Elves’ underground environment.

“Our consideration was to make this world distinct from the other underground environments,” Aitken says. “The elves are an elegant race, so we wanted it to be beautiful.” Working with Howe and Lee, they created a digital set that looks carved from a honey-colored sandstone type of rock, and filled the chamber with oil-burning lanterns that gave it a soft, amber glow.

The crew adopted a process similar to that used for other environments in the film to create the digital sets and set extensions.

“The layout department artists blocked out the shapes working from concepts created by our art department,” Aitkin says, “and, by Alan Lee and John Howe, who joined Weta Digital once principal photography wrapped. They’ve been with us all through the second year and will be here next year, as well. They are fantastic.”

Next, the layout department sent the simple geometry to the environments department. “That’s a relatively new department,” Aitken says. “It didn’t exist before The Hobbit.”

Before the first film, Weta Digital relied on a typical pipeline in which layout artists send simple geometry to modelers who, when finished, delivered sophisticated models to texture artists. Matte painters worked separately. Now, modelers, texture artists, matte painters, and layout artists work together in one area to create hero, render-able environments. 

“Digital environments are key functions here, along with creature and digital-double work,” Aitken says. “Especially on these Hobbit films where there are so many. There is a benefit in having one core group of people who sit together and own that role.”

Bilbo arranges the dwarves’ escape from the underground cells and leads them into a room filled with empty wine barrels. The dwarves climb inside, Bilbo releases a ramp, and the dwarf-filled barrels roll down into a fast-moving stream. Orcs arrive, the elves catch up to the dwarves, and a great battle ensues with all the characters fighting each other: the dwarves and Bilbo fighting from within and around the barrels, the Orcs and elves from, mostly, the shore. 

The Rapids

“We shot plates with markers on barrels and added CG dwarves and environments,” says Visual Effects Supervisor Jeff Capogreco. “We shot clean plates augmented with CG barrels, Orcs, and elves that we intercut with live action. We had live-action photography without barrels. And, a lot of the shots were completely CG.”

On an oval stage outside Wellington, New Zealand, the production crew built a circular tank filled with water that made it possible to film close-ups of actors in barrels. “They’d shoot stunt actors wearing costumes with green masks, and we’d replace the heads,” Capogreco says. “But, often, to get the scale right, we went all digital. Once we’re past the sluice gate, all the Orcs are digital, and the elves in the background are digital, as well.”

For the digital water, the crew built a kilometer-long environment sloped downhill to produce the correct water flow. 

THE WATER, DWARVES, barrels, and environment are sometimes real, sometimes digital in the fast-moving sluice sequence.

“The layout and environment departments built each section in its chronological order,” Capogreco says. “They strung them together and moved the world back to its origin, and we ran the simulations on that. We tried to simulate the water as close as possible to live-action reference – the speed, the shading, everything that drove the final look.”

To accomplish that photorealistic look, the crew worked with the R&D department, where CG scientists spent about a year rewriting the studio’s water tools from the ground up.

“We needed to create water that goes from calm to full-on rapids, that can intercut with live action shot from a helicopter or crane, and that marries in custom shaders,” Capogreco says. “We needed more control and the ability to crunch more data sets. With a large ocean, we can break up the surface with SPH [smoothed particle hydrodynamics] noise, but our water had to be simulated down to droplet size.”

Moreover, on previous films, crews creating shots with digital water typically had relied on multiple render passes to create layers of foam, bubbles, and mist. In this film, the crew rendered most shots in one pass with one additional mist layer. The water is all raytraced within RenderMan.

“Our shading team wrote a proprietary shader to create our beautiful water,” Capogreco says. “You can see all the bubbles from the bottom of the water to the top, and they’re all rendered with a single pass. The particles store what we call ‘primvars,’ variables that the shader looks up and shades according to age, velocity, and vorticity. Because everything is custom and internal, we had complete control over the look. If someone wanted to color a bubble with a little more red at some point, we could do that.”

The result is an indistinguishable blend of CG water and digital doubles with live-action water and footage of actors. “When you watch the sequence, the nice thing is that, no pun intended, it all flows together,” Capogreco says. “There’s no beat where you think, ‘This is all-CG water.’ For me, that’s the most satisfying thing after working on the shots for a year. You know water is there, but it just becomes part of the overall image.”

Lake-town

A different kind of water became an important element in the large, canalled city of Esgaroth, otherwise known as Lake-town, the last city before the dwarves reach the Lonely Mountain. A local trader named Bard the Bowman (Luke Evans) smuggles the dwarves across the lake and into town on his barge. Inside Lake-town, tipsy wooden buildings stand on pillars sunk into the lake bed near the shore and on ancient ruins. Citizens chase dwarves through the ramshackle city, as do, eventually, elves and Orcs, too. On stage, the actors walk from partial sets into fully-CG environments.

“Almost every shot has some sense of digital set extensions,” says White. “The lake is CG, and although they filled canals with water for up-close set shots, the water is all-CG beyond. It’s cold, so we added ice flows and mist.”

“Beyond” describes a large area of approximately 50 acres filled with 1,400 highly detailed buildings. “We designed each building to be close to camera,” White says. “And, they’re all unique. The idea is that the city is decrepit. The buildings lean against the old columns of a city underneath, each twisted, tipped over, and contorted. We have paint chipping off the wood, moss, stains – all those weathering details. We made millions of planks.”

LAKE-TOWN’S 50-ACRE digital environment houses 1,400 unique, decrepit wooden structures built into and over CG water, each with enough detail for close-ups.

To fill the city with people, the crew motion-captured actors doing everyday actions – kids playing, fishermen hauling in a catch, women gathering goods, and so forth. The motion-edit department then applied that data to a library of digital men, women, and children, and placed them into the scenes. 

“We tried to have a library that was as diverse as possible,” White says. “We created 10 variations from a generic woman and man that we could dress in different clothing, hats, and hair. We also made digital doubles of all the extras and actors shot for the film.”

Working with the animation department, the crew in the motion-edit department placed many of the crowd characters in Lake-town by hand in order to control when and where they appeared. “Some scenes had only a handful in the background,” White says. “But in others, they might place as many as 50 or 60 characters by hand.” 

Digital Doubles

For digital doubles of the actors, the crew captured their performances and those of their stunt doubles on the motion-­capture stage. To replicate their look, artists created skin textures using a meticulous method that Acevedo first developed for Avatar and has since enhanced: He does life casts to capture fine details, and then uses a unique technique to scan the result into Photoshop and produce displacement maps (see “Animation Evolution,” December 2011/January 2012).

“We even did a new life cast of Ian McKellen,” Acevedo says. “We also cast his teeth and hands, and Keven Norris painted textures for his digital double.” Norris worked on Azog the Orc, too.

The painters additionally worked with displacement maps that provided skin texture from life casts of Martin Freeman, those playing the dwarves, and some of the other actors, painting color variations and other details into the result. “Evangeline [Lilly] who plays the Elf Tauriel has flawless skin,” Acevedo says. “Her skin is so healthy we had to exaggerate the texture or it would have looked like plastic in CG.”

Although the filmmakers typically use digital doubles for action shots too dangerous for actors and stunt actors, and sometimes in very wide shots, Aitkin describes a situation in which a digital double helped Jackson film a cozy domestic scene. The scene takes place at breakfast time in Beorn’s house and includes Gandalf, the dwarves, the hobbit Bilbo, and Beorn.

Beorn (actor Mikael Persbrandt) is a shape changer – sometimes a bear, sometimes a man. When he’s a man, he’s 10-feet-tall. To put the dwarves and Bilbo into the same room as Beorn, the filmmakers used a simulcam technique. 

WETA DIGITAL CREATED digital doubles of all the main actors in the film to use in wide shots such as this and in close-ups such as those in the sluice.

“We had the actors playing the dwarves on an overscale set with giant chairs, so they looked small around the table,” Aitken explains. “At the same time, on an adjacent greenscreen stage, we filmed Mikael [Persbrandt] with a scaled-down camera, so all his moves are smaller. We slaved the camera filming him to the camera filming the dwarves through motion control. You could look at a monitor on set and see Beorn looming over the dwarves.”

Thus, Jackson could shoot the sequence as if he were filming a 10-foot-tall shape-shifter talking to a roomful of dwarves. But, what of Gandalf? His human scale stands him between the dwarves and Beorn.

“The most effective way of achieving the third scale was to replace him with a digital double,” Aitken says. “Our digital doubles are now at a level that makes it possible for Peter [Jackson] to make creative decisions he otherwise couldn’t make. They give him flexibility. Our performance-capture technology, the way we shade the hair, the skin, the cloth simulation – across the board we’re achieving a higher level of quality.”

The same artists and techniques made it possible for the artists to create believable characters that did not need to mirror actors. The Orc Azog, performed and voiced by Manu Bennett, his tortured-looking son Bolg, performed by Lawrence Makaore, and the other Orcs all benefited from the scrupulous work by texture artists, a sophisticated muscle-simulation system that has evolved over many films, performance-capture techniques, and the work of other artists and animators.

“With creatures like the Orcs, our digital performance and digital double work cross over,” Aitken says. “They are entirely digital characters, an extension of the performance work that started with Gollum (The Lord of the Rings, The Hobbit) and Kong (King Kong), and carried on through Caesar (Planet of the Apes), Neytiri (Avatar), Captain Haddock (Tintin). They continue that thread which runs through our work. Peter works with the actors on set, and their performances are clearly there, and we want to honor that. But, we get to realize the appearance of the character carrying out that performance.”

As the artists at Weta Digital move on to film three, The Hobbit: There and Back Again, they will revisit all the technology and techniques developed for the first two films and pull that thread into new state-of-the-art visual effects.

“Working on these big Peter Jackson films is about the most fun we can have,” Aitken says. “The scope is so broad and he has us working on such a high level that we get to operate at the top of our game. I really appreciate it.”

Barbara Robertson is an award-winning writer and a contributing editor for CGW. She can be reached at BarbaraRR@comcast.net.