Imageworks manipulates photographic elements, and puts it all togather in stareo for Tim Burtons Alice in Wonderland.
Listen to the visual effects crew at Sony Pictures Imageworks who worked on Disney’s Alice in Wonderland describe postproduction for this film, and you’d think director Tim Burton had dropped them down a rabbit hole, too.
“I like doing something different, and this film begged to use bizarre techniques that we haven’t done before,” says Imageworks’ senior visual effects supervisor Ken Ralston, who counts an Oscar for the pioneering film Who Framed Roger Rabbit among the five he has won for best visual effects.
“Everyone was not normal,” says Carey Villegas, referring to the live-action characters in the film. “We scaled or manipulated everyone in some way.”
Villegas was one of two VFX supervisors who worked with Ralston on Alice. Sean Phillips, the second visual effects supervisor, adds, “I’ve never seen anything like this before—mixing and matching a wacky blendo of stylized photography with CG characters. We’ve done that on a small scale, but never before had a CG character next to a live-action character next to one with a CG head next to one scaled up or down, all having a conversation together. We had to make them live inside that stylized world and not get lost in it.”
And this time, they had themselves to blame, at least in part, for the wackiness.
From the top down: Wooden frogs painted green give Helena Bonham Carter a proper eye line on set. Imageworks filmed Carter’s head at 4k resolution, shrank her body to HD resolution, and blended the two. The frogs, birds, and monkeys are CG; the courtiers are filmed elements. At bottom, once animated, the animals get fur, feathers, and textured skin. At left, the final image.
|
We’re All Mad Here
Ralston, who joined the project before Burton had finished the script, worked with the director to design what became the “wacky blendo” look of the film. “We decided we didn’t want to go down an all-mocap look, like Beowulf,” he says. “We wanted to see the actors.” And with a cast like this, it’s no wonder: Johnny Depp (Mad Hatter), Mia Wasikowska (Alice), Helena Bonham Carter (Red Queen), Anne Hathaway (White Queen), Crispin Glover (Knave of Hearts), and Matt Lucas (Tweedledee and Tweedledum).
“Tim [Burton] wondered what we could do to put actors in a limbo area, where they were not quite real, but real,” Ralston says. So, Ralston worked with artists at Sony Pictures Imageworks and character designers Michael Kutsche and Bobby Chiu, who he found after an Internet search, to give Burton interesting ideas.
The team decided to enlarge Johnny Depp’s eyes, swell Carter’s head and shrink her waist, and fiddle with color timing for Anne Hathaway. They sat Crispin Glover’s head on a tall CG body, and pasted Matt Lucas’s face on two digital Tweedles.
“When we started, Tim [Burton] wanted to be sure we didn’t just have live-action actors in a virtual world with CG characters,” Villegas says. “We did all that manipulation to bridge the gap and bring the two worlds together.” The only real person is Alice, in that the artists didn’t manipulate her image, but she’s never the same size twice.
In addition to photo manipulation, Imageworks created some 30 CG characters, including the White Rabbit, Cheshire Cat, Jabberwock, Bandersnatch, the Red and White Knights, March Hare, and more—every creature in the film that isn’t human is animated. Artists in the studio blended all these CG characters with the manipulated shots of the actors and the photo/CG characters inside virtual environments. Then, they converted everything into stereo 3D. And all on a short schedule.
“I’m always amazed by the sheer volume,” Phillips says. “We had 1700 shots with fully fleshed-out CG environments, and we had a year to do the whole thing.” Among the tools used by artists at Imageworks were Autodesk’s Maya and Mudbox for modeling, rigging, animation, cloth and hair simulation, and some effects; Side Effects Software’s Houdini for effects; Maxon’s Cinema 4D for projection painting; Adobe’s Photoshop for matte painting; The Foundry’s Nuke, Autodesk’s Flame, and Imageworks’ own Katana for compositing; Katana for lighting; and Imageworks’ Arnold for rendering. Other studios helping on the project, largely with the stereo 3D conversion and environments, were CafeFX, Matte World Digital, Nvizage, and Svengali, with The Third Floor providing previs.
Once Alice arrives in Wonderland, the environments are nearly always CG. The scale of the computer-generated plants in this garden helps us understand that she’s very small.
|
Come, We Shall Have Some Fun Now
The film begins and ends in Victorian England, in live-action scenes, but most of the story takes place in Underland, also known as Wonderland, where Alice finds herself, a decade after her first fall, once again on a trippy adventure with her wacky childhood friends, all of whom managed to be on set with her during filming. Even the CG characters.
“We recorded the animated actors in London, but we had voice talent on the set to have eye lines,” Ralston says. “Alice is six inches, two feet, eight and a half feet, and, for one moment, 20 feet tall, and then at the end, her normal height. So, the eye lines changed from scene to scene. If we didn’t have the real voices, we had voice talent in green suits.”
Because most of the film takes place in CG environments, the sets were green with green props arranged to follow production designer Robert Stromberg’s plans. “We had at least three, 360-degree greenscreen environments 60 feet tall by 300 feet long,” describes Villegas. “We had them on rails so we could open and close them as needed. We lived in a greenscreen world for many months.”
The actors could view a miniature to see what the environment would look like, and when Alice changed sizes, green platforms on set helped the actors have the right eye lines. “The scales are all nuts,” Ralston says. “It took an awful lot of tweaking.”
In one shot, for example, a tall Alice holds the Mad Hatter’s hat. “Mia [Alice] is shorter than Helena [Red Queen], and Alice is supposed to be eight and a half feet tall,” Villegas says. “So we shot Mia on a platform. In post, we scaled everything so her feet touch the ground, but her eye line stayed in the same place.”
In another scene, a tiny Alice jumps onto the Mad Hatter’s hat, crawls around the brim, drops to his shoulder, and then the two walk into the CG forest. “We used motion-control photography for that shot,” Villegas says. “It was the only time we shot motion control.”
To make the scaling possible, the crew filmed scenes with Alice in odd sizes, all shots with Helena Bonham Carter, whose head needed to double, and other shots involving abnormally sized humans with a Dalsa Evolution 4k digital camera; a Genesis HD camera handled the rest. By scaling Carter’s body down to HD resolution, for example, they doubled the size of her head, which they had shot at 4k. Actors working with Carter pretended her eyes were about two inches higher when they looked at her. The same technique made Alice shrink and grow.
A World of My Own
In the film, Alice begins her journey at a garden party. Led by the White Rabbit into the forest, she falls down the rabbit hole and lands in a round room, a set built at normal size. Then, she shrinks to two feet tall and walks into a CG garden that looks like an English garden in disrepair. From that point until the end of the film, with few exceptions, the environments are digital: the mushroom forest where she meets Absalom the CG caterpillar and is chased by the vicious CG Bandersnatch, the Tulgey Wood where she meets the CG Cheshire Cat, a big desert environment made from projected matte paintings, the environment around the tea party (that takes place in a set), the ruins of an old temple, the Red Queen’s castle, and more.
“There are a lot,” Phillips says. “And we had the challenge of using the environments to reinforce Alice’s size.” When she’s six inches tall, big dandelion seeds float through the environment. When she’s large, she ducks through CG doorways. The modelers created details based on how close the camera would move, the scale, and the amount of time the characters would spend in the environments.
So that Burton wouldn’t be shooting blind in this greenscreen world, Imageworks modeled every CG environment before he filmed the live-action actors. “The idea was that Tim could virtually see the characters in that [digital] environment on set,” Villegas says.
To help Burton see Alice and the other characters in their Wonderland environments as he filmed the actors, the crew used two solutions for real-time camera tracking. The EncodaCam system from General Lift and Brainstorm tracked anchored cameras and inserted characters shot on the greenscreen stage into the backgrounds from the camera’s viewpoint. An optical system developed by Imageworks’ R&D department tracked handheld cameras by focusing on markers in a grid on the ceiling. In addition to giving Burton and his director of photography a way to see the actors in the forest, in the Red Queen’s bedroom, and various other places, Burton’s editorial department used the composites as a guide when they cut the movie.
What Size Do You Want to Be?
Every shot in the film involving live-action photography moved through a pre-composition department led by Villegas. First, though, the matchmovers tracked the camera using the 4k-resolution frames. “The matchmove systems don’t really know about resolution,” Villegas says. “So we could scale the film back and maintain the matchmove.”
Once the pre-comp department had the camera, the artists could prep the plates. “On a traditional movie, pre-comp is usually the easy part,” Villegas says. “On this movie, pre-comp is where some of the really hard work was done.” The artists had to key the characters as usual, and then had to fit the variously scaled live-action elements into CG environments.
“We had to reframe most of the shots because of the scaling,” Villegas says. “And, we made those decisions in the pre-comp phase. [DP] Darius Walsky did the initial framing on set, but he was gone by this time. So, we worked with Tim [Burton] and Ken [Ralston].”
When the artists had positioned all the characters in their final sizes and reframed the shot as if the characters had all been photographed in those sizes, the photographic elements moved on to layout for integration with CG props and the CG environment. From there, the rough compositions moved to animation and stereo layout. “At this point, the pipeline became a little more standard,” Phillips says.
For characters like the Knave of Hearts and the Tweedles, which were part-photographed actors and part-CG, the animators could see photographic elements glued to 2D cards that always turned toward the camera within a 3D scene. “Our biggest rigging challenge was incorporating the photo-heads,” Phillips says.
At top left, Matt Lucas on the right and a stand-in on the left wear the Tweedle suits for motion capture on set, then switch positions. At top middle, artists apply Lucas’s eyes and mouth to the CG characters and use video from each performance as reference. At top right, animators tweak the body mocap and technical artists add cloth simulation. Above, the final image.
|
This Is Impossible
To create the Knave of Hearts, for example, animators needed to fit a seven-and-a-half-foot-tall CG body onto Crispin Glover’s moving head. On set, Glover sometimes stood on a platform, and sometimes walked on stilts. He wore a suit fitted with Xsens Technologies’ inertial motion-capture system MVN (formerly Moven) to give animators the option of using motion-capture data for his body. “It would have been really hard for the animators not to have had the head in the 3D system,” Villegas says. “They needed the system to visualize.”
Similarly, when Alice rides a CG Bandersnatch, animators had the live-action photographic element on a card in the 3D environment. And, for the Tweedles, a card that tracked with Matt Lucas’s shoulders made it easier for the animators to perform CG bodies and place the photographed eyes and mouth on the CG heads.
“We aren’t projecting photographs onto geometry,” Phillips says. “We’re trying to orient the geometry to the photographs. But, we gave the animators the flexibility to change the photography if they needed to.”
As it turned out, the animators did not use motion-captured data for the Knave of Hearts. “Crispin [Glover] practiced walking on stilts and got it down, but when you translate that into a properly portioned [tall] human, the walk looks funny,” says David Schaub, animation supervisor. “So we animated him from the ground up using his performance as reference. It was difficult to animate his body under the head, which was extracted, stabilized, and on a card in Maya. It was the same with the Tweedles.”
For the Tweedles, the team mocapped Lucas wearing a padded costume on set. The animators used that data and reference footage to help perform the two characters’ bodies. “We restricted his movements with a pear-shaped foam suit, so his arms didn’t move past the limits of the CG character,” Villegas says. “He performed with a stand-in. When Tim was happy with Tweedledee, [Lucas] switched places and became Tweedledum to get facial performances for both characters.”
To make those performances believable on the CG characters, the animators worked with fully rigged faces using reference footage as a guide. “Even though we used Matt Lucas’s real eyes and mouth, all the stuff around them needs to move,” Schaub explains. “His eye movements need to pull the skin around the eyes. It looks weird, but in a good way.”
Because Alice’s size changes from sequence to sequence, the CG Tweedles often needed to grow and shrink, and stereo made all this even more complicated for the animators, who needed to consider Alice in depth as well. “If a CG character holds hands with a live-action actor in 2D, you can cheat that,” Schaub says. “But in stereo, the depth gives away the cheat.”
A Grin Without a Cat!
At the peak of production, Schaub led a team of 63 animators: specialists who concentrated on the hybrids, and others who animated the non-human CG characters. “We had so many specialized characters that appear in only a small cluster of sequences, it didn’t make sense to have a team working on them,” Schaub says, “although in the throes of production, it became a melting pot. We had nine months to do the animation.”
The characters ranged from the whimsical, fanciful Cheshire Cat to the vicious Bandersnatch, to natural animals like a horse and a bloodhound. “Tim wanted to shoot the movie and not deal with animals,” Schaub says.
Riggers started with generic bipedal rigs for the humans, chess pieces, monkey, frog footman, and executioner, and with standard rigs for the quadrupeds, such as the dogs, pig, and hoofed creatures.
“On past shows, like Narnia, we rigged characters so they could do calisthenics,” Schaub says. “No matter how you posed them, they’d be anatomically correct. But it turns out you need only 20 percent of those cases. So gone are those days.”
All the animals in Alice are CG, including this bloodhound cuddled by Anne Hathaway. Imageworks used cloth simulation to help make his floppy ears and loose skin believable.
|
Instead, the riggers created simpler rigs for what they knew the character would do, rather than for anything a character could do. “We don’t rig for every eventuality and carry all that stuff around,” Schaub says. “We have a rigging team ready to build one-off features on a shot-by-shot basis.”
The Bandersnatch provides an example of such a one-off rigging solution. “He has a bear body and a bulldog face, with shark’s teeth,” Schaub describes. “The teeth pierce his lips, which was already tricky from a rigging perspective. When he licks Alice’s wound, we had a one-off rigging solution to get his big tongue through the teeth.”
Similarly, when they realized the horse needed to talk, riggers built solutions only for the phonemes they needed for that specific dialog, rather than a rig that would allow the horse to say anything.
At left, precomp artists frame the Knave of Heart’s (Crispin Glover) head within a CG environment. At right, the final image shows the Knave’s seven-and-a-half-foot CG body attached to the head and the environment fully textured, lit, and rendered.
|
For the animators, the fanciful characters provided some of the most interesting creative challenges. “At first, we had the Dormouse run on its hind legs, and the White Rabbit did the same, Tom and Jerry like,” Schaub says. “We put that in front of Tim [Burton] to see his response. I think he was entertained with it, and we went down that path for a while.” After Burton finished filming, though, he asked for a subtler performance.
“He didn’t want Roger Rabbit against the human performances,” Schaub says. “He wanted to be sure all the characters were in the same world. So we started going more and more subtle. Nothing overacted. Just the bare minimum to make the point.”
The Cheshire Cat, for example, one of the first characters the team animated, moves its tail to accent the dialog spoken by actor Stephen Fry; it acts like a cat, not a human, even when it floats, appears, and disappears. To create emotions for a character with a perpetual grin, the animators played with extremes for the smile. “But much of it is in the eyes,” Schaub says. “As he goes in and out of light, we played up the idea of brightening the eyes. The iris lights up in a special way that suddenly brings a spirit of life into the character, as well.”
Similarly, the animators played Absalom the caterpillar, voiced by Alan Rickman, with a subtle, disdainful attitude. “He wears a little monocle on his squished nose, so we have him look down his nose at Alice,” Schaub says. “We make sure his body undulates as he talks and exhales, so you can feel a gelatinous movement in time with the dialog, all through keyframe animation.”
The tense and nervous March Hare was one of the last characters the team nailed. “Tim [Burton] described him in a way that reminded me of Charlie Chaplin in Modern Times,” Schaub says. “We emulated Chaplin’s funny walk for a while, and gave [the Hare] a nervous twitch. And then, we pulled back.” The animators put the tense creature on all fours and had it hop like a hare, but every once in a while, it has a big nervous spasm.
(Top left) Animators used the Cheshire Cat’s tail to help emphasize the dialog for this character, which has a perpetual grin. (Top right) The tense and nervous March Hare is about to lose control once again.
|
Many of the CG animals are under the Red Queen’s rule. She hates them and proves it by using the animals as furniture, tools, and servants. Flying birds hold up chandeliers, capuchin monkeys are bellboys, frogs are waiters, and flamingoes are mallets for a croquet match. A fish butler scoots around on its tail. Most of these characters have fur or feathers, or wear clothing, and the studio has been working on these modeling, grooming, and simulation problems since Stuart Little.
“We’ve settled into a blend between hair and feathers that we call ‘furthers,’ which are like fat pieces of hair,” Phillips says. “We do the majority of feathers that way, although flight feathers on the wings are geometry. For the bloodhounds, we did a cloth-like simulation on the skin to get the jiggle, and for his ears.” One of the most impressive CG creatures in the film is the Jabberwock, which Alice must confront at the end, and which the team based on John Tenniel’s illustrations of the dragon-like animal in the original Alice in Wonderland book. “Even though he looks like he should be able to fly, we make sure he can’t,” Schaub says.
While Alice confronts the Jabberwock, a battle rages in the background between the red and white knights. Inspired by chess pieces, the white knights look like human figures with alabaster armor, which put the studio’s renderer to the test. “They’re translucent, so we used subsurface scattering in Arnold,” Phillips says. “It looked very cool when it was done.”
The wide and thin red knights, by contrast, look like playing cards and are made of slightly bendy, interleaved steel plates. “Everything in the battle is CG,” Schaub says. “We had animators tear loose with lots of battle business.” Libraries of fighting characters created by the animators powered through the battle with help from the studio’s own crowd animation software.
The combination of familiar techniques, challenged by creative designs with new techniques developed to handle the photographic manipulation and hybrid blends unique to this film, made working on Alice in Wonderland an amazing adventure for the crew.
“After all these years doing so many movies, this might be the most creative thing I’ve ever done,” Ralston says. “And maybe the most fun. It’s been a blast.”
|
Like No Place on Earth
Director Tim Burton filmed most of the movie on greenscreen stages. In the film, the actors appear alongside CG characters in CG environments. Knowing this project would be shown in stereo 3D, the crew on Alice considered shooting the live-action elements with a stereo 3D camera, at first.
“We started that way, but once we realized how much photo manipulation we’d do, we knew we couldn’t use a standard HD camera,” says Sony Pictures Imageworks visual effects
supervisor Carey Villegas. “At the time, there were no 4k stereo camera rigs on the market.” But, they wanted to use a 4k-resolution camera so that in postproduction, they could scale some elements down to HD resolution and leave others, such as the Red Queen’s head, large.
Moreover, says senior visual effects supervisor Ken Ralston, “this is so complex, and to have shot a true 3D movie would have caused so much grief. Also, we thought that having stylistic control of the conversion would be interesting.”
Interesting and tricky. “When you combine CG characters, and environments, and photographic elements, and try to dimensionalize everything, making everything feel like it’s in the same environment is complicated,” says Corey Tumer, Imageworks’ stereo 3D supervisor. The photographic manipulation and scale changes magnified the complexity.
Tumer describes one shot by way of example. In it, the Hatter, actor Johnny Depp, is trying various hats on the Red Queen, Helena Bonham Carter. Carter’s head is twice its normal size, and Depp’s eyes are larger than normal, too. The Red Queen is looking into a mirror. Her back is to the camera; we see her reflection. On screen right, courtiers photographed by a second-unit team speak to the Red Queen. The environment is CG with photographed props—hats, hat stands, and furniture. CG monkeys hop around and give photographed hats to Johnny Depp on screen left.
To create this shot, the pre-composition team cleaned up the photographic elements for the Hatter and the Red Queen shot on greenscreen stage, scaled the Queen’s head, added the courtiers, and framed the shot. The stereo team added dimension, but it was a mix-and-match, back-and-forth workflow.
For example, to give the photographed courtiers dimension, the stereo team gave the pre-comp team a scene with stand-ins and a stereo camera. The pre-comp team projected the photographed images onto simple shapes, a pair for each image (one for the left eye, and one for the right). The stereo team then converted those shapes into more detailed geometry, a technique they used throughout the film for photographed elements—the Red Queen and the Hatter in this scene, Alice in other shots, and so forth.
“We swap in shapes to give us curvature, such as a shoulder that protrudes, a nose, a huge belly, and then re-project the image and warp it to fit,” explains Tumer. Similarly, they projected the photographed hats onto 3D shapes when the CG monkeys pull them off the hat stand.
“As the monkeys hopped across the screen, it felt as if they walked through one of the hat stands,” Tumer says. “So we animated a stereo camera to cheat them forward, and then animated it back to get the connection with the Hatter.”
Multiple stereo cameras with specific interocular and convergence settings came in handy throughout the film. In this scene, the Red Queen, Hatter, and props have one set of stereo cameras. The courtiers have another. The CG monkeys have a third. Similarly, when CG characters surround Alice, she has her own set of stereo cameras and the characters have another.
“I originally thought I would use one set of cameras and it would be fine,” Tumer says. “But I found that the projected elements are so flat to begin with, if I overdo the stereo for them, it helps sell the shot and you no longer feel like they and the CG elements are two separate things.” |