In 1816, E.T.A. Hoffmann, a German author, wrote a children's story in which a nutcracker, a young girl's favorite Christmas toy, comes alive. The nutcracker defeats an evil Mouse King in battle and whisks Marie Stahlbaum to a magical kingdom populated by dolls. In 1892, the Russian composer Pyotr Ilyich Tchaikovsky and choreographers Marius Petipa and Lev Ivanov turned an adaptation of Hoffmann's story by Alexandre Dumas into the much-loved and often-performed ballet "The Nutcracker."
This year, Walt Disney Pictures released the sumptuous live-action film The Nutcracker and the Four Realms using the Hoffmann story as a starting point. It begins at a holiday party, where a Mr. Stahlbaum gives his youngest daughter, Clara, a locked, handcrafted, egg-shaped box and a note from her deceased mother, Marie. The box is locked. That night, at her godfather Drosselmeyer's Christmas ball, Clara follows a string supposedly leading to Drosselmeyer's gift, but it takes her instead to a parallel world in which the key to her mother's box hangs from a tree. A mouse snatches it before she can reach it and races across a frozen lake. Guarding the lake is a soldier named Phillip, who tells Clara that she is the daughter of Queen Marie Stahlbaum. He leads her to a palace where she learns through a ballet performance about the land she has entered and her mother's role, which she is expected to assume.
Lasse Hallström directed the filming, with Joe Johnston stepping in as director during postproduction and for re-shoots.
Max Wood on the production side worked on pre-production and was the on-set visual effects supervisor throughout filming. MPC's Richard Clegg supervised artists working at that studio's facilities in London, Montreal, and Bangalore who created 95 percent of the film's visual effects - the fully CG realms, a Mouse King made of 60,000 individual mice, 23 waterfalls and a waterwheel, the giant Mother Ginger marionette, and more. Rodeo FX artists contributed effects for shots in the real world, and Luma helped with shots, as well.
"We started with our previs team and got MPC involved early on the environments in the fantasy world, working from artwork from the art department," Wood says. "We knew the Mouse King would be in the movie, so we also got MPC going on the look and shape of him. And, we had Rodeo work on a fully-digital Victorian London."
Setting the Stage
The story begins with the visual effects shots in 1879 Victorian London created by the artists at Rodeo FX. As the camera flies along the frozen river Thames, we see digital ice-skaters below. Animators created the ice-skating performances from data captured from roller-skaters. Motion capture also provided data for people in market stalls, horses and carriages, and carriage drivers.
Once Clara enters the fantasy world, approximately 500 artists at MPC took on the visual effects work, creating environments, characters, and effects.
A huge palace dominates the fantasy world, and the fantasy world dominates the film. Surrounding the palace are four realms placed almost like points on a compass: Lands of Sweets, Flowers, and Snowflakes, and a mysterious Fourth Realm. Separating each realm is a CG river complete with waterfalls. All the realms but the fourth are 100 percent CG. The fourth realm was a set extended with CG.
Sugar Plum with Clara
Because the environments for three of the realms were digital, the filmmakers utilized Ncam Technologies' system for virtual production on set. Actors had props, but all the backgrounds, including the ground they walked on, would be CG.
"There were moments where Lasse [Hallström, director] had actors point to the palace, but all they could see was bluescreen," Wood explains. "So we would do an overlay of the palace and place a marker on the bluescreen where the palace would be. The camera operator only saw the bluescreen, but the director could see a monitor with the background filled in. We'd feed that onto an iPad as well, so the actors could see what the background would look like in the movie."
For the fourth realm, the filmmakers created a forest on the stage with trees imported from Italy. Outside the forest was bluescreen that MPC artists replaced with a CG forest stretching into infinity.
As for the other three realms, "We laid out and planned where everything would be," Wood says. "With Ncam, we could move the world farther away, or to one side or another. Then we could pass that along to MPC, showing them the new position of the palace from the camera view."
Worldbuilders
At MPC, artists started with the basic geometry and a 3D map created in pre-production, and began designing a workflow for building the final environments.
"We took the 3D geometry, rendered it from key angles, and sent it to the art department," Clegg says. "They painted-in correct perspective and scale so we could flesh out the environments when everyone signed off. We broke everything into quadrants and tiles, even the water. We never wanted to box ourselves in. So, we built the environments in almost a watercolor way. We'd lay down the base and build the world layer by layer. When shots were signed off by everyone, we'd finish areas that needed to hold up to the camera, and sometimes do custom set dressing for close shots. In the end, we could put the camera wherever we wanted."
For the palace exterior, the team referenced the ornate onion domes of the Kremlin. Inside the palace, the modelers worked from a limited set to create the "engine room," and within that room, the toy machine that turned toys into people.
"Even though there was a plate and a practical set that looked fine, sometimes for continuity reasons the camera changed position, so the set became fully digital," Clegg says.
23 Waterfalls and a Waterwheel
Outside the palace are 23 waterfalls and a giant waterwheel, all CG, that power the engine room and the giant toy machine. Effects artists at MPC used Side Effects' Houdini and Autodesk's Maya Bifrost to create the waterfalls - Houdini to send the water flowing down and Bifrost to create volumetric atmospherics at the base. The waterwheels were trickier.
"The waterwheel designs were very much fantasy," Clegg says. "They look like a Victorian train station mixed with a giant factory that has a network of pipes and aqueducts. They have pipes shooting water from the top that spins the wheel. There was no way our waterwheels would work in the real world. Any reference we could find had waterwheels driven from the bottom, from a river. So, it was tricky to sell the speed and weight. We ran Houdini simulations, but it took a lot of art direction, too. We spent a lot of time worried about how they'd look, but also how we would render them."
Indeed, the biggest challenge for all the environments was in rendering them. MPC uses Pixar's RenderMan RIS.
"We had all these trees, buildings, lollipops, flowers, chimney steam, smoke from the different towns, water, reflections," Clegg says. "It was definitely a challenge. We could put each kingdom, the palace, and the water on layers. But we also fit quite a lot into memory."
Adds Wood: "We made sure we cached as much geometry as possible to render as one big pass, all one render, so it looked like it all held together. Obviously, we had deep passes to tweak, but we learned how to optimize and render as one thing rather than piece it together."
Mouse King
The biggest challenge and, except for Mother Ginger's marionette, the biggest CG character in the film is the Mouse King, which appears in two main scenes: one in the forest and another toward the end of the film. He's made of 60,000 mice. He lurks in the forested Fourth Realm.
"His design evolved through the whole movie," Wood says. "We started with one idea of what he would look like but realized he's more of a concept than a character. He's not a stable being: He continuously changes. He's different in every shot. He could be whatever he liked. He can walk through a tree, so we had to work out how the volume would stay. In some scenes, his ears disappear and reappear. His shoulders become part of his neck."
His default was a bipedal character with a tail, but his shape changed as required. At first, modelers built a detailed creature with thin legs like a mouse, an inner core filled with mice, and mice on the surface. The team simplified that model and made his legs feel like they were strong enough to support his weight.
"Animators had a base mesh and a base rig they could use to animate him," Clegg says. "It was flexible and stretchy, so they could build different shapes. They'd focus on the shape and form, on action points for the character through the scene. In one shot, he engulfs the tin soldiers. In another, he picks up Clara."
For motion, the animators referenced the dancer Lil Buck (Charles Riley), who performs a street-dance style called "Memphis jookin" that originated 30 years ago. The film's production notes quote Riley describing the dance thusly: "It used to be called the gangster walk. It was really simple, but it has evolved into complex movement with intricate footwork. It's like Michael Jackson times 10: There are slides and glides and fun toe spins, ticking and pushing of the feet, and the shoulder bounce. It's all about the bounce."
As Mouse King glides along, mice continuously run up his legs to his head, fall off, and climb back on.
"He isn't a solid object," Clegg says. "He's an amorphous turbulent thing, a pile of mice moving around. Animators used Lil Buck's performance as inspiration to create a base action, and they would apply simulations to make him more fluid."
First, they ran a cloth simulation on the mesh driven by animation parameters that gave him what Clegg describes as a thick, syrupy movement.
"It's like a custard that managed to stand up," he says.
Then, they applied a crowd simulation on top that was driven by the performance speed and directed by guide passes and curves on the cloth-simulated surface. Falling mice became rigid bodies, and rag-doll physical simulations took over. Once a mouse landed, it jumped back onto the Mouse King and ran back up as if it were on a conveyor belt.
Mother Ginger
"We had paths with offsets so the motion looked chaotic," Clegg says. "The mice all had different levels of detail depending on how close they are to camera."
The team used Maya for animation, rigging, and cloth simulation, then fed the mesh into Houdini for the crowd simulation using mice instanced onto the surface.
"We then exported the result into our crowd format and rendered with RIS through [Foundry's] Katana," Clegg says.
Mouserinks
In the film, the character Mouserinks controls the 60,000 mice tumbling along in the Mouse King. He works for Mother Ginger, and he's the mouse that stole Clara's key. Is he a villain? That remains to be seen. On set, he was sometimes a gray, 3D-printed mouse and sometimes a stuffy. In the film, he's CG.
"He's a mischievous mouse, bigger than an actual mouse," Clegg says. "He spends a lot of time on shoulders and hands. We treated him as a standard digital character. The challenge was in figuring out how mouse-like he should be. The directors wanted him to be cute and fun, but not overly done."
Mother Ginger
Although inspired by a character from the ballet, Mother Ginger is quite different from the ballet character whose little gingerbread children emerge from a giant gingerbread-house skirt, and dance. In this film, Mother Ginger has been banished from the realms and become regent of the Fourth Realm. She appears as a terrifying 40-foot-tall giant with a 30-foot-wide circus-tent skirt. But appearances are deceiving: The giant is a marionette. Mother Ginger (Helen Mirren) is actually the puppeteer inside.
Shiver
Although a practical marionette skirt, torso, and head were on set during filming, in most of the feature the marionette is fully CG, and when practical, enhanced with CG. CG steam leaks from the steampunk pistons and chimney stacks of the practical marionette, and CG arms and mouth enliven the giant prop. Rather than children dancing from under the skirt, polichinelles tumble out.
"They're like Russian dolls," Wood says. "They can split in half, open up, and jump out of each other. They had stunt performers play polichinelles in the film, but in some shots, they're fully CG. We motion-captured acrobatic clowns bending and doing flips and turns."
Tin Soldiers
Everyone in the realms was once a toy that someone transformed into a living being by a toy machine. The key Clara searches for not only opens the egg-shaped box, it turns on this toy machine. So, of course, the villain in this film uses the toy machine to quickly create an army from toy tin soldiers. The soldiers are CG. To create them, MPC artists referenced concept art and antique toys.
"We tried to make something that changed from a six-inch toy into a seven-foot-tall character and still have details like scraped-off paint," Wood says.
They also tried to preserve the toy-like movement of the six-inch toys without compromising the actions of the seven-foot soldiers they'd become.
"The toy tin soldiers have simple joints that rotate on only one axis," Clegg says. "We tried to give our seven-foot tin soldiers enough flexibility to do things the toys couldn't do, but not so much it looked obvious. The gag is that they don't quite convert into people yet because there's such a huge volume. They're almost like a robot army. They don't change as they grow. They just scale up to be seven feet tall."
Mother Ginger vs. the Tin Soldiers
In one fully CG scene, the Tin Soldiers attack Mother Ginger's marionette, thinking she's inside. In fact, another character is driving the huge marionette through a small forest; the marionette towers above the trees.
"We have tin soldiers climbing up the circus-tent skirt like the zombies in World War Z," says Clegg, who had worked on that film at MPC. "She's bashing them on the head with a tree. It's like whack-a-mole. It was tricky because the soldiers are climbing over a cloth surface and are pretty limited in their range of motion."
The team started with animation for the marionette and with hand animation for several hero soldiers on top of the skirt. Then, working in Houdini, they did a cloth simulation of the moving tent, popped the soldiers on top, and ran crowd simulations for close to 100 soldiers climbing up the skirt.
"Where there were intersections, we ran a secondary cloth simulation on top of the cache with a weighted input," Clegg says. "We simulated-in regions and applied restrictions, and those finer details fixed most collisions. When we needed to do manual cleanup, we'd raise the cloth up or lower it down to avoid any visual problems. It was painstaking work to make it look like the soldiers weren't intersecting with the skirt."
Creating a fantasy is not easy. But, although some movie critics haven't appreciated this new take on the Nutcracker story, the film's visuals have been highly praised.
"There was a lot of subjectivity in the work on this film, a lot of artistic freedom," Clegg says. "Every shot is different. We'd give the moviemakers things and feed off their reactions to bring alive their imaginations."
Barbara Robertson (BarbaraRR@comcast.net) is an award-winning writer and a contributing editor for
CGW.