Heralded as the most artistic use of stereo 3D since
Avatar, and perhaps even including
Avatar, Martin Scorsese’s love letter to filmmaking takes place in 1930s Paris, as seen through the eyes of a boy and realized as if filmed on an early 20th century movie set.
Hugo, based on the award-winning children’s book by Brian Selznick, stars Asa Butterfield as Hugo Cabret, the orphaned son of a clockmaker who now lives in a secret part of a Paris train station. Hugo’s father left him a broken automaton, and Hugo believes that if he can repair the machine, a small mechanical man, he might bring back something of his father. To operate the automaton, though, he needs a key, and as if by magic, Hugo meets Isabelle (Chloë Grace Moretz), a girl with the key. But, the real key to the story’s secrets and to the filmmaker’s vision is through Isabelle’s godfather, a toymaker named Georges (Ben Kingsley). The toymaker, we will realize, is Georges Méliès, a pioneering filmmaker who instilled the movies he made between 1896 and 1914 with cinematic versions of the illusions he had created in his magic theater shows. He invented special effects. But, driven out of business by larger studios, Méliès became a toy salesman at the Montparnasse train station.
In the beginning of the film, we see a vision of Paris enhanced, as is much of the film, with visual effects used to mimic and augment traditional special effects. In an aerial shot of the city from above the Arc de Triomphe, time-lapse photography of traffic on the 12 streets that radiate out from the center circle give the sequence a mechanical quality. As the camera pans past the Eiffel Tower, we see the hint of a clock mechanism.
“We wanted to plant something in your head so the later dialog will make sense,” says Rob Legato, second unit director and visual effects supervisor. The later dialog is a bit of philosophy Hugo shares with Isabelle: that machines are never built with extra parts; that all machines have only the parts they need to run and no more. He posits that if the world is a machine, he must be a part, which means there is a reason why he exists.
“We wanted to create a subconscious visual of that philosophy,” Legato says, “of Paris as part of a mechanism, so the audience has that in mind when he says his dialog. It’s a touchy kind of thing. Delicate. But John Knoll and Industrial Light & Magic did a fantastic job.”
Ben Grossmann led the visual effects teams, working from Pixomondo, which handled the majority of the shots. Nvizage developed the previs, Yannix helped with matchmoving, ILM created the opening sequence, Lola “youthenized,” Matte World Digital produced matte paintings, and Uncharted Territory built a scene on the banks of the river Seine in Paris. Paramount Pictures and GK Films produced the movie, which Paramount is distributing.
All told, the feature contains 850 VFX shots. “We did every trick in the book,” Grossmann says. “The film is a homage to Georges Méliès, so we did the visual effects checklist. In stereo.”
At its core,
Hugo is a story of parts fitting together, of art and craft. And so, too, the making of the movie—beginning with the use of stereo 3D. Legato, credited with creating the virtual production for
Avatar, has worked with Scorsese on
Shutter Island,
Aviator, and other films. He won an Oscar for
Titanic’s visual effects and received a nomination for
Apollo 13. And, he helped Scorsese design
Hugo.
Artists at Pixomondo created most of the shots in the film, which include digital environments
to extend sets, such as the train station (at top), period tweaks to create 1930s Paris (at bottom),
and dozens of visual effects that pay homage to Georges Méliès’ illusions.
Stereo Design
“We planned [stereo 3D] from the beginning,” Legato says. “And everyone was on board. [Production designer] Dante Ferretti designed the sets with depth, [cinematographer] Bob Richardson lit the scenes with depth, Marty [Scorsese] directed the scenes and blocked stereo out as another tool to tell the story. We were all blown away. You can’t add 3D later. It’s like any other piece of art. It has to be planned from the beginning.”
One rainy day in New York City, Scorsese, Legato, and others screened 3D movies from the ’50s in a private theater, movies that been made in 3D but never shown in stereo because the craze had ended. They also watched
Avatar, Dial M for Murder, and 2D moves from the ’40s and ’50s, especially those directed by Carol Reed, such as
The Third Man and others that featured children.
“The fun part about working with a director like Marty is that he adores the history of moviemaking, and this film is about the history of moviemaking,” Legato says. “There’s a sense of reality in the films back then that changes the story. It’s hard to describe. You just feel it.”
With the help of Pixomondo and the other visual effects studios, Scorsese embraced that sense of reality and deepened it with stereo. “It’s hard to separate one from the other now,” Legato says. “We had stereo in the forefront and the back of our minds in every scene, every edit, the way we lit the scenes…it all became part of the mix. Everything was designed, viewed, and staged for the dramatic value of 3D; the depth became part of the storytelling.”
Legato provides an example: “We have a little boy in a 1932 Paris train station, in overwhelming surroundings. So we use stereo in those shots to emphasize the size and structure and largeness of the building against the smallness of the boy. When you block out the scene, as soon as you see it in depth, it alters the way you consider it. Maybe a wide shot will sell the shot, maybe there is something interesting that you want to look at for a long time. It’s a cumulative thing.”
As was true in the early days of filmmaking, Scorsese shot most of the movie on soundstages. He used production facilities in the UK, and much of the visual effects work involved extending those sets and building virtual environments for previs and then later for the final shots. “We rebuilt Georges Méliès’ original studio on a back lot at Shepperton Studios [in Surrey, England],” Legato says, “constructing it to the exact plans, and then photographed it for real. It was a great moment. One of the thrills of moviemaking is to create history and walk around in it. But, we didn’t build much of the train station, and Marty didn’t want to walk onto the stage guessing what the shot would be.”
Thus, previs helped Scorsese and Legato design shots prior to set construction, and see digital environments in shots with sets that the visual effects crew would extend or create. “Nvizage did previs on set and prior to production,” Grossmann says. “We had mechanical representations of common camera equipment so Rob [Legato] could direct a shot with lots of visual effects in previs. We could show Marty [Scorsese] what we were thinking, and he could pre-approve an edit of the sequence. And, if Marty wanted to design a shot before he built a set, he could operate the camera virtually.”
To shoot the film, the crew used a Fusion 3D system from the Cameron-Pace Group. Before production, artists from Nvizage reproduced the sets to scale digitally and loaded them into Autodesk’s MotionBuilder for real-time playback. Then, during filming, motion-control encoders mounted to the camera equipment fed the movements into MotionBuilder.
“It was similar to the virtual camera system used for
Avatar,” Grossmann says. “We had encoders on camera cranes, dollies, pan-and-tilt heads, anything that moved. Wherever Bob Richardson moved the camera, our real-time CG matched it and replaced the greenscreen. So, Marty could see the train station, the trains, whatever, anywhere the camera pointed, with a real-time composite of the actors. If the cameras pointed toward 500 extras walking around in front of a greenscreen, he would see the city of Paris and bridges behind the extras.”
This process of providing directors real-time composites of actors in virtual backgrounds has become a familiar part of filmmaking these days. However,
Hugo’s director was Martin Scorsese, who is anything but typical.
“This wasn’t a documentary,” Grossmann says. “It was a movie someone would make on a movie set. We might be panning to follow Hugo, and Marty would say, ‘I want the Eiffel Tower over here, and maybe over here we’ll see the Arc de Triomphe.’ If he was in the moment, he might walk to the visual effects tent and previs what a shot would look like if the stage didn’t have a roof. He’d tell the actors where he wanted them to be, and then tell us what he wanted in the set, and we’d design the shot while he was shooting.”
Previs from Nvizage and on-set composites helped director Martin Scorsese and senior visual
effects supervisor Rob Legato think about how to film shots with stereo 3D, even narrative
sequences such as this with Hugo (Asa Butterfield) and Isabelle (Chloë Grace Moretz).
Matching the Vision
All the information gathered on set—the witness camera footage, the data from the motion controllers on the camera equipment, the shots from the stereo camera that Scorsese directed—went to Pixomondo to help augment the matchmoving and camera tracking. “For every camera position, our visual effects wranglers would feed the data into the system and create an [Autodesk] Maya file that showed where all the cameras were and where they moved,” Grossmann says. “We’d know plus or minus one degree where the camera was. But, it took several months to develop the matchmove pipeline to get all the cameras tracked.”
Pixomondo’s Beijing studio did much of the matchmoving using a customized version of Andersson Technologies’ SynthEyes. Yannix also did matchmoving for the project.
“On some productions, the directors don’t embrace the medium—they direct as though they’re shooting a traditional film,” Grossmann says. “But Marty was passionate about shooting in stereo. He’d ask, ‘What is the most amazing shot we can do in stereo?’ We’d have these famous actors, Christopher Lee, Ben Kingsley, and Marty would say, ‘Ben, you did good. I just need to do another shot with a slight stereo adjustment.’?”
All those tiny adjustments made it difficult for the matchmovers later. “In stereo, the interocular difference between two cameras is so precise you can see a difference of a quarter of an inch,” Grossmann says. “The entire city of Paris can look like a tabletop set if you’re not careful. It took months to work out all the relationships—the cameras are two inches apart, and the left camera panned this degree at this frame and that degree at that frame. There’s no forgiveness in stereo.”
Meanwhile, at various Pixomondo offices, artists began building sets and set extensions and creating the effects, with Grossmann parceling out the work by sequences and specialty. “Some offices specialize in animation,” he says. “Another might be good at effects and destruction; another might complete the lighting, rendering, and compositing. Much of my job was deciding what went where, but most of the assets started in London.” There, modelers worked from blueprints received from the art department, and then distributed assets to other offices.
Most of the Pixomondo offices have Maya-based pipelines with Chaos Group’s V-Ray for rendering, but in some, the artists used Autodesk’s 3ds Max, as well. In addition to the train station—the concourse, lobby, tunnel, clock tower, and so forth, inside and out—the artists created the trains and several sections of Paris.
“The big problem for our asset team was that in visual effects, we’re obsessed with making things realistic,” Grossmann says. “So, the asset team built the train station as planned, and then they’d hear, ‘Why don’t we knock down that wall and get more trains in here?’ Or, ‘In this shot, we’ll remove the roof.’ It drove them bonkers.” The answer was to break everything into components that the artists could turn off and on, and move around.
For textures and reference, the artists had footage shot on location in Paris, and firsthand information from two visual art directors who were on location. “They’d see the materials and textures, and sit with Dante Ferretti and do concept work,” Grossmann says. “They became immersed in the visual guidelines.” With the movie set in the 1930s, one challenge was to create materials and textures that made the digital assets look as if they were new in 1930, or built before and weathered appropriately to that age. Concept art and production paintings that the art directors created helped the visual effects artists create the look Scorsese wanted.
“They helped keep the creative consistency,” Grossmann says. “Continuity was out the window in major ways—the Eiffel Tower moved where it needed to be, some routes made no sense at all, the train station would look different in some shots—but there was consistency in that everything looked good. That was the continuity.”
The second challenge for the artists was in understanding how to achieve the look of movies from the early 20th century. “The hardest part and the most exciting part for the artists around the world was the exploration,” Grossmann says. “In most movies, you’re doing something like swinging Spider-Man across a bridge. For the artists on this movie, it was never as simple as, ‘Here’s your desk and your shots.’ It was, ‘Here’s your desk and here are 16 hours of highlight reels, some books, and a thousand images of old sets, old trains, old train stations.’ No one cranked out work for weeks, sometimes months, until they got into the mood and tone and look. And then, so much of this movie is a homage. An artist might present a shot and point to something that was distracting, and we’d say, ‘Yes, but it’s distracting on purpose because it references this old film, this old clip.’?”
The lighting artists had similar challenges. As always, they would light the scenes to be photographically real, but their reality needed to be a film shot on a back lot in 1930. “This wasn’t an available light film,” Legato says. “It was a lit movie, and the lighting is part of the storytelling. It’s not real life. So, we had more than one sun. We put lamps behind windows, arc lights behind alleys. It took a while for the artists to get it because it isn’t what we’re trained to do. We usually try to fool the eye that something is hyper-real. We were still making it photographically real, but the photograph had a tone to it. So we’d show people examples, tear sheets, clips from old movies.”
Making it even more interesting for the lighters was that the angles might change from one shot to another, as if the sun moved 180 degrees. “It works because the shots are beautiful,” Grossmann says. “It all feels the same, but if you mapped it out, you’d see that it’s all over the place.” Knowing this, the visual effects crew didn’t bother shooting chrome balls on set to gather HDRI and match the lighting.
“We realized that if Bob Richardson lit something, he’d light what’s there,” Grossmann says. “If there were five people in the room, he’d light those five people. So, if we added a glass roof, a train, and a luggage cart, it wouldn’t do any good to have HDRI because if those elements had been on the set, he would have lit it differently. We had to match his intent.” Similarly, the artists needed to match Richardson’s intent in all the CG shots that had been impossible for Scorsese to shoot traditionally.
Lighting artists at Pixomondo learned that the continuity in this film was in its consistent beauty.
Rather than trying to match the lighting on partial sets, they discovered how to mimic the
cinematographer’s intent.
Magic Hour
In addition to set extensions and virtual backgrounds, much of the visual effects work centered on Méliès’ illusions. “As the film starts to explore who Georges Méliès is, we see shots that are magical in nature,” Grossmann says. “I could talk for hours about all the little magic tricks. Hundreds and hundreds of shots. By the time we were done, I realized we had done every trick in the book. They’re not like cool magic-wand gags. They all have grounding in old film tricks and in some part of the story. We did all the classic cinema tricks from modern times to today, and pushed beyond anything done before. Miniatures. Digital characters. Stop motion. Time-lapse photography. Persistence-of-vision animation. Matte paintings. Motion-captured characters. Iris wipes. Morphs. CG augmentation. Even the choreography of a cross-dissolve became a new art form and became visual effects. It was a homage to the kind of work Georges Méliès did, but in a modern-day fantasy film. And we created all those tricks for stereo. I’ve got all my passport pages full now.”
In one scene, the children open a secret box that causes an explosion of CG papers to fly out. The images on the papers represent the collected work of Georges Méliès. They swirl around the room in a way that creates an optical illusion, the perception of animation. It’s a persistence-of-vision trick, like a flip book, but a 21st century visual effects version.
In another scene, Hugo fixes a mechanical mouse, a mouse that, when Méliès winds it up and places it on the table, spins around, wiggles its tail, and looks up and down. The crew used stop-motion animation for the mouse, shot it in stereo, and then augmented it with visual effects.
For a montage that shows the degeneration of Méliès’ studio from happy success into post-World War I bankruptcy, the artists mimicked time-lapse photography using computer graphics to create the images.
“The movie is full of these things,” Grossmann says. “We’d take sections of sets and performances, string them together, and choreograph them as if they were one shot. In some shots, we’d have two minutes of visual effects strung back to back. We didn’t have to do a CG tsunami. But, we had CG crowds, fire, snow, wind, water, steam, smoke—a crazy amount of effects. It was humbling to do this while referencing and studying someone who invented the genre. Méliès’ work was pretty miraculous. When we studied his work, sometimes it took days to figure out how in the hell George Méliès did this.”
In this film, the innovation was in making the visual effects created to bring Méliès’ illusions to life seem real, and to do so artfully. “We used visual effects and stereo 3D not as separate items, but as a tasteful, integral part of the storytelling, as important as music and lighting and acting,” Legato says. “Our innovation is in appreciating the art of filmmaking by using the tools that used to blow us away with how clever and technical they are, with, now, how beautiful they are.”
Barbara Robertson is an award-winning writer and a contributing editor for Computer Graphics World. She can be reached at BarbaraRR@comcast.net.