Once again, all matter of characters are brought to life in the film – some played by actors and others by digital characters. Directing the film was Shawn Levy, with Erik Nash as production VFX supervisor. Visual effects facilities included Digital Domain, MPC, Method Studios, Zoic, Cinesite, and Lola.
Digital Domain executed approximately 340 shots. Among them were all the CG environments used in 100-plus shots, primarily in the diorama of Pompeii and Escher’s “Relativity” lithograph. According to Digital Domain’s Lou Pecora, visual effects supervisor, while building the environments and bringing the statues to life both posed their unique challenges, he believes the environment work was the most difficult in the film. “I say this because there was so much more in terms of scope and variety. Pompeii, for instance, was challenging because we faced the unique challenge of trying to make the environment look photoreal, but also like a hand-made miniature – just "fake" enough!” he says. “That was a first for me!”
Of course, the “Relativity” environment in the Escher sequence was a very interesting puzzle to solve, as well. “CG Supervisor TIm Nassauer devised an extremely clever interlocking shader setup to deal with the need to have the etching pattern scale up and down smoothly without ever giving up the lithographic look of the environment,’ explains Pecora. “This was no small feat, and one I was very impressed with when he showed me how it worked.”
The following is a summary of the work Digital Domain did on the movie, along with a Q&A with Pecora about the various work.
LITTLE TROUBLE IN POMPEII
This sequence required an all-CG build of a tabletop Pompeii diorama environment, including buildings with cobblestone streets and a very active Mt. Vesuvius. Digital Domain was tasked with re-creating the 79 AD eruption of the mountain that destroyed Pompeii. To this end, multiple complex simulations for the eruption itself had to be done, including smoke, fire, lava, and fireballs, along with the moving lava flow choreographed to match up to the live-action performers being composited into the scene. This sequence features crowd-favorite Dexter the Capuchin monkey, which saves the day in quite a memorable way.
Choreographing the lava flow was fairly time consuming, as simulations, by their nature, are difficult to art direct. Getting the right amount of lava at the right speed to put it in the right place on certain action beats required the team to work at the sequence level rather than at the individual shot level. Any changes in editorial – cut length changes or shot reordering – often caused the artists to go back and re-sim the surrounding shots as well as the one in question.
The diorama itself was a unique challenge in that the group was trying to achieve a photoreal look of a miniature, hand-made model. To some degree, it had to look “fake” in order to look right. The cobblestone streets, for instance, were modeled and lit with the idea that it was a vacuform solid-plastic base, like one that would come with a hobby model kit. The same concept applied to the walls and balconies of the set.
Another exciting part of this sequence was the brief but memorable appearance of Augustus’s animated head. The group used the studio’s cutting-edge Direct Drive facial animation system to take the performance of the actor on set and apply it directly to the model of Augustus’s bust.
For Pompeii, did you based your work on any historical data, such as the layout of buildings and streets, or was it purely artistic?
On an individual building, street or road level, we definitely took from historical reference on what the town would have looked like. But in terms of layout, shot composition and sequence clarity demanded that we adopt a more modern grid-like layout rather than the radial layout of ancient towns.
What software did you use to create buildings and streets?
The usual suspects: Maya for modeling, V-Ray for lighting, Mari and Photoshop for paint, and Nuke for composting.
I assume the most complex part was the volcano?
Yes, mostly because of the amount of elements involved, but any shot where lava was flowing was complex due to the need to precisely choreograph the sims. The cooling lava shots – especially the ones with the CG monkey pee – added even more complexity in terms of timing.
Please detail the eruption animation
.
The main eruption smoke plume, as well as the additional smoke elements, were all done in Pyro within Houdini by our FX Lead John Sparks (yes that's his real name!). I think the main challenge with this element was creating something that fit in our hybrid world of miniature set with real-world-sized physics. It needs to look big even though we're in the miniature world. The normal instincts of scale and speed/motion sort of had to be rediscovered.
The lava bursts are a combination of particles and FLIP simulations, which help drive the smoke simulations. The flowing lava rivers are a combination of FLIP sims and particle sims, with advected rest fields that were used to control the texture detail and the transitions from hot to cool lava. That was actually one of the trickier bits in this sequence: getting the flow to slow down as it cooled and making sure all of that was timed and choreographed to the rest of the action.
The fireballs with smoke trails were created using clustered Pyro sims. The challenge with these is keeping good detail across an element that travels a long distance. This is why the clustering was used, which is much more efficient than using a single container for the simulation.
Rendering was all done through Mantra in Houdini. We used PBR for the smoke to get internal light interaction of the lava and the smoke as much as possible.
Additional debris passes were created using instanced particles.
How many times did you have to re-sim due to editorial changes?
Hahaha, ummm... five to six?
Of course this had to be a trailer shot, so that was two sims right there just for the trailer. But for the feature, we needed a different timing on the big blast. So that was one more, even with no editorial changes. A few nips and tucks here and there, and some aesthetic calls, made up the rest of them.
How did you balance the realistic/diorama look?
Well, the way we wanted to approach it was to look at reference of miniature towns and what have you that hobbyists have been building. The Internet is loaded with images of model enthusiasts' work. For the most part, they are trying to imitate reality on a smaller scale. There are many limitations to this that they face, but it is surprising how good some models and miniatures actually look. Some of these hobbyists are total nuts and go all out to make you really think you are looking at something real. So we actually had to find some reference of maybe the second-tier miniatures. This way it would look just real enough, but not so much that it just looked confusing.
How difficult was that to do?
It was more difficult than I thought, actually. Our habits of spotting things that look a bit off, or out of scale – years of combined experience in striving for photoreal – kind of worked against us a bit as we constantly reminded ourselves and each other to deliberately make something look a bit fake. Not fake in a lighting or compositing type of way – more in terms of textures and shaders.
ESCHER TABLET PURSUIT
In this sequence, three characters enter the Escher lithograph, called “Relativity,” where a chase ensues to try to get to the magical tablet, observed by the artwork’s native inhabitants who have now come to life.
When the characters enter the artwork, they take on a variation of the distinctive etched look that is immediately recognizable as Escher’s style, and interact with the all-CG environment and its residents in complex and intricately choreographed ways.
This sequence provided one of the most fun and interesting challenges the artists have faced in recent years. Making the CG set look like lithography involved coming up with complex shaders that had to dynamically resize the pattern density based on factors like distance to camera and size in frame.
Studying the artwork itself taught the group that shadows aren’t just darker areas due to light being occluded, but rather a more dense arrangement of the lithographic pattern that is present in surrounding areas. This led them to render a full set of the more dense pattern to be revealed through shadow mattes that were either generated in the scene or rotoscoped or keyed from the plate photography.
VFX always involves a balance of technical acumen and artistry. This sequence definitely provided its share of technical challenges, but most of the journey in this sequence was spent making artistic decisions and creative calls. The etch treatment applied to the photography, for example, was very finicky and sensitive depending on how large or small something was in frame. There was no real formula for it, only artistic instincts on what looked right. Pattern angles, how many different sections the crew would need, how fine or coarse the pattern should be and how heavily the effect should be printed in would all have to be dialed in on a case-by-case basis – sometimes animating throughout a shot if it went from wide to close or vice versa.
Dealing with motion blur, depth of field, and so forth are usually very straightforward and more technical than artistic. In this case, however, the artists had to make a lot of artistic calls on how these aspects should be handled. Do they put the etch on before the defocus and let the lines get blurry? Do they let the etch treatment get motion blurred, or do we put the motion blur on before the treatment?
For the most part, there is some semblance of reality that one can use as a metric to see if an effect looks “right.” The more design-oriented nature of this work made that more difficult as they had to rely so much more on their far-less concrete artistic intuitions. But in the end, that was what made it so satisfying to see it all come together, the group says.
Explain the steps the team went through to create a lithograph look for the CG set.
Let's get the tech stuff out of the way first, because the fun parts in this sequence were definitely the artistic challenges. We used Maya, V-Ray and Nuke.
Ok, now on to the fun stuff!! CG Supervisor Tim Nassauer was charged with look development on this sequence. Now, I am going to say too much here.... and get personal!
Tim's background is in lighting. In fact, he is one of the most highly respected lighting minds we have ever had at DD. When one really stops and studies the ‘Relativity’ artwork – and Escher's work in general – is how unreal the lighting is. There is virtually no realistic occlusion, and the lighting in general seems to follow the rules of physics in much the same way that gravity does: Brightly lit walls sit directly on top of heavily shaded floors, shadows go in all different directions, and characters are sometimes lit completely differently than the very set they are standing on and touching! (Look in the upper left corner of the picture at the guy standing in the window – "the bartender," as we called him.) Like the rest of us, Tim has spent the better part of his career making artificial things look real. Here we had to come up with procedural solutions to make the set and characters therein look like something very specific – and something very specifically not real.
Using precise raytracing rendering software to render scenes and characters that not only ignored the rules of physics, but flagrantly broke every one of them, proved to be a daunting technical and artistic challenge.
Part of Tim's solution was to create a series of interlocking shaders that – when combined – made the lithographic etch treatment on the set look increasingly more dense. This way, opacity mattes could be used to combine these passes in such a way to more creatively control which areas got more dense and which less. Then we were free to match to the artwork – which was of the utmost importance to us and to overall show supervisor, Erik Nash.
Shadows also proved to be an interesting challenge. Traditionally, shadows are just areas where light is occluded; simple enough. But in Escher's world, shadows are denser patches of the patterns present in the rest of the piece. In the end, we used shadow passes as mattes to reveal denser litho patterns present in the multi-channel renders. All of this was just for the all-CG portions of the sequence. The photography required its own complex treatments.
Starting with stabilizing, retiming, and, in a lot of cases, re-projecting the photography onto geometry, we then had to apply the 'Escher look' to the plates. That involved a series of steps and treatments to gather the necessary parts to recombine in the composite.
Our tools for creating the etched look on photography worked best horizontally, so compositors found themselves rotating their plates in 30- and sometimes 10- or 15- degree increments, running the treatment, un-rotating, and then using roto to put the characters back together using techniques pioneered by Dr. Frankenstein almost two centuries ago.
Of course, everything was wedged to find the sweet spot: patterns, line thicknesses, color saturation (liberties were taken here to make our actors stand out from the set and 'natives'). I have to say, it was such a blast to go to dailies every day during the development phase and see what the comp team had whipped up. Comp Lead Brian Rust and Look Dev Artist Joe Spano really delivered the goods with this look.
What was the biggest challenge in that sequence?
In a small expansion of the previous section, I should mention that dealing with looks and settings that are usually very matter of fact and not open to much interpretation were very much open to interpretation for this sequence. For instance, how does one deal with depth of field? What about motion blur? Is the blur applied post treatment or pre? Or 50/50? In the end, there were too many variables from shot to shot to devise a formula for all of this, and we resolved to trust our artistic instincts on what 'looked right,' or what 'cut well.'
That is what made the whole thing so much more satisfying when it all came together in the end.
Was it more or less difficult in general obtaining a special look for the CG – CG that looked like a lithograph or a hand-made diorama?
I would have to say that in terms of achieving a desired look that Escher was more difficult than the diorama simply because of the more design-oriented nature of the Escher sequence. Sure, we had the artwork to match, but when dealing with the realities of various lenses, moving material, animated cameras, and the addition of certain amounts of color, the whole thing gets very clouded with variables and stylistic interpretation very quickly. At least with the diorama, there was a fairly clear target to shoot for and much more latitude in achieving that look.
GREEK AMPUTEE ENCOUNTER
Wall friezes and Greek statues with various missing limbs come to life as the artists enter the hallway and display room. To achieve this, the studio created a CG version of a practical frieze, along with six hero Greek statues and 12 partial statues on stands – all of which came to life when the tablet was around.
Can you give us a recap of the work you did for the Greek statues, creating them and then bringing them to life?
We created whole statues and partials – feet, hands, busts, etc. – to populate the Ancient Greek wing of the museum. With the exception of Augustus, this was fairly straightforward and very similar to how we always create animated CG characters. The big difference here were the stone shaders and the balance we had to find between making them look like solid rock and having them still be able to move around smoothly.
What tools/techniques were used?
We had scans of some of the statues and friezes to start from, and other times we just modeled them based on photographic reference. We did Look Dev in Maya/V-Ray, and set up textures, where applicable, in Mari.
What was the most difficult part of this sequence?
Achieving realistic lighting that matched the photography was probably the biggest challenge – at least that's where we spent the most time. Ben Stiller is walking around the set with a flashlight, kind of waving it around a bit, so getting that lighting to also affect the statues in a believable way was a challenge. All in all it wasn't one of the more difficult challenges we faced on this show, but we definitely wanted to get it right!
JED AND OCTAVIUS (VARIOUS SEQUENCES)
In order to maximize the illusion that Jed and Octavius are 1/23 scale, the group shot the background plates using a technique called “Focus Stacking.” This technique requires photography be shot at varying focus distances. These images can then be combined, or stacked, in such a way to allow for the focal plane to be adjusted as the shot is being composited together to more closely match Jed and Octavius’s photography and better integrate them into the scene.
LLP (LARRY, LAA PRISONERS, MORE)
Night guard Larry and Laa the Caveman – both played by the multi-talented Ben Stiller – perform in several long and intricate motion-control shots crossing back and forth in front, behind and interacting with each other furthering the illusion that these are two different characters.
Overall for the film, did you have to create any new technology or technique?
We used our new Direct Drive facial animation system for the first time on a real production on this show. Augustus's talking head proved the perfect asset to use this on, as it was self-contained and limited in scope. The system allows for very rapid and accurate application of a performance to a model of a face. More nuance is captured more quickly than with any other system I have seen in the past.
We also had to come up with tools and techniques for achieving the distinct Escher look. The amount of experimentation that went into the making of that sequence didn't allow for the most streamlined tool set, but since we knew it was unlikely that we would need much, if any, of those techniques outside of that one sequence, we opted for more flexibility and less structure.
Lastly, we hadn't used the focus stacking technology in production before. We had to rely on a mix of in-house tools and off-the-shelf software to get that working. That is something I expect we will use again and again on projects, so we made sure to streamline the process as much as possible to make it more easy for future productions to pick up and use.
Overall, how did this work compare with work you have done in the recent past?
Well, the most recent project I was on before this one was X-men: Days of Future Past. That consisted of a lot of wild destruction, feather-flipping Mystique action, and huge numbers of elements being choreographed to work together.
Xmen was a heck of a lot of fun, but it was nice to go from that level of complexity and activity to the slower pace and increased focus of a film like
Night at the Museum.
Having fewer elements interact with each other meant we could move more nimbly through changes in the cut or things like that. It also meant that we could stop and really focus in on certain assets that we knew would linger on screen a lot longer than the kinds of assets that come whizzing by in more action-oriented work. I like both kinds of work, so it was really nice to be able to get to do something as unique and different as
Night at the Museum is from the type of work I am usually involved in.
Did you borrow on any of the big developments from Digital Domain’s past feats?
We always try to use our past work as a starting point and then figure out what is still needed to get what we are trying to achieve. Smoke, fire, debris, etc. are things we have been working with for years, so using fluid-sim techniques to get flowing lava, for instance, was a great starting point. This way we could focus our efforts on the surface shaders and the need for mixed speeds of the fluids in the simulations as the lava cooled in places and still remained fluid in others.
Describe the pipeline you used for this film.
A simplified description would be Maya, V-Ray, Houdini, Nuke, with some Mari and Photoshop thrown in there at different points along the way, depending on the asset.
We use our in-house tracking software and many other proprietary tools to glue these packages together, but at the core, these are our bread and butter.
One set of tools that often gets overlooked are the database and dailies tools. We use a lot of Shotgun to keep track of assets, publishing, and shot scheduling, and our in-house dailies setup, along with RV, to keep the review cycles moving along at the pace needed to be efficient.
What work on this movie was the most fun, and why?
This may sound crazy, but I found fun in all of it. The nicest thing about the work on this film was the diversity. Sometimes we will have a large sequence with one type of work that is repeated over and over for hundreds of shots. For Night at the Museum, we faced many very different challenges from sequence to sequence, and I haven't worked on a show with this much diversity of work in a long time. That diversity kept the whole thing feeling very fresh and exciting the whole way through, so in a way, I guess the diversity of work itself was the most fun!
Anything else you want to add about the work?
I feel that the tone of a show is set by the leadership on the show. In our case, Client Supervisor Erik Nash, Director Shawn Levy, client production teams, and Fox Studio executives set a fantastic tone and mood in which we were able to complete our work. Keeping the mood light and hitting the notes hard was our mantra and our charge, and they supported us and we them every step of the way.
Couple that with the team of supervisors, production, and artists that I was fortunate enough to work with on our side, and we had a formula where everyone was able to contribute and feel appreciated, in turn getting the best out of everyone. I love when that happens!!