Dual Approach

Perhaps it is ironic that complex 3D technology was used to create The Mitchells vs. The Machines, an animated feature about a Luddite father who utilizes his low-tech natural skills to save his family and the world from a robot apocalypse. After all, it took a great deal of R&D to achieve the film's "imperfect" 2D hand-drawn aesthetic using a state-of-the-art CGI pipeline.

The Mitchells, developed by Sony Pictures Animation (SPA), is directed and written by Mike Rianda and Jeff Rowe, two friends from their days at CalArts. The action-comedy is streaming on Netflix.

The adventure begins when quirky daughter Katie is accepted into film school and is eager to follow her dream and bond with others who "get her," unlike her family, particularly her nature-loving dad. On the day she is about to head to the airport, her dad surprises her with the news that instead, the entire family is driving her to college across the country as a bonding experience. She is not happy about this at all. Ugh, what could be worse? Finding themselves in the middle of a robot uprising, that's what. And it's up to them to save humanity.

"Our main focus was trying to create great memorable characters that the audience really loved and cared about - everything we did was in service of that," says Rianda. "I think that's why the elemental themes resonate: kids wanting independence, parents wanting to keep their family together, and then appreciating how much your parents have done for you, and how, as a parent, you have to let your kids be their own person."

While there is commonality in the theme, the film's aesthetic is quite unique. Those behind the animation were responsible for the atypical look of Spider-Man: Into the Spider-Verse, which combined CGI with traditional hand-drawn comic-book techniques. In fact, the group at Sony Pictures Imageworks took many of the tools they had used for Spider-Verse and bent them to deliver a look that was unique and special.

Oscar-winning producers Phil Lord and Chris Miller, also from SPA's Spider-Verse (as well as The Lego Movie, Cloudy with a Chance of Meatballs, and others), signed on for The Mitchells.

The Mitchells vs. The Machines is full of humorous moments.

"Their success with Spider-Verse, which was such an experimental movie, gave us a lot of cover to make our movie more wild and idiosyncratic than I think it could have been if they weren't around," says Rianda. "Generally, there's an impulse to shave off all the weird edges of any story, particularly in a big studio. But when they came on board, they encouraged us to chase our wildest impulses and iterate every scene until we got it just right. They helped us sharpen the comedy, strengthen the relationships, and let the emotional moments breathe."

And that peculiarity not only encompasses the story, but the animation as well. The film utilizes two different animation styles: an illustrative, hand-painted approach used for the human world, and a slick, realistic approach typical of CGI for the robotic world. Plus, there's a cartoony approach for elements crafted by Katie, who has a penchant for expressing her thoughts and emotions onto the screen during freeze-frame moments.

Says Miller, "One of the amazing things about animation is that its only limit is the scope of the filmmaker's imagination. It feels like only now do we have the technological tools to make it so that truly any visual style or language conceivable can be represented on screen, and we're just at the beginning of a golden age of creativity."

Mike Lasker, visual effects supervisor on Spider-Verse, had just finished work on that film, which he calls "unique and different from anything else. It was so bold and graphic, and we had to reinvent how we did everything, which was challenging but so much fun." And immediately afterward, he was presented with a new challenge, this time as VFX supe on The Mitchells. He accepted without hesitation.

Daughter Katie is an artist and often draws on the film screen.

"Once you get used to reinventing things, you almost become addicted to it as an artist. You don't want to do anything the same way. You want to keep coming up with new looks and new ways of doing it," says Lasker. "Creatively, it's been great to push things in a completely new kind of artistic direction."

Crafting a look for The Mitchells was creatively challenging, requiring the Imageworks artists to collaborate with the visual development team of directors. "When we first started, they had a lot of beautiful drawings and artwork, but hadn't landed on the final look. Once they gave us a painting that illustrated exactly what they were looking for, we broke it down into components - outlines, brushstrokes, depth of field - and had to figure out how to achieve those things," says Lasker.

Once the team finalized the unique look along with a plan to achieve it, the entire production had to be instructed on how to craft the aesthetic. "We were going to have hundreds and hundreds of artists who were going to produce 3,000 shots with that look," says Lasker.

The Human World

From the start, Rianda wanted the human world to look very different from the robot world. He wanted to portray the Mitchells' suburban Michigan home as honestly and realistically as possible - a bit messy, with dog toys on the chairs, unwashed dishes in the sink.... It was to look handmade (where you could feel the hand of the artist) and reflect the humans and their flaws.

To this end, Lindsey Olivares, production/character designer, integrated a lot of herself and people who she knew into the designs - just as Rianda did in the story - to resonate with the audience.

"Everyone really fell in love with her illustrations," says SPA Production Executive Kurt Albrecht. "Her work has this wonderful, handcrafted, illustrative quality, and we wanted to make sure we preserved that as we translated them to CG animation." Thus, the artists relied on line textures wherever they could add them, such as the squiggly lines on dad Rick's jacket or his hair.

Artists used an illustrative, hand-painted approach for the human world.

According to Lasker, the team developed three major tools for this cozy, familiar, tactile world: an outliner, a brushstroke tool, and a patch projector for simplifying the various forms of vegetation. "Everything we did was pretty much a new workflow - from the ink outlines, to the brushstrokes, to the patcher simplification, to how we treated depth of field," he adds.

The group used an illustrative outline style, one that highlighted a character's form in an artistic way and mimicked the texture and variation of watercolor. These lines also respected light and shadow, inheriting a character's coloring and lighting in order to feel more cohesive and connected.

"The outlines had to inherit color from the object they lived on, but they also had to be lit. That was a very new thing for us to develop that we hadn't done before," Lasker points out. The TDs additionally had to find a way to layer the outlines, which were around the characters and environments.

Another challenge was lighting the characters and determining how the brushstrokes would integrate with that lighting. "Similar to what we did on Spider-Verse, we broke the lighting. We quantized the normals, removed all the soft shadings, and broke the shading down into bands of light. Then we broke that up into brushstrokes," explains Lasker. "Whereas on Spider-Verse, we broke it up with Ben-Day dots and hatch marks."

A painterly process was developed for the vegetation, which was then simplified.

Furthermore, the artists did not want it to look like the lighting was done by computers. "It looks like someone painted light around the body. We are imitating 2D cel animation," says Art Director Toby Wilson. "It's either 100 percent lit or 25 percent lit; there's not a lot of in-between in order to make it look and feel like classic animation."

The team additionally developed a painterly process for the trees, grass, and vegetation, since it takes up so much of the frame. Once the vegetation was generated, the group developed a tool it had started using on Spider-Verse for Gwen Stacy's universe, to simplify geometric detail into swaths of color. "We could take a field of grass, and instead of it looking like every blade of grass was there, we would change it into patches of color," Lasker says.

Meanwhile, the artists used Autodesk's Maya to build models in the human world with irregularities - they are lumpy, squishy, and wobbly - then developed a painterly depth-of-field style and inserted the brushstrokes into that. When the outlines were added on top, it accentuated that look even further. Also, the textures had a painterly, irregular style, which was married with the brushstroke detail. "It was tough. We had to figure out a formula for it," Lasker adds.

For effects, the group used SideFX's Houdini. Lighting was done in Foundry's Katana (which originated at Imageworks), compositing was done with Foundry's Nuke, and rendering in Autodesk's Arnold.

The Robot Realm

The robot realm is introduced starting in the second act of the film. Whereas the Mitchells' world has a very specific visual feel and shape vocabulary, in contrast, the austere tech universe is full of sharp angles. It is more perfect and sterile, with clean lines and sleek, reflective surfaces. It is the exact opposite of the imperfect, handcrafted aesthetic that mimics 2D, since it is a world designed by AI. "Our look had to transition into [the robot world] over the course of the movie. What made it especially difficult was that our characters had to exist in both worlds," says Lasker.

For instance, the outline tools that produced irregular and messy results at the beginning of the film would have clean lines by the end. "We had this evolution happening slowly so the audience would just sense that it's happening but it wouldn't be that obvious," he continues.

The Stealthbots break apart and reassemble into geometric shapes.

Generally, the robot world had a more typical CGI aesthetic but also required new shaders, tools, and workflows for the glitzy, flowy, reflective, ray traced look. Whereas the human world is smaller and filled with objects, the robot world is stark, open, and minimal. And here, everything is in its proper place.

The backgrounds, meanwhile, project a feeling of isolation. All of this helps drive the performances.

The main characters here are the unique Stealthbots, which had to move in a very clean way, despite how they break apart and reassemble into various geometric shapes.

According to Lasker, when they first saw the designs of the Stealthbots, they knew that the bots needed to move in a unique way. Since they are made by an AI, which was made by humans, it would make sense there was some holdout of bipedalism, but that it was starting to evolve into something different. So,, the group performed many tests, experimenting with different kinds of morphing, breaking apart, warping, and so forth, and ultimately landed on what is on the screen.

To achieve this look, they developed two new tools. One is a tool that slices the robot like a knife, and one that uses Booleans to create positive and negative shapes inside the robot. The slice tool performs a macro of actions after a simple click-and-drag of the mouse across the geometry and is relatively simple. The Boolean tool is much more complicated based on the fact that Boolean difference operations make geometry extremely unstable and difficult to pass down the pipeline.

For the film, Imageworks developed an outliner and brushstroke tool.

Moreover, there is no rig for these characters, says Lasker, as every animator used the two tools differently for each scene and, as a result, no two transformations are the same.

As they did for the human world, the artists developed a number of other tools to use for this robot realm, some of which they had been evolving over the past couple of shows. One such tool enabled them to paint a very graphic, sharp reflection and give it virtual depth.

Another tool, called the Magic Cube shader, was used mostly for the showdown in the mall, to build the multiple store interiors and give them virtual depth. Used for the building interiors in Spider-Verse, the tool evolved for use in The Mitchells. "The visual development team could design the interiors of all the mall stores using this very illustrative look, and they would give us Photoshop files with the layers for the walls, ceiling, and items in the stores. So, if we had a flat plane, we would map this on and create a virtual fake interior," explains Lasker. "You get the walls, the floors… everything in the store, and it looks really painterly, but it's all mapped onto a flat plane. It really sold the mall. Between the graphic reflections and the virtual interiors, we got a more illustrative yet sleek look in the robot world."

Katie Vision

Because Katie Mitchell is herself a filmmaker, the team felt it was a great idea to have her creativity and vision spill onto the screen - a visual touch they called "Katie Vision." "The idea was, if Katie is a filmmaker who makes all these goofy movies, what if it felt like she was the one editing our movie? So, she's the one writing on the frame and drawing hearts around her mom and devil horns on her dad," explains Rianda. "It seemed like a really fun way to show her character more within the filmmaking of the movie itself. We wanted to make her this relentlessly creative person who's drawing on her shoes, her pants, the wall, or wherever. So, it would make sense that she was drawing on the frame."

All the artists loved playing with the idea, too, as it gave them an opportunity to mix CG animation, live-action material, and 2D drawings. The imagery had to stand out in frame, but not too much. Olivares adds: "We took our beautiful 3D renders and drew on them by hand as though Katie herself had drawn over them. There's a young energy and freedom that comes from taking a very high quality final frame and drawing directly over it like a teenager."

Sometimes Katie Vision was presented as little expressive marks and small pictures here and there on the frame; other times it would take over the entire frame. As a result, Olivares and others had to create shot elements for animation, involving themselves in SPI's shot production pipeline, and then it would be integrated in comp.

"Animation wouldn't pay attention to it. They would just ignore that there was any Katie Vision going to happen, because Katie Vision was supposed to be reactive to what Animation was doing," explains Lasker. "It was very artistic and gave the film a home-movie feel, and it helped connect the audience to Katie as an artist. It was very cool."

The artists used Maya for modeling and Houdini for effects.

A New Dawn for CGI

Spider-Man: Into the Spider-Verse was very well received by audiences, taking home a number of awards for Best Animated Film, including the 2019 Oscar. Meanwhile, The Mitchells has been especially popular with Netflix viewers. Perhaps it's because of the story, or perhaps it's due to its unique animation aesthetic - or both.

Initially, the new look for Spider-Verse had been somewhat of a jolt for SPI - "It made you tear out every principle that you had been doing for years and years in CG and reinvent it," says Lasker.

Indeed, at the beginning it was a huge challenge, but one that grew on them. "It kind of becomes all you want to do. You just want to make new looks," he adds. "I felt like a different artist. Once you do something that's so new and so creatively challenging, that becomes all you want to do creatively as an artist. It really had a huge impact on the studio and on so many people who worked on Spider-Verse."

Then, they had the chance to travel down yet another unique, new path for this film.

So, does Lasker see CGI moving in different directions? "I definitely do," he says, pointing to some recent examples, including I Lost My Body, Wolfwalkers, Soul, Love, Death + Robots, and Pixar's latest film, Luca. "I think audiences are looking for more creative looks, there's an appetite for it. There are some amazing movies that have cool, amazing styles, and people are seeing animation as a lush medium. There are so many things you can do with it, and I think you're going to see that happening more and more."