Emotional State

With so many emojis available (and used), it can be difficult to decipher what the writer is trying to convey – muddling meaning instead of clarifying communication. If this weren’t confusing enough, what would happen if an emoji could not retain its assigned expression? It could lead to many mishaps for the user, as it does in the animated feature The Emoji Movie from Sony Pictures Animation (SPA).

The film is about the “meh” emoji Gene, born without a filter and bursting with multiple expressions. Gene lives inside teen Alex’s phone, within the bustling community of Textopolis, as do the other emojis. “Textopolis serves only one purpose: to help Alex communicate,” says Tony Leondis, one of the writers and the director of The Emoji Movie. “They wake up in the morning, go to their jobs, and each emoji has a central and every important role to play.”

All emojis have to express the emotion they are assigned, no matter how they feel. And Gene’s unlimited expressions lead to problems, especially for Alex, who is trying to communicate with a girl he likes. Determined to be like other single-expression emojis, Gene sets off with his friends – the once-popular Hi-5 and the code-breaker Jailbreak – on an “app-venture” through apps on the phone to find Code that will help Gene.

Realizing the social relevance of the characters, the movie production kicked into high gear fast. Shot production started in September 2016 and finished in mid-June of this year for a summer movie release. All the pre-production and design work was done in-house at Sony Pictures Animation, and the animation was done at Sony Pictures Imageworks (SPI) by approximately 400 artists, including 125 animators, 100 lighters, and others across multiple departments. The animation was split between two SPI locations, Vancouver and Culver City, California. 

Carlos Zaragoza was production designer, and David Alexander Smith was visual effects supervisor.

Cast of Characters

The Emoji Movie features the biggest cast of characters to ever appear in an SPA feature. There are 322 characters between the humans and the emojis. Nearly 280 of those are emojis that live in Textopolis alone, where special lighting requirements taxed resources. And then there are the human characters from Alex’s world, which had their own stylized look for the film but posed no unusual challenges for the studio .

Conversely, CG Supervisor Michael Muir describes the emoji characters as “deceptively simplistic.” 

“The sheer number of them required the animation team to adopt a different approach. Also, they were hard to animate due to their shapes and simplicity,” says Muir. 

First off, emojis have a slightly different appearance on an Android phone versus an iPhone. Also, audiences are used to seeing the shapes in 2D. But in in the film, they appear as 3D characters, and even pop into their 2D representations in certain instances. The film crew had to consider all these states when designing the movie cast. 

Zaragoza says that while a movie about emojis would seem to be drawn from the current moment, the animators found inspiration for them in some of the oldest animation references. “Ultimately, we are giving life to objects, goods, musical notes – so for me, it was going back to the animated shorts of the 1930s, where everything was animated; objects had life,” he says. 

Most of the emojis, Muir points out, were simple shapes, many spherical, modeled in Autodesk’s Maya; textured and lit within Foundry’s Mari and Katana, respectively; and composited with Foundry’s Nuke. “Emojis are graphic designs, icon, pictograms,” says Zaragoza. “We use them to represent a concept, but they aren’t very complex. But for our story, we needed a complex character that could convey so many different emotions. It’s so important to show how a character feels. So, we had to keep the graphic look while making them very versatile.”

Smith boasts that the animation team can bring just about anything to life, and indeed they did for this film: toasters, fire hydrants, stop signs…and brought a unique characteristic to each. “But the hardest thing was that most of the lead characters are spheres. How do you animate spheres? It was quite a challenge,” he says.

Moreover, these shapes made it difficult for the riggers and animators to get a unique personality and performance out of each of them. In addition, the varied shapes called for a multitude of rigs.

The artists built three types of rigs: a simplistic rig for basic animation, a middle rig for background characters, and a full rig for the hero characters. “We also had to keep consistency across all of the rigs so the animation remained the same in all of them,” says Muir. 

To significantly cut the rigging process – from 15 to 30 days, to just one or two days – the group used a proprietary custom tool called AutoRig to automate some of the process. Then, they finessed each one by hand. In addition, a re-usable 2.5D facial decal system was also developed to reduce the total number of full facial rigs that were needed. This system was used for many secondary characters.

Of the many characters that appear in the film, Hi-5, Muir believes, is the most complicated: He is an open-hand emoji whose thumb and pinkie serve as “arms.” The other fingers also function as an arm or leg as needed as it performs a wide range of motions: running, jumping, dancing, acrobatics. And getting a range of emotions from the shape – a main character – added further complexity.

Exploring Environments

With so many different characters, it’s no surprise that the movie contains equally diverse environments. First, there is the human world and the emoji world – two contrasting settings, with the latter being more graphic in style. The concept design of the human world, on the other hand, is more reminiscent of other SPA features.

To build the expansive phone world of Textopolis – a surreal cityscape – the artists used instancing for variation while reducing the footprint and enabling them to retain the visual complexity without the excessive toll on rendering.

Also within the emoji world, there are many unique locales, at least 20, that are reflective of various apps, as Gene’s journey takes him through several of the most popular ones used by teens, including: Candy Crush Sage, Dropbox, Instagram, Just Dance, Spotify, Twitter, WeChat, and YouTube. Each app has its own distinct world, as the “three emojis” (Gene, Hi-5, and Jailbreak) make their way to the Cloud to find Code. Popular apps Crackle, Facebook, Shazam, Snapchat, and Twitch also appear in the film. 

Selecting the apps for the film was difficult, says Muir, and they were chosen based on their ability to challenge Gene and move him forward on his quest – as well as for their popularity and recognition factor. The designers also had fun playing with visuals in these locations, engulfing firewall in flames, for instance, and streaming music in Spotify. 

Simulations, such as those used for firewall and in Spotify, were accomplished with Side Effects’ Houdini. “The firewall was one of the most mesmerizing environments we did. It’s just a big wall of fire,” says Smith. “To make it related to the other worlds, we made the firewall out of voxels, three-dimensional pixels, running up and down and through it.”

The character rigging hurdles, no matter how difficult, were minor compared to the lighting challenges, which Muir contends were harder and more complex than anything else on the show. “The art department wanted everything in the emoji world – the characters, the environments, everything – to be translucent so a lot of the light would pass through, and that presented some challenges on the rendering side with all the extra subsurface scattering and translucency,” he notes. “Even concrete in the emoji world was translucent.

“Not only did our lighting artists have to pay attention to the lighting details on the surfaces, but also the transmitted light through the surfaces, which added whole new layer of complexity to the shots, especially on the main characters. With some, you can even see through their bodies,” Muir continues. 

With multiple layers to the characters, lighters had to pay particular attention to all the subtle details right under the surface of the light being transmitted to the various internal body layers. “As you watch the movie, at first glance you might not notice the little details in the bodies, but after awhile, you start to notice them,” Muir adds. 

For instance, Gene’s exterior is a translucent surface, very smooth with a microtexture. The interior has a pixelated surface made of voxels to give him a subtle digitized look. He was also given an extra layer of sparkles, or glints that look like glitter inside his body that could be turned on and off as needed to express his emotion in the story – more glowy when he’s happy and less visible when he’s sad.

According to Muir, the crew leveraged some of its newer shading models, such as a new brute-force subsurface scattering technique, which treats objects as a volume , as well as traces internal structures within that volume , as opposed to the traditional subsurface techniques that only diffuse the transmitted light at the object’s surface. All the rendering was done with SPI’s proprietary in-house version of Arnold, making the process more effective and efficient. More light passing through objects and characters meant more calculations for the computer. And more calculations meant more time that was needed to render those assets.

At the peak of production, the crew utilized 73,000 computer cores on any given day. Over the course of the film, Muir estimates 136 million clock hours were used. That translates into 50,000 years if a single computer were used to render the film. To put those numbers into perspective, that is almost double what the group used for 2015’s Hotel Transylvania 2 .

“All because of the addition light transmission and subsurface scattering in the lighting,” Muir says.

Adding Emphasis

We know emojis as simple icons. But in The Emoji Movie, they are anything but. 

“We spent many hours trying to get this film out. We put a lot of heart and soul into it. And, it is a fun, creative film,” says Muir.

In the US, the most popular emoji is the eggplant, followed by others like the poultry leg, birthday cake, and others. In Canada, it is the smiling poop. With such a large cast of characters in The Emoji Movie , which was Muir’s favorite? “Smiler. She is chipper and always has a huge grin but is always trying to hide her evil intentions,” he says.

Everyone has their favorite go-to emoji, and with so many characters in the movie, it’s likely you will find yours among the cast. Or you might find a new favorite.