Mermaids have sung a siren song to sailors and landlubbers since 1000 BC when, Wikipedia has it, the first mermaid stories appeared in Assyria. According to Greek legend, Alexander the Great’s sister became a mermaid. Sea-girls populated the tales in “One Thousand and One Nights,” and appeared in Chinese, Caribbean, European, Philippine, and Japanese folklore. A little mermaid even starred in a Disney feature animation,
The Little Mermaid, based on Hans Christian Andersen’s fairy tale. So, it’s no wonder that Captain Jack Sparrow and his pirate crew eventually would meet one or two, or maybe even 50. And that at least one pirate would fall under a mermaid’s spell.
The encounters take place in Disney’s fourth return to the big screen with a Pirates of the Caribbean feature film, this one titled On Stranger Tides. But, the mermaids in this Disney film are not fairy-tale princesses for little girls to dream on. They’re downright nasty, at least in their full-mermaid form.
Jerry Bruckheimer Films and Walt Disney Pictures produced the movie. At the helm was Rob Marshall, director and choreographer of Chicago and Nine, who took the wheel from Gore Verbinski. Johnny Depp returns as Captain Jack Sparrow, and Penelope Cruz debuts in a Pirates film as a sly woman from Jack’s past. Does she want him, or is she using Jack to help her find the Fountain of Youth? The Fountain of Youth is the reason Captain Jack’s men brave contact with the dangerous mermaids: They need a mermaid’s teardrop to activate it.
Industrial Light & Magic (ILM) is responsible for the fishy temptresses, the fourth time the studio has created sea-skewed versions of humans for the super-successful franchise. Charles Gibson led the overall visual effects effort, with Cinesite and The Moving Picture Company (MPC) helping sail the Pirates ship, as well. Also pitching in were Asylum FX, BlueBolt, Hirota Paint Industries, Hydraulx, and Rising Sun Pictures. Ben Snow piloted the visual effects team at ILM, with an assist from Scanline in Germany and Olin in Mexico.
The first problem the team had to solve was: What did their mermaids look like? Were they creatures or beautiful women with fishtails? “The director, Rob Marshall, favored a more human look even though he liked the creatures,” Snow says. “So we entered production with an approved design for mermaids with a human face, tendrils for hair, and a scaly body—on the creature side. The exception was the lead mermaid played by Astrid Berges-Frisbey, who was younger and would be a little more human.”
With that in mind, ILM created a big sequence in which we first see the mermaids. The pirates, intent on capturing that teardrop, baited the mermaids by sending a group of sailors adrift in a small boat. Seven mermaids surround the boat and, after wooing the smitten sailors, they attack. “We had aggressive creatures with sharp nails and fangs doing an extreme performance,” Snow says. “They were striking-looking. Instead of hair, they had jellyfish and seaweed tendrils. But, Rob [Marshall] decided he wanted the women to look more beautiful above water. So, we needed to change them into something more like digital doubles, still somewhat creature-y, but humanoid-looking. We went back and shot live-action actors above water.”
The conceit they settled on was that the water would be the dividing line. Below water, the mermaids would be CG creatures. If they were halfway out of the water, they would have humanoid upper bodies and hair with hints of “mermaid-ness” above the water, and fishtails below the water. If a mermaid found herself completely out of water, her tail would evolve into legs.
“The creature mermaids have scales and fangs, and their eyes glow,” Snow says. “Out of the water, if the mermaids are wet, they have a mermaid look, but they can use a human camouflage to capture pirates. They can make their scales fall away.”
The scales on the mermaid’s skin hint at the full-bodied fishtail beneath the surface, all created by Industrial Light & Magic.
Because the mermaids have a more human look, even underwater, than originally designed, the crew replaced the tendrils from the creature design with a hair system, but sometimes still layered in jellyfish-like tendrils. “In some shots, we needed hero tendrils doing something specific, so we added them where we wanted and simulated them the way we wanted,” says Philippe Rebours, digital production supervisor. “We used a cloth mesh to mix hair with tendrils. For close-ups, we added hair curves that we simulated as hair. We’d render the hair but not the tendrils beneath driving them.”
When the mermaids dive underwater, the artists took advantage of the caustics to reveal the transformation from human skin to scales. “We have multiple effects to bring in the scales,” Rebours says. “We had a transparent membrane that hid the look of the scales, so we could reveal the scales little by little with caustics and specular to illuminate the surface and by using maps to define the speed.”
The scales on the CG creature are texture-mapped 3D assets. “When the mermaid Tamara [Gemma Ward] transforms, we had geometry matching the actress perfectly,” Rebours says. “So, we did a lighting pass to have a creature that looked like Gemma, with light corresponding to the light in the plate, and then in the comp, we revealed scales in some areas but kept the skin in other areas.”
Out of the Box
One of the most difficult mermaid sequences involved Syrena [Berges-Frisbey], a mermaid with a special attraction to one of the pirates. In the sequence, the pirates have captured her and encased her in a coffin-like glass box filled with water. As they carry her across land, they drop the box. She rolls out, the water flows away, and she becomes human—for as long as she is out of the water, that is.
The technical and artistic difficulties with this sequence were twofold: When Syrena was a mermaid, she needed to look slightly different from the actress, but still recognizable as Berges-Frisbey—and still beautiful. “Even when she’s a mermaid underwater, she still has to be attractive,” Rebours says. “You want the audience to believe that one of the characters connects with her. That would be hard if he had to connect to a fish.” Second, the audience needed to see her transformation into a human.
The scales on the mermaid’s skin hint at the full-bodied fishtail beneath the surface, all created by Industrial Light & Magic.
For Syrena’s facial performance, the studio captured a FACS-based shape library from Berges-Frisbey with MOVA Contour, and applied micro-expressions derived from the capture to a digital model of the actress. The studio’s proprietary Fez system drove the library of shapes. “The shapes exactly represented the expressions and geometry from the actress,” Rebours says. “Tim [Harrington, animation supervisor] really enjoyed that he could move a slider and the character stayed true to the actress. We used to have to do corrective shapes. With this system, the model didn’t get out of character.”
The shape library also helped the animators create a mermaid who people recognized as the actress, even though her face was slightly different. “Her eyes were bigger and her nose was thinner,” Rebours says. “But, we could put the information we captured from her poses onto the geometry representing her as a mermaid.”
Similarly, they used the shape library captured and created from Berges-Frisbey as a starting point for animating facial expressions on the other mermaids, as well. For those mermaids, the crew first created digital doubles using photo-modeling. Then, they applied the shape library to the faces. “We would match-animate the actresses’ performances, and then the animators would tweak the expressions,” Rebours says. “It was a good way to do the transformation from a real actress to a CG mermaid.”
Syrena’s transformation, however, is a hero performance, and one that takes place over several sequences, which presented opportunities for technical and creative problem-solving. “One difficulty was the scales on her face. Too many and she’d look like she had a sickness,” Rebours says. “It was a moving target, and every time we changed her, we had to make those changes on the seven other mermaids.”
When she falls out of the glass coffin and transforms into a human, her scales disappear, and then the primary problem shifted to her tail. “For her upper body, we had a transparent membrane over the scales that we turned opaque to hide the scales when she becomes human,” Snow explains. “But, we needed to transform that elegant 14-foot tail into human legs and feet.”
On set, actors carried an empty glass coffin with bluescreen on the back; the coffin was too heavy to fill with water. Separately, stunt performers dropped a practical coffin filled with water. In postproduction, ILM inserted its CG mermaid inside the glass coffin and added CG water to the practical elements shot on set using a combination of proprietary water tools and Exotic Matter’s Naiad fluid-simulation software.
The pirates drift through smoke—added to the live-action footage by compositing artists at ILM—toward a digimatte set extension created by ILM painters.
Then, to create the transformation illusion, ILM tracked Berges-Frisbey’s performance—writhing on the ground as if she had a 14-foot tail—using its iMocap system. Modelers had created two digital doubles: one, an “Astrid” model that matched Berges-Frisbey, the other, a “Syrena” model which rode in the coffin that was Berges-Frisbey’s double but with a tail rather than legs. On set, the actress wore bicycle shorts with a tracking pattern printed on them, and the studio painted henna tattoos on her arms. In postproduction, the ILM artists matched the Astrid model to the actress’s performance in world space, and then mesh-wrapped Syrena to Astrid so that Syrena could replace the actress writhing on the ground.
“The tracking patterns helped automate the animation process, but when it came down to it, it was a lot of hand work,” Snow says.
Making Tracks
Stereo 3D made it necessary for the tracking and match-animation to be particularly precise. “Everything had to be real in the space of the scene,” Rebours says. “We had to match-animate the actress lying down on the ground perfectly, every single portion of her body, even the shapes of her face, in every frame.”
To do that, ILM developed new matchmoving tools that placed the 2D image in 3D space as the camera moved through the sequence, so the artists could fit 3D elements onto the 2D image and replace the 2D image as needed. “When you have a rigid body, it’s easy to match it during the entire time of a shot,” Rebours says. “But in this shot, everything was moving. Even if she’s smiling just a little bit, it changes the entire geometry. And, we had to match the depth of every single pixel perfectly. We already had good solvers, but Ronald Mallet also worked on a new solver that was nice. For example, we could conform the geometry to the background plate in one frame of the plate to match her smile, and the new solver would transform the geometry while matching it through the shot. We had a couple shots where that really helped. Otherwise, our layout people matchimated frame by frame, by hand.”
Although the above images look like live-action plates, they are actually digimatte paintings created by artists at Industrial Light & Magic.
The animators then manipulated the “Syrena” model to create a performance to help with Syrena’s transition from mermaid to human. “Jung Seung Hong did the shape animation as she scrunched up into a fetal position,” Snow says.
To remove the tail and reveal the human legs inside, the artists used three layers. The legs on an inner layer, placed in a tail-like position to start, parted and lifted as she became more human. Covering the legs was a “sock” with scales. And covering that was a transparent outer membrane that could hide the scales. As she transformed, the outer membranes became more transparent and peeled off. “People were concerned that it would be icky, so we tried to do it in a tasteful manner,” Snow says.
As the geometric model in the inner portion transformed from tail to legs, a cloth simulation slid the outer membranes as if they were a sheet draped over her legs. A fluid simulation layered into the shot slipped the membranes into water, and her skin lost its scales.
“We had to carefully pose her,” Snow says. “We have her hair draped appropriately.”
During one point in the film, a herd of mermaids attack. For that, the animators applied cycles, created for the seven main mermaids, to the crowds. “Imagine you have a library of 3D elements, the mermaids fully simulated but not rendered,” Rebours says. “In this case, the rig was light. The animator could go into [Autodesk] Maya, look at a library of animation, choose one, position the mermaid, play the cycle, look at it, and maybe do an offset. It was like you would do with 2D elements, but you’re placing 3D assets in space. And then, there you are. You have an animation with 50 mermaids that are sim’d automatically.”
These images are live-action shots except for the ship in the bottle, which is a CG ship that ILM added in postproduction.
A library of splashes inserted using The Foundry’s Nuke at the point where a mermaid rises from the water completed the illusion. “We used 2D elements in the far background, and we also shot 3D water elements for when they’re close,” Rebours says. “Stereo made compositing with water challenging. You can’t easily blend multiple splashes if they are interpenetrating each other.”
Fickle Fountain
The goal for the characters in the film was to get to the Fountain of Youth, and they did, but once there, they discovered there are two sides to this fountain. “We did the shots leading up to the climax with Scanline and Olin,” Snow says. “Scanline did a little sequence where a magical droplet of water jumps around Jack Sparrow’s finger. We designed and directed the work; they executed it. Also, they did a second sequence with rivulets of water on the walls and ceiling of a cave that’s a magical portal to the fountain.”
The actors worked on a set built on the 007 soundstage in London; ILM’s digital matte-painting department, led by Brett Northcutt, created the surrounding environment. During a confrontation that develops after various pirates arrive, The Moving Picture Company destroys a digital version of the rocky fountainhead and smashes the surrounding temple.
“There’s still enough water left to activate the magic, though, and Syrena does this to save one of the characters,” Snow says. “But, the conceit of the fountain is that if someone lives, someone dies.” That someone is Blackbeard (Ian McShane). A tornado of water pulls him inside and liquefies him.
“Scanline created the initial effect of the water coming out of the ground,” Snow says. “As it develops around him and he liquefies, we used our water simulation and Plume particle-simulation tools.”
In the shot, you see McShane performing in the water with his hair and beard moving with the flow, and then, bit by bit, his costume and his flesh rip off with the force of the water tornado. To create the shot, the crew first filmed McShane on a bluescreen set, and then filmed the set without McShane. Layout artists tracked the actor’s performance and animated a digital double to match his moves.
“Our digital double matched the actor in costume, but we didn’t worry about his face too much because we had the images of the real person,” Snow says. “And, we deviated from his performance a bit to make the decay slower.”
Some parts of the water tornado were mist and spray, driven by Plume; other parts were more solid, driven by a fluid simulation. “We used Plume to give us the general movement of the water spray,” Rebours says. “And then we ran another simulation, which added more detail. The main difficulty with the water was the rendering. This thing had so much force and rotated so quickly, the motion blur became a problem. But Pat Myers [CG supervisor] did a great job solving that problem.”
To shred Blackbeard’s costume, the crew used a cloth simulation. “That fed a particle simulation,” Snow says. “His flesh emitted particles and the cloth also became particles, so as the flesh and clothes tear off his skin, they get suspended in the water and mixed up in the vortex. “We reveal him in different stages. We see the skeleton with flesh and hair flapping around. We cut away. And then we see the skeleton with only a bit of flesh hanging together.”
The artists rendered each element separately for compositing in Nuke. At the end, the bloody arm of his skeleton reaches out from the vortex toward his daughter.
“It’s fun to do one of these big ‘death of a major character’ things,” Snow says.
With Rango and other films in the works at the same time, ILM in San Francisco and in Singapore created only about 320 shots for this Pirates film, far fewer than for the other three, and Scanline did 80 of those.
“There were so many mermaid shots and Blackbeard’s death shots, and not enough people here to do more,” Snow says. “Also, the show had a short production schedule, and they shot it in native stereo, so there was a whole raft of technical problems. It made sense to split the show among ILM, MPC, and Cinesite.”
But, even though ILM had fewer shots than before, the shots the studio created were among the most spectacular and demanding. And, if you listen to Snow, perhaps the most fun.
Barbara Robertson is an award-winning writer and a contributing editor for Computer Graphics World. She can be reached at BarbaraRR@comcast.net.