Navigating rough waters to create the effects for Soul Surfer
While a live-action film, the heart and soul of Soul Surfer relies on photorealistic effects that are invisible.
Soul Surfer is the inspiring true story of teen surfer Bethany Hamilton, who lost her arm in a shark attack and courageously overcame all odds to become a champion. The film, directed by Sean McNamara, features an all-star cast including AnnaSophia Robb, Helen Hunt, Carrie Underwood, and Dennis Quaid. Amazing surfing, moving performances, and photoreal effects all contribute to this inspirational film experience. Bethany Hamilton’s transformation from shark attack victim to top surfing pro makes her nothing short of a real-life superhero. But for the story to translate to a film’s retelling, audiences have to accept her physical loss as real. To accomplish this challenge, McNamara turned to longtime collaborator Dan Schmit (VFX supervisor) and his company Engine Room to lead what would ultimately become 750 crucial effects shots.
Here, Schmit talks about that work with CGW chief editor Karen Moltenbrey.
Wow, 750 VFX shots sounds like a lot of work, especially for a non-VFX film.
Yes, but as it turns out with 750 VFX shots, Soul Surfer actually is a VFX film! Although, it took a bit of convincing for the director and producers to admit it.
Were all those shots planned from the start, or did the number increase as filming and post progressed?
Around 290 shots were originally discussed, although we had a suspicion that the number would be increasing. At the start, there were plans for mostly lock-off cameras and lots of tight close-ups to simplify the post work. But once film was rolling, it was clear that what's best for the VFX guys could never be a priority over the movie as a whole. It ended up being an exciting challenge and made for a more compelling experience overall.
What did those effects entail?
More than 500 shots involved creating the illusion that Annasophia Robb was a real amputee. When we first began, it was assumed by the filmmakers that all these shots would be 100 percent CG, but once into it, we realized that a good prosthetic stump could do a lot of heavy lifting. As it turned out, we were able to accomplish 90 percent of the arm shots in 2D by using a prosthetic in combination with clever compositing techniques, although we leaned on 3D when it was needed.
In addition to the ‘green arms,’ we had another 250 shots that needed work. Many of these were in support of the surfing, which was extremely difficult to capture in camera. With six young surfers in the water plus actor support, cameras, ADs, and so forth, all in heavy surf, it was crazy. Trying to inject instructions on where the surfing actress or stunt double should hold their green arm was impossible. As a result, a lot of digital work was necessary to create the three competition sequences, which were all rewritten in the editing room to work with whatever shots we could marry. We found ourselves rebuilding many surfing shots completely: changing the number of surfers, which directions they were going, the designs and colors of their surfboards, rash guards, and, of course, their faces. It was all needed to tell the story and maintain visual continuity.
What size was your crew?
For the amount of work we produce, Engine Room has a surprisingly tight in-house crew. We changed our business model well before the start of Soul Surfer away from the traditional facility structure. We now only have six full-time, in-house artists (who are all rock stars) and another five who are producers, editors, and support staff.
Most of us have been together close to ten years. To say that we are a well-oiled machine is an understatement, as we were able to do half the VFX shots in Soul Surfer with just our in-house crew. For the rest of the work, we collaborated with our extensive off-site artist team. There were 18 off-site artists working on the project last fall who we really consider part of the studio. Our whole pipeline is about this type of workflow, and we have gone to great lengths to find and vet the right people for any specific project and task.
How long did you work on the film?
We started doing early tests for the film over two years ago, but December of 2009 is when we did our first real R&D. This involved a design effort to discover how the prosthetic would work, shoot some real camera tests, and take some shots all the way through post completion. Principal shooting was in Hawaii during February and March of last year. Post ran all the way up until Christmas, with lots of preview screenings along the way.
We found that VFX deliveries for the preview, while a massive undertaking, actually really helped us. The producers were willing to put up with an 80 to 90 percent completed composite, just as long as there were no green arms showing. This translated into Engine Room delivering 500 arm shots as temps in a rush, which, as painful as that sounds, made us discover what was really needed in order to finish shots, as well as what was not [needed]. It’s easy to overthink this type of work going in, but in reality, we will not survive in today's climate without finding every corner possible to cut.
When the picture finally locked, we were well on the way on the vast majority of the work. As the cut evolved, we were able to keep up with the changing shot list with a lot of immediacy. It helped that the filmmakers brought on a wonderful VFX editor, Joey Fedele, to work in-house with their editors. Without his constant diligence and skill, it would have been a challenge for us with all the changes to the cut going on through the fall.
Did you contract any of the work out?
The process of creating this particular illusion for the film required us to step out of the usual production chain of command and find an effective way to make these hundreds [of shots] smoothly. We realized early on that what we needed was one make-up effects genius to support our efforts, and that's how we found Mark Garbarino. Mark immediately threw himself into the project, and we were able to design and produce a first-model working prosthetic fairly quickly.
With that element in place, some of the arm removal shots (depending on camera angle) would only require removing the green arm without any stump patching or enhancement. After we proved how it could work, the producers hired Mark to be on set every day to apply the prosthetic before shooting. Below the stump, the actress’s arm was either in a green sock or it was painted with waterproof paint. Either myself or EP Michael Caplan was there on set for all of the shooting, and together with Mark, we acted as our own mini-VFX unit.
In post, we did share several dozen shots with two other companies, including Base FX, which helped with some of the trickier 3D, and Backlot imaging. Both did fantastic work for us. Base helped us with the shark itself, which in the original planning was going to be extensive; but for the sake of editing and the PG rating, it was reduced to only one shot. Our good friend Rusty Ippolito and Valerie Delahave from Make FX also stepped in at the 11th hour to work their expert brand of magic on some last-minute face replacement requests from editorial.
What was the most challenging work?
One of the biggest challenges came from the sheer volume of shots and the fact that almost none of them were locked off. With the camera always moving, it was often impossible to acquire the clean plates needed to paint in the background and replace the actress’s missing torso areas—under every circumstance imaginable. We have discovered the hard way that the ‘small in-house artist team with large off-site backup’ is the only business model that now works for boutiques like Engine Room. It gives us something that the big guys don't have, which is near instant expandability without explosive overhead costs. But it also helped that Soul Surfer was such an inspiring project. Everyone working on it was driven by the story and gave it their best.
Other unusual challenges came in the surfing sequences, specifically the face replacements. Although we did create a 3D model of Annasophia’s face, we ended up relying mostly on photographed elements that were captured during a green shoot we produced last summer. Many of these involved putting Annasophia’s face on the real Bethany Hamilton’s body, which was a unique endeavor, as Annasophia is a 5-foot, 5-inch-tall 15-year-old actor from Denver, and Bethany is now 22 years old and a 6-foot Olympian. We did some interesting things with proportional scaling in order to make those shots play.
What made this work either unique or different in some way, or was it typical VFX?
I don’t think that there have been many films that have so completely relied on the audience accepting a heavily post-VFX-altered actor as being absolutely normal. With an actress in a bathing suit throughout most of the film, there was literally no place to hide. If the effects were not seamless, the entire movie would fall flat. And with the shark attack happening so early in the film, there was a huge variety of conditions to conquer, including surfing scenes with wild body movements shot from floating and bobbing cameras. The fact that it all worked thrills me and makes this project one of the most unique in my 20 years in the business.
One of the interesting things about our post workflow was that few shots were touched by more than one artist. As a result, in most cases, one person would do anywhere from 5 to 50 shots in their entirety. This leant itself perfectly to our model of mobilizing an army of off-site artists, as the work was so easily divisible. This can be one of the biggest challenges when working with off-site teams: finding efficient ways to break up the work.
What tools did you use?
All of the Engine Room on-site compositing was done in Adobe’s CS4. Any 3D was done in Autodesk’s Maya. Our off-site artists team used every compositing software you can think of: CS4, CS5, The Foundry’s Nuke, Apple’s Shake, and Autodesk’s Combustion.
Did you have to devise any new techniques or technologies?
A funny new VFX term was added to industry lexicon as a result of Soul Surfer with the invention of the ‘Pit Plate,’ short for ‘armpit plate.’ Obviously, we shot clean background plates after every setup for the purpose of background reconstruction behind the green arm, but more often than not, the other areas that needed to be replaced were her torso areas beneath her arm. The solution was to have Annasophia raise her arm up with the camera rolling at the end of every set up, so that we had a few frames of what her torso and underarm looked like without her arm blocking the view. Pit plates became an invaluable asset to our process and greatly simplified the replacement of the torso portion of her body as frames of these elements could be tracked, time-mapped, and patched in to fill the missing areas.
With so many different artists each completing intercutting shots of the stump, one of the biggest issues was matching the look. At the start, this issue was compounded by the fact that no one knew for certain what exactly the stump should look like; from certain angles it was completely undefined. We found ourselves needing a gold standard of what her stump should look like. To do this, we relied on a 3D body scan, which we had done at Gentle Giant Studios right after principal photography, for which Annasophia was wearing the stump prosthetic. We took the scan and removed her arm in modeling and completed the missing areas of the stump in 3D.
The Gentle Giant team then made a life-size 3D print of her whole shoulder. This was given back to Mark, who cast the 3D shoulder print in silicon and painted it to match the make-up he had applied every day on set. Now we had something that we could light and shoot. We did these as high-resolution stills on a 7D by placing the stump model on a turntable and shooting it from all the needed angles for any particular shot. These still frames became a library of custom elements that all the artists could draw on when completing shots. It was a weirdly analog approach for the digital world, but it worked great and allowed us to directly maintain control of the look across all the arm shots.
Creating clean background plates over moving backgrounds became something that we found ourselves doing a lot of. Initially, one of the most difficult was to replace the arm over moving water or sand. Fortunately, our compositing supervisor Eric Heavens learned a few tricks in our early testing, including horizontally stretching the backgrounds and discovering what could be achieved by doing so. In a situation where we were replacing the moving ocean behind Annasophia’s green arm, rather than try to patch areas of moving water into the shot, we had much more success stretching the areas of water beside her to fill in behind the green arm. It still required some expert blending to make it perfect, but overall the technique worked well, as the water textures held up, making the stretching we were doing invisible. Having discovered this trick well before principal photography, we were able to tell the water unit team to frame Annasophia in camera with a good area of open water on the camera right side of her body so that we would have something to stretch. They did this and it worked great. We used the same technique on the sandy beach backgrounds with equal success.
How did this work compare to some of the other more recent work you have done?
The last features that we had done were The Single Man and
The Runaways, which were entirely different kinds of projects, and both more normal in their type of VFX. Soul Surfer brought a truly unique set of shots and challenges at a very large volume. It really is not like any previous projects that we have done, but variety is the nature of this business. Almost every film or television show has its own set of circumstances and solutions. The trick is to navigate the unknown using all the skills and tools you have in your sandbox, and in not being afraid to look beyond your normal process for the talent and answers that you need. With
Soul Surfer we brought together fantastic talent and the right pipeline approach, which allowed us to combined all the elements to near perfection, at the production scale of a major motion picture.
What are you working on now?
We wandered into the documentary world this season, besides our usual stable of series and studio projects. We are creating all the VFX for the second season of the Science Channel series Through the Wormhole with Morgan Freeman. It requires lots of cool space graphics, which any VFX artist would enjoy doing. We also are just completing a television special on the construction of the Empire State Building and the new One World Tower in New York. We have produced 10 minutes of period re-creation shots of men working the high steel.
Empire has been a passion project of ours for the last three years and airs on National Geographic Channel in May on the 80th anniversary of the Empire State ribbon cutting.