VR Filmmaking: Two Perspectives

The Studio

Alex Laviola is a senior producer with Prime Focus Technologies, where he oversees all IFC Films, HBO, and independent projects, including feature films, documentaries, restoration for Criterion Collection and Virtual Reality. Some of his high profile projects include digital film restoration on HBO's The Three Amigosand  Cannonball Run, as well as online editor for a variety of feature films and commercials. Here, he talksabout his experience setting up workflows for Virtual Reality projects.

Can you elaborate on the concept of creating a VR post workflow. How does this differ from standard workflows? 

Alex Laviola: “At first, the concept of putting together a post work flow for a VR project seemed daunting. We had to go back to our commonly used film practices and deconstruct what seemed like a large undertaking and apply each step of traditional post to it. Doing so simplified the equation. We started to understand that the concept of VR post production was really much closer to film post production than anticipated. As post supervisor, things began to become clearer once we were given the footage from the shoot and started the process.

“With regard to the workflow itself, the project moved along from editorial, to compositing, to online, to color, to audio, to conform, almost as any other documentary project would. To elaborate, it may be worth noting some of the oddities within the project we encountered along the way. We did experience some rather unusual issues with regard to artifacting in the compositing work. For example, in a film documentary, you tend not to hear from Quality Control notes such as ‘man on right's hand is floating in corner of room’ or ‘woman's foot is three-feet long.’ This was where we encountered the most back and forth on what was, overall, a smooth process from beginning to end. Also, in audio, our mixers Kevin Wilson and Paul Levin had the task of dissecting all interview audio and mono-tracking each speaker so that they could be ‘spatialized’ in the VR playback unit. This might be where we would need the most retrofitting technically in our facilities. There were times we had to trust that what we were doing was correct given the detachment of post to VR play back. 

“Being that VR is so new, you can't just hop online and quickly find answers to problems within film's professional community. We had to rely on what we were delivering to Eevo, the film's distributor, would work with what they were using...And it did. At one point in the film, as you sit in front of two people having a discussion, a voice appears from behind you interjecting, ‘I agree...’ and only then you'll notice that there had been three other people sitting right behind you the entire time. 

“To touch a little more on our post work on the project, once we were through compositing and on to our online, grade, and conform, this project was seamless. David Gauff, now our ‘Elder Statesman of VR,’ worked the conform in Resolve, which is also the system Eric Alvarado used to color grade. Once we were finished with the grade, David then applied our work to the media, incorporating all titling and B-roll that needed to be included, and output a stereo ProRes 8K file for mastering.  

David Gauff: "The process of conform at first felt surreally alien because the image on a 2D screen has a complete disregard for perspective and composition. Additionally, the space the subject occupies may not in any traditional sense occupy an area of the 2D screen, which would seem to make sense. However with a bit of faith that the VR headset will reassemble the 2D stitched image into a 3D virtual world everything else can be managed using a very traditional online, VFX, color grade, tilting and mastering rule set. The only other hurdle was the massive amounts of data generated by the 12-camera rig shooting in a documentary style, the subsequent stitched version of all those cameras, and the stereoscopic color graded high resolution master for delivery." 

Is there a significant amount of investment, both time and money, to create these workflows? Is new equipment needed that isn’t already standard in post houses? 

Alex Laviola: “Our investment at Prime Focus Technologies was time. Call it 'sweat equity.' We shot most of the documentary here in our New York office using our DI theater and audio theater as our interview locations. Technically speaking, this was a project that gave us an amazing learning experience and was a lot of fun to work on. Virtual Reality is coming up fast and I believe Prime Focus Technologies, with such vast resources and talent, is delivering quality work to customers. I believe in this even more so now after finishing this project that post houses can thrive in the Virtual Reality community simply due to the fact that the file sizes, necessary online and conform capabilities, and talent extend beyond what the independent filmmaker can now achieve on their MacBook. 

“The investments we were deciding on were basically with regard to stitching software and play back. We worked on some of the stitching using a program called Kolor. This company is now owned by GoPro, which is a perfect match as this documentary was shot using a 12 GoPro cameras. Other investments would include plug-ins to Premiere or Resolve for VR playback and whichever HMD you're producing the content for. Down the road, regarding audio, Dolby Atmos will most likely be the in-house tool of the trade for mixing. If this is the case, we hope the industry as a whole goes this route due to the expense that would be incurred to mix in Atmos. 

“The last bit I can share with regard to new equipment and delivery is the importance of having a running dialogue with the distribution companies. Everything is new here. We're not delivering a DCP, duplicating an HDCAM, or outputting a file to Netflix or iTunes spec. What we're doing is very fresh and must be integrated with software or hardware that is, at this time, more universally unknown outside of the programmers and engineers behind it. Therefore, it was crucial that Prime Focus Technologies and Eevo worked extensively, daily, on figuring out the technical needs and requirements.”

What are the main challenges with doing post for VR? 

Alex Laviola: “The main challenges we faced in post for VR were data management and play back. Monitoring your color and conform work on a Dolby monitor (or any 2D monitor, really) still exposes a gap between the film work and the delivery. We're simply not working within a space that the viewers of this documentary will be watching it on. In an ideal world where Virtual Reality is already a ‘thing’ and not in an experimental, or ‘Wild West’ phase, the playback options would have been rounded out. For example, Resolve and Oculus would have a true realtime play back option that didn't take jumping through hoops to set up. In audio, ProTools and Samsung Gear VR would have an earphone mechanism that allowed you to spatialize the audio and give it a faux-place within the content you're working on. Once these areas of post are smoothed out, you'll see the gap between post and Virtual Reality bridged.

“Regarding data management, we can safely say that the sheer file size of the media we were moving around was time consuming. Having a SAN to share the media across different systems with is as necessary here in VR as it is with any mid- to big-budget feature film. Also, you have to always keep in mind that, even if you're building the offline editorial using one camera as your source, on the back end you're going to have to conform anywhere from six to 12 cameras. Add on the stitching and compositing work to the raw media and this could prove to be a headache. We managed the work successfully at Prime Focus by having storage solutions that allowed us to keep the media in one place and work on different systems.”

Does PFT have any other VR work in process?

Alex Laviola: “PFT has an ongoing relationship with Eevo and Datavized and will be in production of more Virtual Reality content in 2016.”

The Filmmaker

Hugh McGrory is the CEO & founder and Datavized, and was the producer on In\Formation.

Could you elaborate on the experience working with the team at Prime Focus Technologies?

Hugh McGrory: “I've worked on many projects with post production facilities in the UK and US, and there's usually an intangible feeling of resistance, either from the management or the staff, that has to be factored into the workflow as a variable that adds unnecessary delays or uncertainty, especially when a production is a departure from the norm. But the team at Prime Focus Technologies New York had a really refreshing attitude. They were always accommodating, ready to roll up their sleeves and see problems as opportunities to learn and experiment. 

“This VR film was a journey of discovery for all of us. It was about taking apart everything we thought we knew, finding the edges and putting it all back together again in a way that made sense. And that's a big task when there's always other things demanding immediate attention. We all learned a lot as a collaborative team and now the challenge is to put all that knowledge to use in future projects.”

What is the future of VR from a filmmaker’s perspective?

Hugh McGrory: “The most disruptive piece of technology in virtual reality right now is the blank piece of paper and the most radical innovators have an equally malleable state of mind. I come from thinking about film so I’ve brought VR headsets into production houses to give demos to seasoned cinematographers, editors, graders, sound designers. On all previous projects these people arrive with the professional baggage of the right way to do things. They have crew t-shirts from shoots in the 90s, have read all the books and taken all the classes. But once they get their heads around the idea of a 360, 3D, all-encompassing orb, out comes the blank piece of paper.

“And a blank piece of paper is a very liberating thing. There’s nothing on there to tell you what to do. There’s no guidance but also no control. We are usually unaware of the self censorship that comes from trying to fit what we want to say about the world into the locked, linear, rectangular restrictions of a screen. The prison of the medium itself is not exposed until it is taken away. 

“But when it’s gone there’s a transformation that is almost comical to observe in creative people. The ideas for what’s possible turn from the usual trickle into a flood. It’s a great thing to witness this liberation of imagination. It’s another thing to channel it in meaningful directions. Yes, going to the deli would be fantastic in 360 video but calm down, think for a moment, what can we do with this upgrade that we couldn’t do before? Where are the edges between 20th century media and 21st? What are we trying to say/do? What kind of a world do we want to build/live in? 

“What if we forget about making movies, games, live events or anything we can already conceive of, and decide instead to make VR in VR? What can that be? Where’s that blank piece of paper?”