Pre-production in Real Time

Some might consider the pre-production work Dan May and his Painting Practice company do as "gameplay." And, in a sense, it is.

Painting Practice, a West London design studio May founded more than a decade ago, can be described as a previsualization/VFX facility, offering services such as previs, animation, VFX, production design, concept art, and digital matte painting. It has recently garnered attention for its design and visualization work for the large sets and complex action sequences on HBO's His Dark Materials. And they did so using real-time technology by implementing Unreal Technology's Unreal Engine.

His Dark Materials is an HBO epic fantasy drama series produced by Bad Wolf Studios and based on the book series of the same name by Philip Pullman, the first of which, "Northern Lights," was the basis for the feature film The Golden Compass (see "Animal Control," CGW, December 2007). The story introduces Lyra, an orphan, who lives in a parallel universe where science, theology, and magic are entwined, and where everyone has a personal daemon, a manifestation of his or her soul in the form of an animal.

A look at scene lighting for Trollesund.

The feature film was ambitious, yet the television series is even more so in terms of its scope and scale, with a plethora of creatures from witches to armored polar bears, and mythical cities and vast landscapes created by CG artists at Framestore (see "Crafting Daemons," at the end of this feature). Before that happened, though, Painting Practice provided real-time mock-ups of various sequences that enabled the director and producers to make decisions concerning the series' complicated sequences.

"Unreal was a great way of getting people to physically be in a space that was not even conceived," says May, creative director at Painting Practice and one of the company's co-founders.

As Joel Collins, executive producer and production designer on the show, notes, "These are very expensive sets, and on a show like His Dark Materials, where you're building an entire town for one episode, you've got to make really good decisions that are practical, financial, and creative. What Painting Practice's team created gave us an ability to make critical judgments that meant we could get it right on the edge of affordable and, absolutely, on the edge of shootable, in the sense there was no fat on what we built."

Painting Practice originally was set up to facilitate production design, whether physical or digital, but as more greenscreen sets were being incorporated, the company's work on previs became more plentiful, though it still does quite a bit of animation as well. "Primarily we're first-in-the-door, working for producers, directors, or showrunners on the vision of the show," says May. "We often help get them off the ground, and then we work through the traditional pre-production art department space, previs, and then in the last few years, with Black Mirror and now with His Dark Materials, we're very much hand-in-hand with editorial and the effects-room postvis, which is obviously a quite complicated and involved process because it involves spending money on things that might not even make it into the cut, but has to be good enough for people to make informed decisions about."

In some instances, the company will do final shots, especially if it is an environment and within the company's strengths, whereby it will do final matte paintings, as they did on the Netflix dystopian sci-fi anthology series Black Mirror.

"We're not looking to be a post house," says May. "We're purely there to serve and help facilitate productions or post houses to get them through design or tricky problems, whether it's an animation problem or [involving] previs storytelling."

Embracing Real Time

According to May, he had been looking at VR and game engines for some time as way to explore a space. He used the real-time technology to build the night club for "San Junipero" (Season 3 Episode 4) of Black Mirror. "Rather than just doing a few visuals and white-card modeling, I went the extra mile and built a VR experience that the director and producers were able to look at and experience firsthand," he says.

May began exploring this medium further, examining the use of game engines beyond gaming applications.

"His Dark Materials is full of complicated challenges. The show has a good budget, but with the ambition of the show, that budget is quickly stretched. So, you have to come up with creative solutions and find ways to resolve situations, and this is often the technology that can do it for you," May says.

Painting Practice uses real-time tech to aid design and vis for His Dark Materials.

A look at a smoke-filled scene of Bolvangar.

While filmmakers like James Cameron and Jon Favreau have been using virtual cameras to scout locations for some time now, the process up to now has been out of the cost range for television. But, Unreal has opened up that access to far more people.

Nevertheless, the integration of Unreal Engine into the process called for a retooling of Painting Practice's pipeline. When the company began its work with the engine, it had difficulty getting the different software to communicate, whereas now there are more scripts and plug-ins available to make the process easier and more automated.

For the most part, Maxon's Cinema 4D serves as Painting Practice's backbone - "It's great for previs and postvis, for effects work," notes May. In addition, the facility employs the Adobe Creative Suite, along with Adobe After Effects, and Premiere for editing and Pixologic's ZBrush for complex environment and character designs. The group also uses Frame.io for collaboration as well as delivery, archiving, and referencing, along with Quixel products (Bridge and Mixer) and PureRef, freeware for making giant art boards.

Worlds and Characters

In essence, Painting Practice's Concept team for His Dark Materials on Season 1 (Season 2's release date is still pending) worked with the environments and creatures teams as well as with the production designer to help determine what the worlds and characters would look like. They would then review that with Framestore and Russell Dodgson, VFX supervisor.

"Sometimes the directors wouldn't come on board until late in the process, and we'd need to plan a lot of the sequences. So, it was my job, really, to have a go at making those sequences, with various outcomes and different tone levels," May explains. "Then when the directors would start, we would show them a bunch of things. Sometimes they would like the ideas or the principle, or they wouldn't necessarily like all the cameras or the speed or tone. It would be a hit or miss, but a lot of the time we got it quite close, and then we'd make the changes. But, it is really important and essential that the directors have creative ownership of the sequences so they can shoot some of that previs; otherwise, [the work] becomes null and void."

As May points out, there were many instances on Season 1 where the work paid off and was used shot-for-shot, and other times when it was used but was filmed a bit differently although the elements and beats were the same; alas, there were even times when everything changed, particularly when there were script alterations.

Unreal Engine enables the director and showrunners to plan out scenes.

A visual comparison of work in Unreal (left) vs. broadcast footage (right).

One of the particularly challenging sets Painting Practice had to design for His Dark Materials is Trollesund, the main port of Lapland in the world of Lyra Belacqua, the story's heroine. For this, the team made a scan of a quarry using photogrammetry drones and used it to create a high-resolution model to import into Unreal Engine. To this, the group added additional models and textures to complete the real-time mock-up.

The previs enabled the director and producers to explore the conceptual environment and make key decisions concerning the scale of the buildings and the set. The previs also aided in the storytelling aspect, so even though the area was called Trollesund Port, not all the action had to take place dockside, providing more flexibility for constructing the set, which was then extended digitally.

An overview of Trollesund by Painting Practice.

The environments are complicated, but a good deal of the series' budget from the start went toward the complexity of the daemons, which were photoreal, not stylized. After all, the characters are a big part of the story.

"His Dark Materials and Black Mirror have an air of fantasy and science fiction [respectively], but there's also a reality, a grounding, and that's the space where we fit well," says May. "And, they have a raft of very complicated, weird, and wonderful challenges that we aim to solve."

Another critical piece of previs the company provided was for a fight between two of the series' armored polar bears. Episode 7, called "The Fight to the Death," involves mortal combat between Lyra's friend, Iorek Byrnison, and the usurping king Iofur Raknison. According to Jamie Childs, who directed the episode, being able to experiment with a virtual camera on a set that did not yet exist enabled him to explore the best camera angles to maximize the scene's believability.

"I wanted to walk around and look at the bear fight because I wanted that fight to feel like it was a real fight being shot, not a CG fight," says Childs. "I could actually do that with Unreal. I could move around that room physically, get the camera in the position I wanted, and see on my camera monitor what was going on, and record those shots and cut it together - and that was really freeing for a director."

A sample of a video plan involving the characters in the popular series.

According to Childs, the sensation reminded him of his early days learning his craft, when he would go out and just film things to try them out, letting his creativity find the essence of the story. "I didn't really think that previs would help me do that side of things," he says. "I thought it might help me with technically [determining] where to put a camera and things like that, but it actually made me go, 'Right, I don't need to worry about any of the noise; I can actually just go and create something.'"

In the last episode of Season 1, "Betrayal," a great battle plays out with exploding airships firing at the bears and Lyra. The Painting Practice team dropped the director and DP into the virtual environment, some of which "never saw the light of day due to flooding [at the real location]," recalls May. Initially, that scene "blew the VFX budget out of the water because we were having so much fun," he adds. "Then we pared it back to what we could then shoot, although it was a painful process to create something that was really interesting and fun and quite beyond our wildest dreams, then smash it into pieces in a reality check."

The previs saves valuable shot-planning time on the complex production.

One of May's favorite and most successful sequences occurred in Season 1 Episode 4, "Armour," as Iorek retrieves his armor. "We planned out this town and built it in the game engine, and could visit it virtually inside the studio, rather than having to drive an hour and a half into the Welsh mountains," says May. "And we were able to make complicated and very cool animations of the bear running around and trashing the town. We were able to get in there and discover shots with the director [Otto Bathurst] in the various sessions. We would make multiple versions of the sequence, do an edit, and then have different versions of the edit, with different endings. We ended up with something that could be shot, was achievable, on budget, and Framestore was happy with the volume of shots. It was a collaboration among all the partners that pulled off well."

Plan V

Soon interest piqued concerning Painting Practice's proprietary tools and Unreal Engine setup. The company received funding that helped push further development of the tools, and after some fits and starts, the visualization/effects company released Plan V. A new software app based on Unreal Engine, Plan V is a bespoke studio environment that enables artists, producers, directors, and others involved in a production crew to experiment with lenses, storyboards, previs, and more.

With a simplified workflow and a user--friendly interface, Plan V is designed to enable less technically inclined users to interact in a high-quality 3D environment so they can design worlds, sets, and scenes for films, television, advertising, and games. The tool also supports local and remote collaboration.

"You can get an amazing visual look out of [Unreal Engine] very quickly, and there's an incredible amount of assets that are now available [for it], so it makes building environments super quick. But expecting directors and writers and others who are not necessarily technically-minded to pick it up is a bit of a tall task," says May. Plan V makes viewing the Unreal Engine scenes simple and affordable.

Moreover, as crews continue to work remotely for whatever reason - be it pandemics, travel costs, personal or professional obligation - tools like this will become even more valuable.

As May points out, this is such a fast-paced industry, and it is more important than ever to keep relevant and keep moving forward.

Using Painting Practice’s Plan V.

As of this writing, work is in progress for Season 2 of His Dark Materials. Season 1 was a learning phase of sorts, as the group became familiar with the virtual production tools and cameras as well as the operation of the engine itself and the resulting collaboration with the vendors. On the upcoming Season 2, the group pushed the usage further, creating a virtual space where the actors could better understand the sight lines and so forth. And as the seasons progress, the technology will become even more useful, May predicts, in handling the increased fantasy elements and pushing the storytelling.

"A lot of Season 1 was just trying to get things to work and discovering cool ways of working and making some great sequences. In many ways, we made great things in previs that we probably couldn't make for real, ultimately. But doing those techniques and finding discoveries were still useful," says May. "Even though we were limited by our imaginations in Unreal, we had budgets, health and safety, and a million other things to consider. You have things in previs that we probably couldn't make for real, ultimately. But doing those techniques and finding discoveries were still useful," says May. "Even though we were limited by our imaginations in Unreal, we had budgets, health and safety, and a million other things to consider. You have to get realistic about what's achievable in terms of time and money."

In addition to His Dark Materials, May and Painting Practice are using the virtual pre-production process on other shows, which are under NDA at the moment.

"I think we're still in the early stages about how it's going to be used," May adds.

Meanwhile, the series recently received two BAFTA awards, including one for visual effects.