The buzz around real-time production right now is palpable. Games engines are certainly en vogue, and it’s easy to see why — the potential they hold for jump-starting content creation for visual effects, animation, and beyond is undeniable.
That’s why we at Foundry have been hard at work researching new ways to integrate game engines into pipelines, with the aim of accelerating the creative process for artists working on visual effects.
Our focus is on bridging the early stages of production, like pre-production, principal photography, and post-production, where tools like Foundry’s compositing software Nuke are used as standard. For example, virtual production using LED stages often needs a lot of work in post-production, but what if you could visualize better results on set?
Enter the UnrealReader Node, an experimental tool for doing VFX work in NukeX and Nuke Studio using data generated from Unreal Engine. Crucially, Unreal Reader makes it quick and easy for compositors to generate live renders from Unreal Engine and control the results, for example by breaking objects into layers or tweaking shot framing, all within Nuke’s already-powerful tool set. It will be released as a beta feature for Nuke 13.1.
The UnrealReader Node works by connecting Nuke to Unreal Editor over a TCP/IP connection, with Unreal running on the same machine as Nuke or on another machine. Once this live connection is made, Unreal Reader renders images and utility passes from Unreal and delivers them to Nuke on-demand. It also allows cameras in Nuke and Unreal to be linked, enabling Nuke’s powerful camera tools to work against the final shot. What results is a workflow whereby two major tools work in harmony to help content creators achieve their perfect final image quicker, and at the highest possible quality.
The UnrealReader Node has enjoyed a decent amount of stress-testing by studios and artists already, and we identified a number of opportunities for both off the back of these use cases.
Quality Control: Achieving that Final 10%
Some studios are turning to Unreal Engine to create animated content in place of using a traditional offline renderer. Creators can build short- or high-volume content such as children’s shows in real time for quicker turnaround time and project delivery.
The quality problem persists here, however. While game engines are being used more in animation and VFX, they provide less modular and lower fidelity output than artists are used to from a standard offline renderer. Simply using Unreal Engine often doesn’t get an artist to the final quality needed, and extra final tweaking, color grading, and so on is done in Nuke. Where standard Unreal rendering may get you to the 90% mark, the UnrealReader Node lets you pull data across easily into Nuke to push your creations over the line, to achieve that final 10%, and, more importantly, to achieve effects that are simply impossible using a game engine.
Speed and quality combine via Unreal Reader to give artists the best of both worlds: the efficiency of Unreal Engine, and the pixel-perfect results so synonymous with Nuke.
Propping Up Virtual Production
The UnrealReader Node has also proved its worth in supporting virtual production workflows, particularly those involving LED stages.
Where directors may want additional elements that LED volumes aren’t physically capable of providing — such as wide shots, set extensions, and so on — greenscreens can be put on either side of a smaller volume and the camera then tracked and everything brought into Nuke. On one particular production, the team rendered the whole shot in Unreal, brought it all back into Nuke via UnrealReader, and then re-rendered it with the set extension. The original shot captured on the LED volume and rendered in Unreal was wiped out and replaced with a bigger, higher-quality shot, in front of which they composited the live actors. This was all done on -set, and had the added benefit of allowing the director to tweak the Unreal original scene on the spot in Nuke after the shot was captured in-camera.
Although they are often more complicated and expensive to manage, the biggest benefit of using LED volumes is the lighting. They’re fantastic at lighting physical elements in a way that feels natural and avoids the green hue of greenscreens. Because of this, volumes can negate one of the most expensive, time-consuming processes in post-production — fixing up lighting.
In our example, the on-set VFX supervisor was able to take the in-camera VFX and then render out the extensions and a high-quality version of the LED background within a matter of hours. When the director wanted a change in the shot, it was a simple case of moving objects around in Unreal, hitting a button to re-render, and then the final image composite appeared in Nuke via the UnrealReader Node. Thanks to the LED volume, the lighting was more closely aligned to what the director had in mind, leaving more time for compositing artists to focus on other areas of the shot in Nuke.
Subtractive Rendering for Faster CG Projects
A particularly ingenious use of the UnrealReader Node came out of a studio’s desire to combine control, flexibility, and speed over their CG creations.
The challenge centered around their rendering pipeline; they typically needed very specific render passes and the ability to pull out individual attributes on individual objects, whether it be the specular part of certain background objects or control only the reflections captured on their surface. This is very typical of high-end, fine-grained CG work whereby studios want a massive amount of control over the imagery.
In our example, the studio, limited by Unreal’s rendering architecture, used a string of UnrealReader Nodes and employed a technique called subtractive rendering. Instead of generating the desired image, the team generated a sequence of images that weren’t wanted, but that contained the element that was. They then subtracted the sequences until left with the desired information, getting exactly what they wanted. This was all accomplished thanks to Nuke’s powerful compositing tools, while also benefiting from the rendering efficiency of Unreal.
Ultimately, the team were able to get passes out of Unreal that it was otherwise completely incapable of rendering, by using math inside of Nuke. The whole experiment is testament to the Unreal Reader’s versatility; it provides a level of flexibility over Unreal that people yearn for when they're working just with Unreal and subsequently hampered by the limitations of a game engine.
Two Tools, One Workflow
I was fortunate enough to host the artists, experts, and studios involved in the above use cases as part of Foundry Live’s recent Research and Innovation Showcase. In this session, each speaker expands on the aforementioned examples and how they’ve been using the UnrealReader Node so far, and how they’ve been putting real-time technology to work on recent projects.
One common theme emerged from the panel: the beauty of Unreal Reader is its promise of connecting two disparate tools and workflows. This is especially for burgeoning artists new to the tech. It allows for artists to learn both Nuke and Unreal in conjunction with each other, rather than independently, safe in the knowledge that anything rendered in Unreal can seamlessly transfer into Nuke.
The ability to use Unreal as part of a post-production workflow reflects Foundry’s wider belief that real-time tools like game engines are going to be a part of everybody’s tool kit moving forward. The reason we’re hard at work on things like Unreal Reader is to ensure that people who work with Foundry products can get the best out of everything else they work with in their pipeline. Whatever process or technique they're using, we want to have the tools that compliment, accelerate, and improve the quality of what they're trying to do.
If a piece of technology is here to stay, our long-term research plans begin to orient around it, much like they have for cloud and machine learning. In the case of game engines, real-time rendering and virtual production, the question for us becomes: How do we make this something that’s sustainable and pipelineable for years to come? The UnrealReader Node is just the start, and Foundry is currently forging a future which promises more in this space.
For more insights into more related topics, visit Foundry’s Insights Hub here.