When people ask me what I do for a living, my response varies: from the simple, “I work in the movie business,” to the more informative, “I am a visual effects artist,” to my actual job, “I am a digital compositor.”
The hoi polloi understands the first response, and if slightly acquainted with the industry, they will understand my second answer as well. However, most do not know what a compositor is.
A compositor combines digital images to produce an integrated result, combining elements to make the final result look “real,” as if the image was shot that way. Everyone has an innate ability to look at something and discern whether it is real or not real. We have been trained in this faculty since birth.
Ever since the first movie with VFX was released, the audience has been fascinated by the ability to create fantastical images that look “real.” Technology has allowed movies to look more indistinguishable from real life than ever before. Even so, audiences receive a daily bombardment of “training,” so fooling them into believing the unbelievable becomes increasingly difficult.
The need for high-end visual effects compositing has never been so great, and the need is accelerating. Visual effects films are becoming more and more prevalent; even movies that wouldn’t be traditionally classified as “visual effects” films land in the compositor’s hands. This need for high-end results isn’t limited to the film world; television is also demanding a higher quality from visual effects.
Compositing requires not only the inherent ability to discern what looks real, but also the ability to take that knowledge and translate it to the actual manipulation of the image to achieve this reality. This requires a thorough understanding of color theory and design, a firm grasp of technical knowledge to control the software tools, and an artist’s eye to transform an image into that seamless, natural look we all take for granted but know, even subconsciously, if it’s not right.
The arrival of new technology is accelerating, resulting in an increasing number of tools and techniques in this area. This dynamic nature of technology allows the boundaries to be pushed with every new software release. Effects once considered difficult to achieve are now more commonplace, creating the necessity to produce something never seen before. The audiences’ hunger for visual stimulation is an insatiable one.
The range of development and improvement in the industry is too extensive to cover here, but I want to share some ideas. Designing and implementing stunning effects requires teams of artists. An important cog of that machine is fostering a collaborative workflow; merging 2D tools and 3D tools is a positive step. I see the merging of 3D and 2D responsibilities a concept integral to the forward progress of digital compositing. In addition, the continued use and development of technology, such as optical flow, has the potential to save time and allow the computer software to assist in tasks often too difficult to accomplish by hand.
3D Compositors
Traditionally, 2D compositing pipelines have been separate from a studio’s 3D pipeline. The compositors’ main purpose is to take the images and elements delivered from 3D and integrate them into the film background plates. A certain separation of talents and understanding was built into the process, which is reasonable considering the technical expertise required in each area and the limitations of available software.
However, lines of division between a studio’s 2D and 3D departments have been blurring in the larger facilities, and this trend will filter down to smaller studios as well. Compositing software that usually targeted only 2D image manipulation is now extending into 3D space. An example is Adobe Photoshop CS3. While Photoshop is not considered a film compositing tool and is in a different category altogether, it is software in which 2D and 3D technology are merging (see “Brave New World,” March 2007, pg. 36).
Both lighting 3D elements and texture control have been essential parts of any 3D application, though usually they are not in the realm of the compositor’s influence. However, the latest compositing software is putting those tools in the hands of the compositors. The ability of a compositing artist to adjust lighting and textures within the 2D composite allows for more control of the final look and decreases time between revisions.
Bottom line: Dedicated compositors in today’s market need to have a substantial grasp of 3D in addition to the standard 2D knowledge.
Optical Flow
While in use for more than a decade, optical-flow technology is still in relative infancy. Optical flow effectively is the computer’s ability to see and understand an image. Optical flow analyzes a sequence of images to determine motion vectors. The motion vectors, in essence, allow the computer software to distinguish shapes, motion, and action in the frame. Once the software has an understanding of the movement in the frame, it has a variety of uses for visual effects, including but not limited to tracking, frame repair, and motion blur. Some of the possible benefits of optical flow are done manually, such as object removal for clean plates and rotoscoping to create mattes. However, the use of optical flow can allow automatic object removal and automatic matte creation, thus saving time and creating pixel information that would be time-consuming or impossible to generate manually. In the past, the most common use has been for retiming shots. Through optical flow, the software can accurately produce missing frames in a sequence, thereby allowing for smooth slow-downs normally impossible by traditional frame-blending methods.
Optical flow is very computationally expensive, but with faster machines and new developments in the efficiency of the technology, it has the potential to become vastly more widespread and offer more tools for the artist.
Available Compositing Software
The trend of combining 3D and 2D workflows and the adoption of optical flow can be found in the latest releases of the most popular compositing software. Composting software available for high-end work has been in a state of change for a while. Unfortunately, there is a relatively small pool of competitors within the niche market of film visual effects. Large visual effects houses have dedicated programmers solving production-based technological issues on a daily basis, and often these facilities have their own proprietary compositing software. But there are four commercially available products that I regard as viable options for feature-film compositing. They are in different stages of integrating 2D/3D and optical flow, but are all moving in that direction. They are: Apple’s Shake, Autodesk’s Toxik, Eyeon’s Fusion, and The Foundry’s Nuke (formerly from Digital Domain). While other packages are available that could fit into this group, these four are the widely accepted options.
Apple announced that it has stopped the development of Shake with Version 4.1. However, Shake is still quite potent in the film compositing world and remains a contender. It is especially attractive considering that Apple has reduced Shake’s price significantly. If Apple releases a new product, I imagine that it would closely align to the company’s Pro applications a la Final Cut Pro, encompassing a larger market than the small, specialty market that Shake has made home.
Autodesk’s Toxik, although relatively new, has had an extensive development process with all the resources afforded by a large firm. It has been built from the beginning to handle an HDR workflow and large images. Unlike other compositing applications, Toxik has a database backend that is designed to tie in with the production workflow. The first releases had a limited tool set available, but with Autodesk’s extension program, more and more tools are being added.
Eyeon’s Fusion had a major code rewrite with Version 5. While the transition from Version 4 to 5 was technically only an upgrade, for all intents and purposes, Version 5 is a new product, due to the code rewrite. Eyeon recently released 5.1, which includes a number of improvements, including eight new tools and much-needed upgrades to the paint tool.
And last, but currently not least, is The Foundry’s Nuke. The Foundry recently acquired Nuke from Digital Domain. The Foundry creates high-end visual effects plug-ins, such as Furnace, Tinder, and Keylight, and uses optical flow in a number of its products. The acquisition positions Nuke, a strong product, on even stronger ground with a larger dedicated development team and frees production studios from buying what was often thought of as a competitor’s product.
Software development is allowing the compositor to step outside the 2D world and become part of the 3D fold. Technology such as optical flow can harness the power of the computer, allowing software an understanding of an image that once was in the realm of science fiction. This acceleration of technology is a basic truism and empowers all artists to push image creation to a new level.
Ryan Smolarek is lead compositor for Digital Dimension, a six-time Emmy Award- and two-time Visual Effects Society Award-winning visual effects and animation studio.