LOS ANGELES — International creative design and VFX studio Pixomondo (www.pixomondo.com) has evolved at the pace of the entertainment industry, bolstering its technology infrastructure and diversifying capabilities along the way. Founded in 2001 by CEO Thilo Kuther, the company spans six offices on three continents and offers a range of services starting from award-winning VFX for film, TV and commercials to interactive digital content, VR, live events, theme park entertainment and original IP. By sharing render resources across facilities, Pixomondo maximizes efficiencies; its globally dispersed farm runs 24/7 thanks to a proprietary process built on top of Thinkbox Software's Deadline, on which the company standardized in 2010.
“Part of what influenced us to integrate Deadline so deeply is its open architecture and usage of the Python programming language. With the source code for application integration available, we can constantly tweak the software to our own specifications; it’s very extensible. Also, the Thinkbox crew has always been willing to grow with us and is quick to respond to our needs,” shared Patrick Wolf, Pixomondo’s Global Head of Pipeline.
Deadline high-volume compute management software is used to render, manage and process files locally and across the cloud. Pixomondo primarily leverages Deadline for remote rendering between branches. Wolf and his team customized Deadline’s job submitters to allow users to select the branch(es) to render in; they also extended the Deadline GUI to allow render wranglers to split local jobs to render on one or more remote farms simultaneously, a feature that has been extremely helpful on tight deadlines.
When a remote job is sent to the farm, event scripts in Deadline automatically determine the job dependencies. Necessary file transfers are queued and then a replica job is created on the remote farm recreating the settings of the user that submitted the job locally as well as remapping job priorities, pools and group configurations. As soon as a frame is rendered remotely a Deadline after render event informs Pixomondo’s Celery-based global queuing system. Celery then triggers the transfer of the rendered frame and also marks the task complete on the local side so that the remote render is transparent and seamless to the artists. After the remote render is complete and all frames were verifiably transferred back, a cleanup job removes the frames on the remote side to save storage space. Celery is then used to mark the job as completed locally, which allows dependent follow up jobs to run like a proxy image generator. Pixomondo’s production management system, Autodesk’s Shotgun Software, is also notified when the render finishes. Currently, the studio solely uses physical computing resources but Wolf is already investigating the integration of cloud rendering to the workflow.
“One of the cool things about our remote rendering setup is that every step is a separate dependent Deadline job with automatic retry and failover. Since every job has log files attached, you can easily see and troubleshoot if a connection drops or there is some other issue. It’s very easy to monitor how a job is performing, whether it’s local or remote,” Wolf explained.
Deadline jobs’ associated log files provide a wealth of statistics that help Pixomondo optimize its render farm performance. For “Furious 7” and “Game of Thrones” Season Five, supervisors concerned with RAM usage leveraged metrics from Deadline job log files to guide how they utilized various machines. Another appreciated Deadline feature is bracketing: for example, rather than waiting days for frames 1-100 to render, Pixomondo requests Deadline to render frames 1,100,50,25,75 first, then every 5th frame and only then the rest. This allows artists to quickly determine render issues before wasting resources on executing a complete pass.
In addition to remote rendering, Pixomondo taps Deadline for file transfer and its asset library. With file transfer, coordinators can select a playlist with hundreds of associated versions in Shotgun and queue them for transfer to another branch in Deadline. Deadline then automatically retries failed transfers, provides detailed logs and notifies users on completion. For asset archival, once a show finishes, Pixomondo selects the assets to archive in Shotgun and then a script generates Deadline jobs that asynchronously perform the actual archiving on the farm, allowing for hundreds of assets to be archived simultaneously, without blocking the computer. The archived assets are then automatically entered into an Asset Library project in Shotgun. The restore is again triggered from Shotgun and kicks of a custom Deadline asset restore jobs. In case an asset isn’t yet available locally, a transfer job takes care of pulling it from a remote branch. On completion of the restore Deadline automatically notifies the user via email.
“Deadline is incredibly fast. With its new database repository, I can connect to a remote branch easily from my desk and watch the jobs output as it occurs; I don’t have to wait until a job is finished. To maintain consistency, we've committed all our repositories to a version control system so that we can change any of the plugins in one location and it will apply the changes across the other branches,” said Wolf.
Further simplifying workflow customization, Deadline’s event framework allows Wolf to hook into any stage of a job on Pixomondo’s farm and tailor its execution. This allows for useful functions like the ability to customize jobs directly after submission in one central place, rather than being required to implement these code changes in every plug-in. Regarding applications, Wolf rarely encounters one not already supported by Deadline. Additionally, Deadline’s pools groups and limits groups, make resource sharing frictionless.
Wolf concluded, “The beauty of Deadline is that we basically don’t have to worry about it processing our jobs. And if we do have to investigate the level of detail we’re able to pull out is fantastic, it’s extensible and it’s very user friendly. We’re happy that we standardized on it and really like how the feature set keeps growing year after year.”