In its latest project for Marvel Studios, Rising Sun Pictures delivered nearly 300 visual effects shots for Shang-Chi and the Legend of the Ten Rings, the first Marvel film to feature an Asian superhero and the studio’s first film to premiere exclusively in theatres in more than two years. The film is directed by Destin Daniel Cretton.
RSP’s work centered on the creation of a vast, CG environment representing a village called Ta Lo and its jungle environs. The digital jungle serves as the backdrop for much of the latter part of the film and is the site of some of its most electrifying action. “We were thrilled that Marvel Studios trusted us with the development of one of the film’s key assets,” says RSP Managing Director Tony Clark. “The enormity of the set posed a huge challenge for our team, and they responded with their best work.” RSP worked under the direction of production VFX Supervisor Christopher Townsend and VFX Producer Damien Carr.
The Ta Lo set describes a swath of jungle spanning 75 square kilometers. Loosely based on locations in Southeast Asia, it is meant to appear real to audiences but not identifiable as any actual place. RSP artists, led by Visual Effects Supervisor Malte Sarnes, designed the geography of the landscape, including lakes, mountains, ancient forests and rice paddies, populated it with plants and myriad other details, and integrated practical elements shot on set in Sydney.
“We studied forested areas in Vietnam and Indonesia and reproduced the variety of plant-life found there as well as how plants are distributed across the environment. We mimicked the light, humidity and feel of the jungle” notes Sarnes.
Production of the Ta Lo environment occupied the team for months. Artists began by laying out the physical structure of the environment and then added successively finer layers of detail. “We built a library of plants that were interesting and appropriate to the environment, including many varieties of bamboo,” recalls CG Supervisor Julian Hutchens. “Once we had the proxy geometry in place, we began fleshing things out. We ran erosion simulations to wear away parts of the landscape where water would naturally run and carve out deep crevasses. Then we added species of trees and other plants, distributing them based on wet versus dry areas and the amount of sunlight.”
The complexity of a landscape so large presented numerous challenges. “We developed a logic that allowed us to build an environment that looked organic but did not appear overly art-directed or require too many individual decisions,” Hutchens says. “We used procedural methods that were flexible and enabled happy accidents like those that occur in the natural world.”
Finished shots required the integration of live action elements, including actors and a partial set that were shot near Sydney. Getting those elements to blend with the background required detailed attention from RSP’s compositing team. “We had to adjust the live plates, which were shot in the harsh Australian sun, to match the moody mix of light and shadow of our jungle,” explains Compositing Supervisor Guido Wolter. “It required extreme care, especially at the edges of actors, to keep background lighting from pushing through, but the team accomplished it seamlessly. The results are fantastic.”
Extra effort was also required because the jungle environment had to be shared with other studios working on different shots. The entire expanse of the jungle needed to be fully detailed and viewable from any angle. “It was a technical feat because every asset we created, from the library of plant species to cloud shaders, had to be easily accessible to other vendors using different software and methodologies,” notes Sarnes. “It required good communication, tight logistical coordination and a collaborative spirit.”
VFX Producer Jeanne-Elise Prevost notes that the logistics of sharing huge assets between multiple studios were daunting. “There was a lot to coordinate, particularly near the end when deadlines were approaching and shots had to be delivered,” she recalls. “Our approach was to stay positive and focused on the goal of getting the project done and ensuring everyone was happy with the work.”
In addition to creating the Ta Lo jungle environment, RSP contributed to several other aspects of the film. Notably, the studio collaborated with technologists from the Australian Institute for Machine Learning (AIML) to develop a novel technique for facial replacement using Artificial Intelligence. They used it in certain high-intensity combat scenes. “We used machine learning tools to train neural network models of principal actors and apply them to the faces of stunt doubles engaged in martial arts fighting,” explains Wolter. “The output is incredibly real and sets a new bar for believability.”
While initially intended to be traditional 2D and 3D face replacements, the team opted for Artificial Intelligence, with Dr John Bastian and Dr Ben Ward building on the principals of Open Source “Deep Fake”.
Bastian explains the process by which “an 'identikit' of both the stunt double's and actor's faces is learnt against a shared 'dictionary' of facial features. The faces are then detected in the plate, and the machine rebuilds the image using the identikit of the intended actor."
Ward adds “We used approximately 30,000 face images across five characters, training five principal models that combined to over 4 million training iterations”. The team’s work grew to a total of 51 face replacements over 6 key scenes”. In more complex circumstances where training material was not available, performance was driven by digital doubles as the basis for the face replacement.
“This would have been unheard a couple of years ago” notes Clark, “what would have been a costly, labour intensive process, is now we in the hands of Machine Learning, with no special hardware.” Clark adds, “where RSP differentiates itself is with the expertise of Dr Bastian and Dr Ward, and the collaboration with AIML who worked closely with our creative team to produce solutions that demonstrate RSP’s is at the forefront of using AI in its pipeline”.
RSP’s previous work for Marvel includes Captain Marvel and Black Widow. “Trust has been built between our team and Marvel,” notes Clark. “We share common ground and work in the same direction. We knew what VFX Supervisor Chris Townsend was looking for and he knew what to expect from Rising Sun Pictures. Our goal, as always, was to deliver visuals that look fantastic.”
Head of Production Meredith Meyers-Nichols notes that the studio hit its quality and delivery targets despite added pressures from the ongoing pandemic and the occasional workflow disruptions it caused. “Malte, Jeanne and their team pulled together and rose to the challenge, technically and creatively,” she says. “We are very proud of the way they remained focused and produced outstanding visual effects.”