An Open Book

Creators of the interactive VR narrative Baba Yaga recently brought the fairy tale to life, and what better way to do this than with a storybook aesthetic?

Baba Yaga is a contemporary portrayal from Eastern European/Slavic folklore about a supernatural being assuming the appearance of a deformed, old woman who has a strong connection with wildlife and the forest, where she lives. She may choose to help or hinder those who encounter her or seek her out - an aspect of the legend that melds well with the VR platform where a "player" is able to make options that impact the outcome of the story.

Crafted by Baobab Studios, the VR film is directed by Baobab Co-founder Eric Darnell (Madagascar, Antz) and co-directed by Mathias Cheleboug, with star power behind the character voices. The sixth virtual-reality film from the studio since its inception five years ago, Baba Yaga pushes beyond Baobab's previous projects with a higher level of stylization, the addition of humans as real--time characters, and with more variability affecting the story outcome through decisions that are meaningful and matter.

The player assumes the role of Sasha, who journeys with her sister, Magda (Daisy Ridley), into the forbidden forest (voiced by Jennifer Hudson) to seek a special blossom in order to save the life of their dying mother, the village chief (Glenn Close). But the witch Baba Yaga (Kate Winslet) does not look kindly onto those who trespass on her land, as for far too long she has used her powers to stop the villagers and their settlement from encroaching upon her enchanted forest. How will Baba Yaga react to this latest intrusion? Each decision the player makes along the trek impacts the narrative, the witch's reactions, and the ending.

As Darnell points out, there are many folktales about Baba Yaga, and the story the filmmakers came up with is essentially their own, an amalgam of pieces from those versions as well as elements of their own.

To make Baba Yaga available to a wider audience beyond that of the Oculus Quest platform, Baobab created a 2D film version as well. Also presented through the eyes of Sasha, the theatrical adaptation was filmed from the player experience of Darnell. As the cinematographer, Darnell shot almost all of the virtual camera work himself to achieve a first-person emotional POV.

A Classic with a Modern Twist

Baobab presents the tale in a storybook--style fashion, giving the 3D imagery a stagecraft - handcrafted and theatrical - feel. "We're giving you that experience you had as a child looking at a fairy tale in a book, but now you can actually go into the book and into that fairy-tale world, and not only stand in it, but be a part of the story," Darnell explains.

The framework of Baba Yaga is based on the literary construct of presenting a prologue, epilogue, and chapter headings in between scenes. Those interstitials are narrated by Baba Yaga herself, giving the viewer an understanding of how the story is progressing, what transitions are happening, and which personality the witch embodies. As Darnell points out, people are familiar with chapter headers in books, so it's easier for them to make that leap into a storybook world such as this.

Technically, it also allows for easy set swaps on the mobile VR platform as well as properly orients the viewer for upcoming scenes.

Meanwhile, the storybook prologue sets the stage for the experience, explaining how the humans settlers began challenging the power of nature, prompting the protector of the forest, Baba Yaga, to cast the spell on the chief.

According to Larry Cutler, Baobab co-founder and CTO, the prologue and epilogue from the start were designed to look like a 360-degree pop-up storybook with 2D cutouts crafted by production designer Glenn Hernandez and animated by Ken Fountain, supervising animator. "You are in the middle of this 2D world that is supposed to be flat, but it also has depth. So, it has to work in 3D," says production designer Matthieu Saghezchi.

The 3D imagery has a stagecraft feel, giving it a storybook aesthetic.

Setting the Stage

The overall aesthetic is an infusion of illustrative 2D pop-up animation, hand-drawn, and stop-motion styles, although crafted in CGI. As a bonus, this design conceit meshed well with the constraints of having to run on an Oculus Quest: The filmmakers are telling a story whose content is rendered in real time at 72 fps on what amounts to a mobile phone chipset on the Quest.

"It's very easy to have complexity for complexity's sake, especially in CG because you've got volumes and you've got the reaction to light and all these things we love to see. To consciously step away from that, though, is an interesting artistic and even technical challenge. And there's certainly a technical advantage to keeping things simple; you are always looking at how to get as much data as you can through that relatively tiny pipe," says Darnell.

Using the stage language of the story, small islands of content pop in and out of the light as the story unfolds, as opposed to having large, open environments that the player navigates through. Since technical limitations prevent dynamic lighting in VR, most of it was baked into the textures, and then the individual materials were animated to transition from one lighting scenario to the next. So when the lighting changes, the scene switches from one set to another.

A large part of the story takes place in the forest, which is a mixture of both 3D and 2D elements, with a complicated set of 2D cutouts representing the trees. The forest, which is very dark, comprises complex layers involving 20 lighting transitions and blended lighting setups (over 40 different ones). "As it turned out, that was really hard to do in VR because having different lights that cast shadows is very expensive," explains Cutler. "However, that lighting really added to the fear you feel when you're in the forest, and it also made [the experience] feel like a storybook and a fairy tale."

Content was generated procedurally based on the decisions of the player. So, early on, the script was separated into chunks and coded into behavior trees. Later, the filmmakers moved into a VR storyboarding process whereby the editor and storyboard artist/animator worked inside the Unity game engine where the interactivity was combined with the narrative.

The asset creation process, especially the modeling and rigging, was more closely aligned to that of an animated film. The animation has a stop-motion, handcrafted style, animated on 2s within Autodesk's Maya, which was used mainly for all the front-end work, including modeling and rigging as well. The assets were then ported over to Unity, where the surfacing and further animation were done, along with lighting, effects, and interactivity.

The studio's pipeline includes a number of standard industry software, along with some proprietary tools, including the all-important non-linear Real-Time Storyteller Platform, which Baobab has been extending significantly with each subsequent VR project. Built on top of Maya and Unity, Storyteller's initial function was to enable the group to create feature-film-quality characters in VR, but has since expanded to perform other tasks, including cinematography and making the review process more robust.

"We have our own unique workflow to take the animation we create in Maya and get it into the game engine in a way that is more expressive than a typical game rig you would have in a typical VR experience," explains Cutler. "Then everything else is done in Unity; there are AI systems for the different characters and interactivity associate with it."

Storyteller synchronizes the various processes and enables the team to combine handcrafted animation with procedural animation, inspired by the AI, to uniquely complement the viewer's actions within the adventure. This enables the animation team to elicit as much character empathy from the audience as possible.

Character and Environmental Concerns

Within the chosen design conceit for Baba Yaga, viewers will see a lot of simple graphic shapes, especially in the environments. In fact, the graphic, stylized look of Baba Yaga translated throughout the production, even to the characters - though they are more volumetric, which allows them to integrate well into the environments. "They may not look like real people in the real world, but they needed to act like real people in the real world," points out Darnell.

Baba Yaga marks the first time the studio has created human characters in one of its VR films. While crafting a believable human character in computer animation is never easy, to do it in VR - where it has to run at a high frame rate and the characters have to achieve subtle, nuanced facial movements necessary to convey information to the player and be responsive to where the player is and what the person is doing at all times - requires a great deal of complexity.

Magda appears throughout most of the film, and her facial performance is important as she has to influence the player assuming the role of her sister. "Magda features a lot of work that is new and difficult to do well in real time, such as cloth and really nuanced facial expressions," says lead rigger Justin Fischer.

In addition, texturing and surfacing of hair, skin, and clothing is an important detail to consider in VR, since the viewer can be standing close to the characters and can take time to note how the hair falls or how the fabric looks and moves. "Our pipeline can handle pretty complex deformation, but instead of relying on cloth simulation, we chose to muscle through and hand-animate her clothing, and it gives it a really nice tactile feel," says Fountain.

The character Baba Yaga wasn't without challenges either, with her unusual shape and layers of clothing modeled on top, which deliver her striking silhouette. Yet, she needed to move in a believable way so that her performance could be understood, despite her mostly tubular shape, making animation and rigging fairly difficult. "We had to be careful about how we pushed and pulled controls to avoid collapsing her midsection or creating creases in her neck," adds Fountain.

Add to that the fact that Baba Yaga wears a mask which covers her facial expressions, making her body motions that much more an important part of her performance.

"We put all this painstaking work into this straight-ahead character animation, where you can believe there is a living, breathing thought going on behind the eyes, that there's this illusion of life," adds Cutler. "I've spent my whole career in animation, and Eric [Darnell] is an animator at heart. For us, having a strong character performance that combines really broad emotions and really subtle, heartfelt moments were important but something we hadn't seen before in VR."

Baba Yaga marks the studio’s first VR film to include human characters.

Moving the Craft Forward

With each new project, Baobab Studios builds on the experiences that came before, forging new inroads within virtual reality. As Darnell points out, not many people are working in VR right now, so it's not like there's a rule book to follow for making good VR experiences.

Interestingly, most of the Baobab staff came from the film world, including Darnell (DreamWorks) and Cutler (Pixar, DreamWorks). And even they were surprised at how different the process was. When they first looked at VR, their initial thought was that creating a VR experience would be akin to creating a film in 360. "We were so wrong," says Cutler.

But, they figured things out and have produced some amazing work in this genre since, tackling new challenges and continuing to build their VR tool set.

"Someone once asked me what it was like to give up the control that I used to have as a director on flat-screen projects. I told them I don't feel as if I gave up control. I gave up the tool kit I had for making cinema, but gained the tool kit for working in VR," says Darnell.

He continues: "The problem is, I'm not sure what's in that tool kit yet. I have to dig down into this vast, dark cavern and find the tools because I don't know what they are or what they can do. Rather, we're slowly adding those tools, and each time we add them, everything gets a little easier. That, combined with the work the team has been doing under Larry's guidance to build out a pipeline for efficiently producing VR content, [are things] we didn't have when we started. At that time, everything we did was trial and error."

Despite the pandemic that affected most of the production of Baba Yaga and continues to plague studios, the award-winning Baobab has continued to push forward, releasing yet another production ( Paper Birds with Latin America animation studio 3DAR) this past December, with another, Namoo, set to debut at Sundance. In effect, this group is becoming somewhat legends themselves in the world of VR films.