The challenge of making classic entertainment products work on current technology
Entertainment products are more "timeless" than most software products. Whereas few would really want to use Microsoft Word 1.0, even if they could get it to work, millions of adults and children still watch Pixar's Monsters, Inc. and Finding Nemo, and read classic interactive storybooks released a decade or more ago. But making such classic entertainment products work well for today's markets and users can be as challenging, if not more so, than making Word 1.0 work on Windows 8.
Just as computer technology evolves and introduces new features that make revising software necessary, technological innovations - such as the introduction of stereoscopic 3D movies for the mass-market - created the need to release revised versions of Pixar movies in a new form and format suitable for stereoscopic projection. That required reviving the software and systems that created the original Pixar movies to re-render each frame of the movie as two frames stereoscopically aligned.
Reviving the Pixar rendering software (RenderMan) wasn't actually necessary - it continues to be used to produce Pixar's latest movies as well as special effects for innumerable major motion pictures. But getting it to correctly render each frame was a challenge because not all the information necessary to produce frames from two new viewpoints was available. This made the creation of the stereoscopic versions of Monsters, Inc. and Finding Nemo (originally released in 2001 and 2003, respectively) impossible without overcoming the software rot that had occurred in the meantime.
Converting Monsters, Inc. for 3d required reviving previous technology to generate stereo renders.
This effort, though costly, pales in comparison with the original effort of creating the movie, since all the animation, graphics, sound, and music are unchanged. Only the resulting product format (that is, the "3D") has changed.
Similarly, the array of mobile touch-screen tablets and phones provided an opportunity to revive some of the "timeless" Wanderful Living Books as fully functional, interactive children's books for the iPad and iPhone, and Android mobile tablets and phones. (These were classic interactive storybooks originally published by Broderbund Software in the 1990s for Mac and Windows computers.) As with Pixar's movies, the effort, though costly, pales in comparison with the original work of creating these products, since once again, all the animation, graphics, sound, and music are unchanged. Only the product platforms are different.
The term "digital alchemy" has been adopted to apply to efforts such as these. Digital alchemy requires archaeology - locating and resurrecting software/code and digital assets (images, animations, sound files, music files, and so forth) that have long been lost or archived but not accessed. It also requires the invention of new software and processes to "morph" the old products into new ones suitable for distribution and release in today's markets.
On the technical level, when looking at some of these successful digital alchemy projects, lessons emerge regarding best practices for software architecture. There is a common theme of data-driven software that makes such projects possible. This is the story of what it took to make these projects happen, and the challenges and technical difficulties that had to be overcome to resurrect the systems, code, and digital assets in order to revive and make them better than the original products.
Defining the Project
As with all good projects, it is important to make sure that there is a clear set of "requirements" before embarking on the project. For Pixar's 3D movie endeavors, this meant avoiding the temptation to add new scenes or revisit scenes that could be artistically enhanced. (Each movie would contain exactly the same frames - no more, no less). It also meant being open to reworking a scene to make it "better" in 3D. Because the sound track would not be modified, any reworking of scenes had to be limited to making the imagery work better stereoscopically.
For Wanderful's new Living Books project, this meant making decisions about what else, besides having the interactive storybooks working on iOS and Android platforms, would have to be added to make the products viable in today's markets. This meant dynamically resized graphics to fit the screen resolution for any new phone or tablet; improved interaction and touch/gesture support; dynamic language changing capabilities; dynamic page selection support via scrolling page thumbnails; and new options pages that provided valuable, new information regarding the apps as well as hiding obsolete information presented in the original options pages.
The Pixar 3D Projects
Figure 1: Pixar followed this 3d data-driven architecture block diagram to render stereoscopic images.
Pixar 3D projects require two new frames be rendered from two new viewpoints for each frame in the original movie. Therefore, the Pixar team needed to begin the production process by re-generating scene descriptions from the animation system (MENV) output, rather than just re-rendering the existing scene descriptions from two new viewpoints, as shown in Figure 1 (on this page).
It was necessary to return to the original scene descriptions in order to adjust certain scenes for more comfortable or dramatic presentation in stereoscopic 3D. Surprisingly, the depth-of-field settings for the scenes were attenuated, since the depth of field was intrinsic to the stereoscopic projections, whereas in the original movies, it was implicit in the images. There were certain movies where, because of the 3D effects, scenes had to be adjusted in order to "look right."
The Wanderful Interactive Storybooks Project
Figure 2: Wanderful's data-driven system shows the interactive component.
The Wanderful project had a different complication, since the products are narrative (read to me mode) and interactive (let me play mode). Because the interactivity is closely tied to the platform the product runs on, considerable work went into creating a new interactive layer that sits above the Action Interpreter that derives its actions from input by the user, as shown in Figure 2 (on this page). This system must also render the sound and music interactively, something that happens outside the Pixar system during a postproduction phase.
Creating a new generation of the Living Books provided an opportunity to improve them. There were two major areas where distinct improvements were made for all the products: interaction/responsiveness and more convenient page navigation. Both required some major modifications to the way the books operated.
Pixar Technical Difficulties
Pixar's movies are huge in scope, resulting from the collaboration of hundreds of people, spread among a large number of teams and disciplines, and working under significant time pressures. Because they chose to re-render these movies from the original assets to assure the highest quality, they needed to reconstitute a significant amount of that production pipeline - but with a small team and a limited amount of time.
Not having access to a lot of the primary leads who worked on the original movies (they were in high demand on active projects) meant they had to recover a lot of knowledge from digital archaeology - documentation, wikis, and database notes.
Even when the Pixar 3D team could simply recompile software, they often needed to debug problems caused by changes during production that were unfortunately not archived properly.
One such example was the fact that the Pixar 3D team could not reasonably render its films using old hardware, so it had to port old versions of the software onto modern architectures and operating systems. The total amount of code is vast: millions of lines. And again, the 3D team does not often have access to the principal software engineers of even the major systems, much less the minor glue that holds a lot of the production pipeline together.
A further complication was that Pixar had migrated from Sun SPARC to Intel-based Dell computer systems after many of the original movies had been completed. Problems arose from the differences in the byte ordering of the SPARC versus the Intel processors (little endian vs. big endian).
Daniel McCoy, one of the technical leads on the Pixar 3D project, says, "One of the algorithms used in the software for generating pseudo-random numbers turned out to have a deeply obscure sensitivity to the endian-ness of the underlying architecture. It was necessary to build in some byte-swapping deep into the software in order to generate pseudo-random number sequences approximately matching the ones used for the original film to comparably produce some of the algorithmically computed textures."
And then there are the difficulties presented by stereo, says Mark VandeWettering, one of the technical leads for the Pixar work. This affects camera setup, reflections and refractions (which were often faked on Pixar's older films), focus, special effects, and dissolves.
Wanderful Technical Difficulties
Wanderful's technical challenges were huge. None of the carefully archived source code was able to be located or provided by HMH, so Glenn Axworthy (the original Broderbund software architect of the Living Books engine and, later, lead programmer for Wanderful) spent more than a year examining the data and programs from the packaged consumer CD-ROMs. Eventually, he developed a new Living Books playback engine from his knowledge of how the original engine worked and insights gained from running the existing programs. And all this was done without access to the original source code.
This new engine also established an insulated environment for the original data to play back in, logically separated from modern OS platforms and making cross-platform development of the products possible. Furthermore, where the original engine was single-threaded and event-driven, modern platforms require separate threads for graphics, audio output, and user interaction. Creating this new playback engine as heavily multi-threaded often created problems.
Matt Siegel, the lead iOS and Mac programmer, recalls that "subtle timing-sensitive problems showed up during testing - and they tended to appear occasionally and only in real-time interactive playback. Tracking them down was challenging and often took a lot of patience and time to do so.
One of the other difficulties was devising scripts to properly decode two different data compression algorithms used for the graphics: Lempel-Ziv sliding window compression and a customized run-length encoding for animation frames.
"Data compression was very important in the 1990s to speed up loading from slow 100kb/sec CD-ROM drives and to conserve scarce memory, which totaled only 2mb to 4mb for both the operating system and application," says Siegel. "So the Living Books products relied heavily on keeping animation frames run-length encoded in memory and drawing them directly in that format - it was one of Living Books' original technical breakthroughs."
Siegel continues: "It reduced the graphics size and sped up drawing both by a factor of about 3x, enabling sufficiently fast playback of hand-drawn animation without constantly stopping to load new graphics from the CD-ROM. It was critical to get this done efficiently and correctly. Additionally, the new platforms are now powerful enough to not only decode the data and display the data, but also scale it dynamically using bi-cubic interpolation to fit the optimal resolution for each device the apps run on."
Additionally, lots of tweaking had to be done to get the new engine to play back the animations correctly. Objects on each storybook page have action scripts set by the original animators. Says Siegel, "The abilities and interpretation of these scripts were extended and refined over several years throughout the production of the original storybooks. To re-create the storybooks' exact behavior, the new engine needed fine adjustments to get all the script capabilities right, and to negotiate conflicts between different interpretations."
One issue in Finding Nemo that also arose in other movie scenes illustrates how a stereoscopic film has more of a need to have "everything look correct."
Keeping Things the Same
Like the Pixar projects, all the storybook animation remained essentially the same. But for Wanderful, it was even more important to make sure none of the animation had to be modified. The original scene production was done in Macromedia Director, an animation tool long since discontinued.
Creating a new animated Wanderful title screen to play back in the engine required resurrecting some of the original 20-year-old production software. "Rob Bell (one of the other original Living Books programmers) and I spent days pulling obsolete Mac computers from storage and cobbling them together into an archaic configuration we finally got to work, patching scraps of software from eBay and ancient CD-ROMs into a new production pipeline," Siegel recalls. "Bootstrapping a functioning system that could run the software and access the Internet for data transfer was a test of our determination."
Other difficulties arose from the desire to maintain one core code base for all platforms. This included the ancillary assets that were created to enhance the information provided with each storybook application. A decision was made to create these ancillary assets in HTML/JavaScript and display them using an embedded browser that appears to be part of the application.
A New Beginning
Despite these challenges and technical difficulties, both Pixar and Wanderful have demonstrated that producing new, vibrant products based on products created and released a decade or more before are possible and practical if motivated, talented teams are assembled for this purpose and the software they must revive is suitable to being updated.
In fact, the new products they produced have earned awards and acclaim for bringing new life to classic entertainment. Moreover, they have proven that digital alchemy can be realized through the box-office success of Pixar's 3D films and sales of Wanderful's popular applications.
Author's Note: This article could not have been written without the help and insight from the Pixar 3D team, especially Mark VandeWettering and Daniel McCoy, and Wanderful's programming team: Matt Siegel, Glenn Axworthy, and Rob Bell. Special thanks also goes to Pixar's Joshua Hollander and Chris Wiggum, who helped get the conversation at Pixar started and monitored the communications effectively.