Though computers have only been around for approximately 75 years, the rate of change for computer technology has been dramatic. Computer software and programs rapidly stop working as computer systems are discontinued and operating systems upgraded, often in incompatible ways. A by-product of these inevitable changes is that applications and software systems, even the most popular ones, just stop working. Programmers have a name for this phenomenon: software rot.
However, the fact that many leading applications and software systems have not been maintained over the years and are no longer working or available does not mean they have no value. In fact, certain entertainment products are “timeless” and will maintain their impressive value for the foreseeable future. This article examines some notable projects that have overcome “software rot” and continue to deliver entertainment value to millions – performing digital alchemy by literally taking lead “bits and bytes” and turning them into gold “bits and bytes.”
The goal was to revive some of the computer-based interactive Living Books storybooks from the 1990s for today’s iPad and iPhone, and Android mobile tablets and phones. After all, the animation, graphics, sound, and music are unchanged; only the product platforms would be different.
The Digital Evolution
The term “digital alchemy” has been adopted to apply to efforts such as these. Digital alchemy requires archaeology – locating and resurrecting software/code and digital assets (images, animations, sound files, music files, and so forth) that have long been lost or archived but not accessed. It also requires the invention of new software and processes to “morph” the old products into new ones suitable for distribution and release in today’s markets.
The motivation for this type of endeavor – resurrecting software/code and digital assets – required considerable investment, time, energy, and technicalresources. Yet, in the case of Pixar and Wanderful, the motivation was not purely financial.
Mark VandeWettering, one of the technical leads for the Pixar work, explains. “From the beginning at Pixar, its movies have been crafted with rounded models acting in three dimensions. With the latest innovations in stereoscopic displays, it was now possible to reveal this beauty – which was always in our classic, beloved titles but lay dormant and hidden because we lacked the technology to bring it directly to the audience,” he explains. “Pixar's Catalog 3D projects attempt to bring all that is great about our titles to the audience in a new and fresh way, to create the most amazing versions our already great films."
One of the strong drivers behind the Wanderful Living Books projects was more poignant. These offerings were especially popular with autistic children, and those who worked on them believed no one else was filling the void with similar products. Some e-mails, such as this, helped spur the development: “My daughter is autistic. In 1997, she received the ‘Sheila Rae the Brave’ Living Book for her birthday. At first when we played it, she put her hand on my mine (on the mouse) as I moved through the game. Gradually she started to use the mouse herself. She focused all her attention on the screen, clicking on the story word for word, over and over again. Then I noticed that she had started reading other things, like traffic signs (‘stop for school buses loading and unloading children’) and other signs (‘drive your engine clean’). The world had been opened up to her.”
It’s these types of motivations behind making “great products” that often fuels great efforts beyond the simple drive for additional corporate revenue.
Planning the Future
Through clever design and implementation, the necessary additions to both the assets from Pixar and the interactivity of the book engine could be made without modifying any of the original digital assets.
An attribute common to both these projects is that the software systems were based on a data-driven architecture, which, by definition, means “one that performs computations in an order dictated by data dependencies.”
Since both the projects deal with entertainment that is narrative in nature, it is not surprising that their architecture is similar and well suited to data-centric projects.
In both projects, the actual scene creation, which generates the scene description, with graphics and animation of objects, was completed in a “production stage” where teams of animators and artists created the rendered scenes. Although the tools they used were quite different from each other, they nonetheless required an incredible amount of labor-intensive effort to produce the resulting products.
For the projects discussed here, all that work, plus the sound editing and music production, were completed years ago by the original teams on the original projects. That is what sets these projects apart – the “digital alchemy” of leveraging all the work completed years ago and making new, improved versions of the products with a fraction of the original labor involved. The task required taking the existing assets and producing new products targeted at different formats and platforms.
Pixar’s Challenges
Pixar had to deal with a number of issues, which are detailed in the CGW magazine article. Another issue pertained to stereo.
One example in Finding Nemo that also arose in other movie scenes illustrates how a stereoscopic film has more of a need to have “everything look correct” – much more so even than in the original movies. Figure 1a shows a frame of Marlin and Crush looking at each other. Figure 1b shows an interactive display of the same frame with yellow eye-line guides.
Figure 1a
Figure 1b
From the point of view of the camera, Marlin and Crush appear to be looking at each other. However, a top view of the scene, shown in Figure 1c, shows they clearly are not looking at each other, and these problematic frames had to be fixed in the 3D version of Finding Nemo.
Figure 1c
The Pixar team carefully reviewed and fixed all the scenes in each movie for issues such those in Figures 1a 1b, and 1c. Most of the films worked fine, but some scenes required special care and attention. The various 3D Pixar movies are filled with such examples that needed to be fixed for the 3D version. There were other issues that had to be addressed, and those appear in the Technical Difficulties section later in this article.
The process of creating a photorealistic or stylized rendering, such as that produced by RenderMan for even a single frame of a movie, typically involves thousands of shaders, textures, shadows, and displacement maps that provide a scene’s detail. All these components are rolled up into one “renderer” box.
Wanderful’s Challenges
Improving the interaction and responsiveness of the Living Books required some major modifications to the way the traditional Living Books operated. These changes were made in the Action Interpreter module, based on actions by the user and fielded by the User Interaction module of Figure 2. These fundamental changes made page introductions and animated sequences interruptible, and were vital to making the products work well on touch-screen devices. No changes to the user interface or appearance of the App were made, though settings were added that allowed the parent or teacher to modify the interactivity to customize the mode of operation so it is appropriate for the child using the device. Figure 3 shows the final appearance of the App.
Figure 2
Figure 3
Adding more convenient page navigation and easier language selection required a whole new layer of interactivity to the applications, which upped the complexity level. It provided for pop-up navigation page thumbnails that gave immediate random access to any page of the storybook, and for products where two or more languages are available, pop-up language selection buttons that immediately changed the language for the storybook. These capabilities are accessed by touching the upper right blue triangle to invoke the page navigation scroller or the upper left blue triangle to invoke the dynamic language selection buttons as shown in Figure 3. The appearance of these new capabilities is shown in Figures 4a and 4b.
Figure 4a – The new page navigation scrolling thumbnails
Figure 4b – The new dynamic language selection buttons
A single page in any of the interactive storybooks ups the complexity level. The graphics may be 2D, but the animations must be controlled interactively and synchronized with the audio sounds and music. Additionally, the animations and sounds must be interruptible and able to be repeated rapidly if the user choses to interact frenetically with the system. All this complexity and the many components that implement these capabilities are rolled up into the User Interactivity and Action Interpreter boxes.
The Challenges
Pixar and Wanderful each had a number of challenges in their endeavors, including maintaining artistic integrity, obtaining rights to the property, assessing the available assets, and, of course, handling the technical difficulties at hand.
For Pixar, the greatest challenge was to create versions of the studio’s films that take nothing away from their original greatness. The films are the result of hundreds of talented artists, technical directors, and engineers working simultaneously and under tight deadlines. “In every shot, there are hundreds of decisions. What was intended? What was delivered? What was important, what was not? We can never achieve absolute, pixel-for-pixel matches for shots, but we needed to make sure everything that was significant had been addressed, and we introduced only changes that never detracted, but only enhanced, the audience experience,” says VandeWettering.
For Wanderful, the challenges were initially to secure rights to the original storybook assets (graphics, animation, sound, and music in the original software products) from the company that now owns the rights to all Broderbund’s products – Houghton Mifflin Harcourt (HMH) – before formally starting the project.
Additionally, HMH had let the author’s rights lapse to most the children’s books upon which these products were based, so Wanderful also had to secure the digital publishing rights from those authors, as well. Unfortunately, securing those rights was almost more effort than producing the updated products. But in the end, Wanderful did secure the exclusive rights to all the storybook assets from HMH and most of the author’s publishing rights, too, and was able to begin actively working to create these new products in late 2011.
The first step in both projects was to begin examining the archives from the original projects to determine what documentation, scripts, code, and assets were readily available.
Of course, both Pixar and Broderbund carefully archived all the software and assets used to create the original products, with archival processes well defined over years of practice by both companies. However, resurrecting the archives is always more problematic than one would hope.
Pixar’s documentation, scripts, and source code are all archived in a source-control system (RCS) on Unix servers in the studio’s data center. Film assets, such as models, shaders, textures, and animation files number in the millions for the more recent movies. In fact, Pixar deploys millions of lines of code that has been developed in-house and is under continuous development, often even while film projects are active.
“Because Pixar develops so much of its own animation software, we have complete access to the versions and changes of the majority of software we use, but often this software has to be adapted to run on new architectures and operating systems, versions to be of use on Catalog 3D projects,” VandeWettering says. “This means we often rediscover old bugs and find bugs that we cannot fix [to make the content] compatible with what was originally used on the shows.”
He continues: “For software that isn't developed in-house, we often have to do significant work to make [the content] function with more modern versions of the software – unless the companies no longer exist. Of course, each movie sound track was actually archived separately and married with the film during a postproduction process, so we didn’t have to worry about that.”
Broderbund’s documentation, scripts, and source code, along with all the assets used to create the products (graphics, animations, sound, and music files), were archived onto a set of CD-ROMs and placed in Broderbund’s product archives. However, Broderbund Software was sold to the Learning Company and subsequently passed through several “owners” until finally becoming part of HMH, so many of the Broderbund archives were lost and have not been located for Wanderful’s projects. Wanderful overcame this issue by hiring some of the original Broderbund programmers to re-create certain modules (the Script Parser, User Interaction, Action Interpreter, and Renderer modules) from scratch. The assets were obtained by acquiring copies of the original Living Books products and using the files from the CD-ROMs that were sold directly to customers as the master archives of all the assets for each product.
Stereo Considerations
Mark VandeWettering gave a second example. “When we rendered Cars 3D, a fairly significant change to Mater's texture assets resulted in changes to his rust pattern relative to how he appeared in about 20 percent of the show that was rendered first. Similarly, textures for brick patterns in many of the buildings in Radiator Springs, as well as changes to character eye shaders, were made mid-production.”
But even after Pixar could successfully render the original scenes of any given movie appropriately, the studio still faced a huge number of challenges in making the shots in each scene work stereoscopically. Mark VandeWettering provides just a few of the issues that had to be overcome.
“Great care is taken in camera setup to ensure the best audience experience. We want the center of interest in the shot (the spot where the director would most like attention focused) to have low disparity so the audiences' eyes aren't continually converging and diverging, which introduces fatigue. The original show might use a variety of wide-angle and telephoto lenses, which require different setups. However, there is a delicate balance. Increasing the separation between virtual cameras can make objects appear more rounded and separate them in depth, but it can also lead to ‘miniaturization,’ where the scene begins to look small.”
Also, because reflections are view dependent, conflicts can arise where highlights appear strongly in one eye but not the other. On Pixar’s older films, reflections and refractions were often faked (raytracing is a relatively modern feature in Pixar films), thus requiring special care to place them in depth. Another issue pertains to focus. Incidental objects that may be out of focus and serve to frame the shot in 2D can be too far forward, and their lack of focus may be distracting unless it is modified.
Special effects, such as particles and water, are often done by modified pipelines and more often seem problematic when viewed in 3D. In particular, effects that rely heavily on compositing wizardry often require creative reworking. Moreover, many compositing fixes, where errant effects are simply painted over or copied from other locations in a frame, need to be adjusted for each eye.
Another problem is with dissolves, which need to be designed so the inner portions of the shots have appropriate stereo setups that blend easily and naturally when they are cross-dissolved. And even the credits can be difficult. They are often done outside the original production pipeline and may include “outtake” shots that were originally cribbed from shots that aren't even in the same show. Additional digital archaeology is usually required to make these 3D outtakes work.
And then there is internationalization. The original distributions may have had only a few languages, but Pixar’s films are now retargeted for a bewildering array of languages, often triggering the re-rendering of certain shots to pick up changed texture resources that are subtle, but important.
On the Wanderful front, the new engine was first developed on Windows, and specific platform support was later added for iOS, Mac OS X, Android, and Windows 7/8 hosts. Although the engine's insulated playback environment made adding platforms much easier, it interacted with the user as just a raw point-and-click experience, exactly like the original products of the 1990s. To make the updated products pleasant to use and commercially acceptable, platform-specific user interfaces, including touch and swipe gestures for tablets, had to be developed.
The complexities of managing the display of ancillary assets (to enhance the information with each storybook application) as HTML/JavaScript reduced the burden in creating the graphics and managing the layout (which was done with CSS templates), but added many months of effort to the Android and Windows projects – which probably could have been avoided if a more native platform approach would have been taken to display the ancillary assets.
Summary
The project was indeed challenging but resulted in resuscitating decades-old software and hardware to produce new offerings for a new generation.
Acknowledgements: This article could not have been written without the help and insight from members of the Pixar “3D team,” especially Mark VandeWettering and Daniel McCoy, and help from members of Wanderful’s programming team: Matt Siegel, Glenn Axworthy, and Rob Bell. Special thanks also goes to Pixar’s Joshua Hollander and Chris Wiggum, who helped get the conversation at Pixar started and monitored the communications effectively.
Mickey W. Mantle is founder and CEO of Wanderful, Inc. His insight into Pixar’s RenderMan was gained as director of 3D Graphics at Pixar when RenderMan was first productized in the late 1980s and Broderbund’s Living Books as vice president of engineering and chief technical officer during the 1990s when Broderbund created and released the innovative and successful Living Books series of products.