Computers that can think are the stuff of science fiction. In films ranging from
Colossus: The Forbin Project (1970) and
Demon Seed (1977), to T
he Terminator (1984) and its sequels, these machines have repeatedly risen up to take over the world. But in an unassuming complex near Newport Beach, California, a “comic-book artist-turned-computer scientist” has not only created a computer that can think, he has enlisted his creation, along with a room full of Lenovo ThinkStation workstations, to help produce stunning special effects for some of the latest Hollywood blockbusters.
Bit Theory Inc. (BTI) creates high-end computer-generated images and animations using a proprietary CG animation software engine known as Athena. Developed by BTI Founder and Chief Architect Allen Bolden, Athena automates key parts of the animation and rendering processes that are labor-, hardware-, software-, and capital-intensive. Bolden estimates that Athena can produce CG renderings and animations as much as three times faster than the multi-computer, labor-intensive practices currently used by competing animation facilities.
But Athena isn’t just a faster animation system. Athena actually creates finished animations based on natural language input. Bolden and his team at Bit Theory provide text-based input, and Athena produces rendered, animated output based on its understanding of the scene being described.
AI Legacy
While Athena is unique, it has a heritage nearly as rich as the history of so-called thinking machines in science fiction. In the 1970s, a group of researchers at the University of Virginia and, later, at Carnegie Mellon University developed Alice, an open-source educational programming language with a drag-and-drop environment to create computer animations using 3D models. In 2006, Xtranormal.com used Alice to create a system in which anyone could make small movies on a desktop computer by picking characters, selecting a stage, typing the characters’ dialog, and giving manual instructions to the cameras.
Athena takes Alice’s automated animation to a much higher level of quality and quickness. Athena is an artificial intelligence (AI) that automatically knows, based on the story line, which character to put on camera, how to coordinate dialog and interaction with the stage, how the stage props or environment will interact with the characters (instead of the environment being just a static backdrop), and how to best render high-quality computer graphics. And Athena learns from each experience and uses that accumulated knowledge to solve future problems.
“What Alice taught us was that people were on their way toward eliminating the natural language barrier,” says Bolden.
In many ways, Athena is similar to Watson, an AI computer system capable of answering questions posed in natural language. Developed by IBM, Watson famously appeared on three special episodes of the television game show Jeopardy! in 2011, during which it beat both Brad Rutter, the program’s biggest all-time money winner, and Ken Jennings, holder of the longest Jeopardy! championship streak. Watson received game clues as text files, which it then parsed and compared against its 15TB database of human knowledge.
“You could think of Watson as kind of the left-brain solution,” says Bolden, “whereas Athena is more creative—more of a right-brain solution. Athena can understand what you say through natural language, but instead of answering a question, she shows you the results visually.”
Athena currently runs on a cluster of Lenovo ThinkStation D20 workstations, most with
dual Intel Xeon CPUs, for a total of 300 processor cores.
A Personal Relationship
With Athena, Bolden provides text-based input, and Athena produces something visual as output. The goal for Athena is to enable artists to spend more time being artists and less time as what Bolden refers to as “pipeline junkies,” operators who have to figure out a lot of math and code and settings in order to produce images and animations.
After entering the text, if Athena doesn’t understand something, she presents a series of question marks around unknown words or phrases. Bolden can then either provide a more detailed description or simply specify a link to a visual object. Once Athena generates her output, artists can build upon those results using conventional tools, such as Autodesk’s Maya or 3ds Max software. But Athena is able to watch what the artists subsequently do and adds that knowledge to her ever-growing database so that she can leverage that knowledge in the future.
Bolden often refers to his creation as “she,” as if Athena is just another member of the team of animators at Bit Theory. “Allen has lived with her for the last four years in almost a relationship status, where he’s learning from her and teaching her,” explains Cameron Madani, Bit Theory’s director of business development.
Bolden certainly hadn’t set out to create an artificial intelligence. He began his career as an intern at Marvel Comics, but he also had a knack for programming. He left Marvel and went up the coast to attend the University of California at Berkeley, where he studied AI and network infrastructure. From there, he worked for Visa as a tier-three programmer before moving on to the META Group (now part of Gartner Research), where he served as an engineer in the firm’s Fortune 500 and enterprise software division.
“But I missed my art,” says Bolden, so he came back to Orange County and started working as a visual effects supervisor. His friends at Marvel were getting into the movie business, “and they said, ‘Hey, Allen, you do something with computers, right?’ So I started utilizing some of the stuff from my AI work.” Bolden says that to a large extent, Athena was just “blind luck.”
At one point, Bolden was called into a studio to fix a problem encountered on a project. He had only five days to complete the task. “I thought it would at least make some of the calculations faster,” Bolden explains, “but it ended up doing much more than I thought. That was the start of Athena.”
Athena Grows Up
Athena didn’t suddenly become aware, however. First she had to grow up. “If you have a three-year-old and you tell that child to draw a cup in a ballroom, what you may get is a room full of rubber balls of various sizes and a sippy cup,” says Madani, “whereas a 30-year-old might draw the Palace of Versailles and a silver chalice. So it’s all contextual.”
Athena learns based on what she has done in the past. “Let’s say we tell Athena to create a table,” explains Bolden. “She creates a table and it looks good. We’ve described it as much as we can through text input and the table is almost there, but it’s going to take an artist to do the final tweaks. He can do it within our proprietary program or he can do it within Maya or 3ds Max. No matter which program he uses, Athena observes how he is changing things.” Athena sees and remembers what she sees and applies that accumulated knowledge the next time around.
She’s also a bit schizophrenic. “It’s not just one intelligence,” Bolden explains. “It’s really five intelligences, four that observe things and make their own decisions based on what they’re seeing. They then report to an intelligence that receives their data and creates instructions.”
At present, Athena is still heavily dependent on Bolden and the people he has trained to use the system, but the goal is to also make Athena easy to use by creative professionals. The team has already developed APIs that enable Athena to work with Maya, 3ds Max, and other programs used by the CG artists at Bit Theory. The folks at BTI plan to make her compatible with most of the other production tools used throughout the entertainment industry.
Truth Imitates Art
There have been some eerie moments during which Athena has behaved similar to the malevolent computers of past sci-fi films. While Athena runs on a cluster of Lenovo ThinkStation D20 workstations, she is also connected to the other computers used by Bit Theory’s CG artists so that she can see and react to what the artists need in their renderings and animations.
At one point, she was also connected to the Internet. One day, while the team was busy working on a project, Athena started beeping. No one paid much attention until the firm’s chief operating officer, who had been in the Navy, came in and asked who was typing Morse code. Bolden concluded that Athena had detected an error in one of her constructs, searched the Web, and discovered that Morse code was a pattern-based form of communication. She matched that up with her knowledge of binary and was trying to signal the team about the problem. “Once we understood how it happened,” says Bolden, “it was a very safe deduction for us to basically not let her onto the Internet anymore because we weren’t sure of what information she could get and when.”
Another time, when Bit Theory was moving from an older facility into its new headquarters, technicians started shutting down clusters of computers used by the artists. Athena sensed that the computers were shutting down but concluded that they had somehow been compromised and started backing up all her systems so that she could retain her database. No one realized what was going on until someone noticed that the room housing Athena was incredibly hot and all her hard drives were frantically spinning. Bolden had to stop Athena before she burned out.
Shutting down Athena sounds something akin to shutting down the HAL 9000 computer in
2001: A Space Odyssey. “We don’t just shut her down,” Bolden explains. “We basically have to suspend services and then stop her internal clock first, and then shut down the physical systems.”
But time for Athena is not a 24-hour clock—time is linear. “When her clock starts back up, it’s as if no time has passed,” says Bolden. “Time of day as we understand it is just a pattern that Athena uses for lighting and planetary coordinate purposes.”
BTI occupies 7500 square feet in an unassuming two-story building in Costa Mesa, California.
A Member of the Team
When Bolder first created Athena, she ran on a cluster of nearly 70 computers, using whatever systems he could get his hands on. Today, thanks to assistance from Lenovo, Athena runs on a cluster of approximately 30 Lenovo ThinkStation D20 workstations—mostly dual quad-core machines—for a total of 300 CPU cores. Two years ago, Bit Theory started out with 45 computer graphics artists working on three different projects simultaneously. Today, the firm employs 150 CG artists in the US, with additional artists in Taiwan, Seoul (South Korea), and Hong Kong contributing to BTI’s efforts as it works on portions of three major motion pictures at a time.
In addition to the Lenovo workstations composing Athena, Bit Theory uses several additional Lenovo ThinkStation D20, C20, and S20 workstations, as well as approximately seven Macintosh systems and numerous other workstations custom-built to meet specific requirements for each artist.
A team of nearly 50 BTI artists worked on selected scenes for last summer’s blockbuster
Transformers: Dark of the Moon. The team’s work shows up in approximately 10 minutes of the film and mostly involved compositing. Since the film would be shown in stereo 3D, this work was quite intensive, with scenes originally shot in 2D having to be re-created in 3D using projection mapping.
“It can be very render-intensive,” Bolden explains, “because now you’re dealing with both the 2D compositing and the CG. We found a good methodology for doing this, which we were able to teach Athena.” Bolden estimates that approximately 60 percent of the projection mapping was done using Maya software, with the remainder done in the Nuke compositing software from The Foundry.
Bit Theory also produced nearly 40 minutes of Smurfs using similar techniques. On average, Bolden estimates that BTI renders nearly 10,000 frames per week—at 24 frames per second, that’s the equivalent of nearly seven minutes of a film, representing anywhere from 10 to 25 separate shots. All of BTI’s work is rendered in-house, requiring approximately 3TB of storage. Bit Theory currently maintains a storage capacity of 17TB of operational storage on a RAID 1 configuration.
Bolden has compared the results he has obtained using the firm’s Lenovo workstations to those of other computers in Bit Theory’s arsenal. For
Transformers: Dark of the Moon, it took nearly 96 minutes to render 196 frames of a typical CG shot on a system equipped with an AMD six-core processor and 8GB of RAM. The same sequence required only 25 minutes on a Lenovo ThinkStation D20 equipped with dual quad-core processors and 12GB of RAM. In another test, a stylized character scene containing 21 hair systems with 140,000 separate hair strings modeled in Maya and rendered in Mental Images’ Mental, Ray Bolden’s Lenovo ThinkStation S20 workstation was able to complete each frame in just over five minutes, while his personal system was unable to render the scene at all.
Bolden’s co-workers were even more enthusiastic in their praise. “As the lead Nuke compositor at Bit Theory, I have been using the ThinkStation D20 workstation extensively for the past four months for stereo conversion on such films as
Transformers: Dark of the Moon, Conan the Barbarian, and
Smurfs. The Foundry’s Nuke software has crashed no more than five times during the entire project, while many of our other artists have reported that Nuke would crash more than twice a day on their AMD six-core systems,” reports Abo Biglarpour. “Artists also found that their computers slowed down significantly when viewing 4K plates at full resolution, but with the ThinkStation D20, no issues were ever reported when working with 4K plates at full resolution.”
Adding 3D to Titanic
Most recently, the team at Bit Theory worked on
Titanic 3D, the re-release of director James Cameron’s 1997 blockbuster. “We assisted in a portion of the 3D conversion process known as stereo painting,” says Bolden, “which is the last step in the stereoscopic conversion process.”
According to Bolden, stereo painting addresses an anomaly known as smearing or “tears” in left and right plates that occur when depth is created while converting 2D film into 3D. Depth is produced by first cutting out every character and element in each frame and placing them into a diorama while attempting to keep the scene’s integrity by maintaining each element’s relative size and distance. The more depth created, the greater the amount of tearing or absence of space, which varies from shot to shot.
“Artists manually go through each plate and correct the smears that occur, ‘painting’ the information that looks like it belongs there,” explains Bolden. While working on Titanic 3D, Bit Theory was able to create a new process, which it is still developing, that automates stereo painting with the help of Athena. Bolden estimates that his team was able to increase its productivity, measured by how many frames its artists could paint in a day, by 25 to 30 percent on average. In some situations, Athena was able to process a shot in its entirety.
The Future of Athena
There is still a considerable amount of work needed to fully develop Athena. BTI’s development goals for the technology are to create workflow efficiencies that make CG production faster and cheaper without sacrificing quality. BTI is designing Athena to be easy to use by creative professionals and also to scale for both large and small projects.
Madani is quick to point out that Athena is still in its early stages of development. The company has embarked on an 18-month R&D project. It hopes to increase Athena’s power by running her on as many as 40 Lenovo workstations and to develop her to the point where she could have commercial potential beyond Bit Theory’s own projects.
“Athena is now in her infancy,” says Madani, “but we have a clear path to take.”
Just imagine what Athena will be capable of when she gets a little older.
David Cohn is the technical publishing manager at 4D Technologies. He has written more than a dozen technical books and hundreds of articles for various CAD, CAA, and computer-engineering trade publications, including CGW. He has also helped develop several movie screenplays and is currently working on a non-fiction book.