ABOUT FACE

Almost overnight, it seemed that nearly every professional athlete in practically every conceivable sport had a video game endorsement. Gamers were looking for real action by real athletes—or something close to that. With the first generation of consoles, the digital athletes in the games looked generic and basic; in the second generation, they actually resembled their real-life counterparts in their looks and motions. With the latest consoles, the digital players can be considered virtual replicas; not only do they look just like their real-life counterparts, including the piercing blue color of New York Yankees shortstop Derek Jeter’s eyes and the cleft in the Houston Astros pitcher Andy Pettite’s chin, but they move just like them, too, down to the unique shuffle-like gait of Boston Red Sox left-fielder Manny Ramirez.

While the new generation of consoles offer the necessary hardware to power such impressive imagery, the real challenge lies in the hands of the game developers, or more specifically, their artists, who have to be at the top of their own game in order to deliver the level of realism demanded by today’s gamers. Two top-level companies in the sports-game genre—Sony Computer Entertainment America and 2K Sports—are meeting that challenge through an unprecedented joint effort. Both companies, which are fierce competitors in the market, entered into an agreement with the Major League Baseball Players Association (MLBPA), the union governing MLB players, to purchase digital scan data of every player on every team in Major League Baseball. With the data, the developers would create digital characters that looked amazingly like their real-life alter egos—a quantum leap over what can be accomplished by modeling the characters by hand without the scans.

A few years ago, the association attempted a similar endeavor, but was thrown a curveball by the end results, which only made it through one season of play (see “Double Headers,” August 2002). “This time the MLBPA knew what it wanted and was guided by the gaming companies, and set strict parameters for the quality of the models and the actual process—how much time the company could take per player, things like that,” says Nick Tesi, vice president of operations at Eyetronics, the scanning company that the association eventually signed for this project.

Winning the contract, however, only got Eyetronics to first base; the company also had to agree to “create” its own scanning space if necessary (Eyetronics brought a tent to each location). Third, the company had to disperse personnel and equipment to all the locales throughout the country where spring training was held, and acquire the scans during so-called picture day, a designated time when players were made available for league-sanctioned photo shoots for baseball cards, mugs, bobbleheads, and other objects. The home run finally came this past December when Eyetronics delivered the last of the processed data to the game developers.

Although this was the first time that Eyetronics performed scans for the MLBPA, the technicians are hardly considered rookies in their field. In the past, the company has scanned numerous actors and actresses for film projects such as The League of Extraordinary Gentlemen, Charlie and the Chocolate Factory, The Poseidon Adventure, The Guardian, The Fountain, and more. In the sports arena, the company has also scanned various athletes for game developers, “but nothing on this level,” says Tesi.

During the eight designated picture days in the spring of 2006, Eyetronics positioned five groups of technicians in the field—two in Arizona for the so-called Cactus League and three in Florida for the so-called Grapefruit League. This allowed Eyetronics to hit all 30 spring training locations in the limited time period. “They had to move from stadium to stadium and keep all the sensitive equipment functioning at peak performance,” says Tesi.

Even though the equipment Eyetronics uses is mobile, the items were bound to get bumped and sometimes broken with so much jostling around at such a frenetic pace. To ensure that there would be no downtime, each team consisted of a technical hardware expert, a scanning technician, and a pinch hitter to help move the acquired data from the scanning system to a computer and even a backup drive. “We just had one chance to get the data; we couldn’t risk losing it,” says Tesi. “So we backed up the information excessively. At night we even made copies of the copies.”

With so many professional photographers and their equipment set up in small, improvised areas (some of Eyetronics’ scanning was done in the shower room), the scenes could get a little chaotic. Still, the picture takers, Eyetronics included, had to remain flexible; required to set up early in the morning, the photographers were at the scheduling mercy of the players, who decided whether they wanted to take to the field for some practice beforehand.

At every 45 degrees, a series of 12 images are acquired; most are for reconstructing the geometry, while one is for the texture map. Because the texture map is acquired at the same time as the other data, the texture will automatically fit onto the geometry, eliminating the cumbersome work of fitting the texture onto the model. Proprietary software from Eyetronics then determines the x, y, z positioning of the projected polygonal grid and calculates the geometry of the model.

“Eyetronics first and foremost is a software-development company. We build our own hardware technology around the software, so we are constructing an entire system for acquiring the scanned data,” Tesi points out. As a result, the software has its own internal format, which is then translated into all the standard software formats used by digital artists.

Using this equipment, the Eyetronics team digitally scanned hundreds of MLB players from the neck up, capturing head and facial information, including hair. According to Tesi, the quality of the scan actually increased with such a large volume of subjects because the technicians randomly tested the data between scans, examining the lights, grids, and data, and then tweaking them as needed.

One of the biggest challenges Eyetronics faced, aside from the aforementioned time and space constraints, was dealing with the 1400-plus star athletes who sported a wide variety of personalities. “Some were neutral to the whole thing—get me in and get me out,” says Tesi. “Some were fighting it, but some were really interested in how it all worked.”

On a technical level, the heavier players (over 250 pounds) threw the system off-balance; the technicians learned to counter this by placing a foot on the platform to stabilize it. Some of the longer hairstyles also presented a challenge, since the system was not calibrated to acquire data outside a certain region. But whatever information the crew couldn’t capture with their cameras was “reinterpreted” by Eyetronics technicians off-site when they cleaned up the data before presenting it to the game developers. “Our technicians are former modelers and sculptors, so they can work with the data and re-project maps onto the scanned information if needed,” explains Tesi.

While the actual scanning process took just 90 seconds per person, the cleanup—accomplished in two phases—took much longer, an average of five hours per head. First is the hour to two-hour initial pre-processing phase, during which time the original data is converted from a 2D picture to a raw 3D model. The second phase is more time-consuming (about three hours) and entails putting the actual model together, retouching the information to make sure there are no holes in the geometry, applying the textures, and then prepping the model for delivery.

During spring training, the Eyetronics team used its retooled Rotoscanner to capture
the 3D geometry and texture of players’ heads and faces. 


 

Even though Sony and 2K Sports compete directly against each other in the video game sports genre, both saw this as a wise investment, enabling the companies to deliver titles featuring far more realistic players.

As a result, Tesi anticipates a surge in the number of game developers who will draft their characters using scanned data. “If they are paying an actor or a professional athlete for his or her likeness in a game, they definitely want the character to look dead-on,” he points out. “Subtleties make huge differences, particularly with the new game consoles, which are now using the same models for the cut-scenes and for the interactive play.”

Because Sony and 2K Sports were creating the player models for their respective new video games using the same scanned data, both knew right off the bat that their starting lineups would look very similar. So the developers had to come up with a game plan that would differentiate each title, especially since both companies compete in the same sports-game arena.

Recently, Sony stepped onto the field with MLB 07: The Show, a continuation of its popular hardball simulation series in which gamers can pick their own teams using any MLB player or even a player from the Minors. Or, they can create their own rookie and run that person’s career from beginning to end. When ready to play ball, they can select an opponent or have one selected automatically. “Gamers can mix and match players from various teams, edit the rosters, and trade players,” says lead artist Paul Hainey.

Sony’s MLB franchise is the longest running officially licensed baseball video game for the PlayStation console, and it is lauded as one of the most authentic baseball games available in terms of AI and gameplay. This season, in addition to extending those features, the developer has created a new starting lineup using the scanned data from Eyetronics and the processing power of the Sony PlayStation 3 for an experience and aesthetic that is more realistic than ever.

“We added a lot of new features to the title and expanded on the career mode, but the big difference is in the visual quality, particularly on the PlayStation 3,” says Hainey. “The player likenesses and quality of the models have been greatly improved with the scans—even the hair is more realistic, which contributes to the overall look.”

Hardball Hardware

As Hainey points out, video games have taken a big step forward in terms of quality, thanks to the current generation of consoles, such as the Xbox 360 and the PS3, which can handle much larger data sets contained in the video game software. (PS2 and PlayStation Portable versions of the game were released in late February; the PS3 title debuts next month.) “The new consoles have opened a lot of doors for us in terms of the techniques and processes we can use to create high-quality imagery,” says Hainey. “For the PS3 version, we tried to make the most of the hardware with the system’s graphics card and the Cell Processor.”

Without question, the PS3 is the most complex console on the market. The Cell Processor, developed jointly over a five-year period by Sony, Toshiba, and IBM, is a multi-core chip capable of multi-threaded, massive floating-point calculation. Meanwhile, the Sony/Nvidia RSX (Reality Synthesizer) graphics chip is based on the G70 architecture, the same family as the Nvidia GeForce 7800 series GPU. When combined with the Cell Processor, the chip provides 2 teraflops of floating-point horsepower.

“The RSX is the graphics guts of the machine that enables us to increase the visual quality of the imagery and take the look to the next level,” says Hainey. “It gets us closer to the ultimate goal of making the imagery and the players look photoreal.”

Utilizing this PS3 graphics power, the artists were able to increase the file sizes of their player and stadium models in MLB 07, which are between five and 10 times the size of those in the PS2 version of the game. According to Hainey, the poly counts for the PS3 models are over 30,000 for a single character, while the faces alone contain 6000 polygons. “Also, our textures have gotten much bigger to support the HD resolution of the PS3,” he says. Yet, while HD offers artists a chance to showcase their high-quality imagery, the format is extremely unforgiving; the image geometry has to be present, so artists can no longer use 2D graphic cheats when creating the finished models.

Using the scanned data gave the modelers a significant lead in their game. Sony fed the geometry and textures from the scans into a new pipeline it had set up at the beginning of the project that allowed the group to efficiently process the rough game-asset files. Then, using proprietary tools that Sony had built within Autodesk’s Maya, the artists shaped the head asset so that it exactly matched the scan geometry. “The scans are about 450,000 triangles, which is a little high for us [to use as a game asset],” says Hainey. “We could put one of those on the screen, but we have a lot of other models and textures that have to go into a scene.”

Next, the group used its proprietary tool set inside Maya to re­sample the texture provided by Eyetronics, making it smaller for the interactive game environment while maintaining the same aspect ratio in the layout. Because the UVs were already laid out on the in-game head, the artist could basically drop the texture right onto the model with just minimum tweaks and cleanup afterward. “The work we did earlier last year in setting up the tools really paid off. We only had to do a little color correction and cleanup on the seams and get rid of a few artifacts,” says Hainey.

Some of the players, such as the New York Yankees’ Johnny Damon, have unique, signature hairstyles that had to be included on the models. The scanning process captured that information for athletes with shorter hair, which accounted for about 90 percent of the scanned players; for the rest, that information had to be added to the 3D model. Sony elected to do this in-house, initially investigating the use of a hair-simulation tool, but later deciding to tackle the work by hand for this release, mainly because of performance issues and time constraints.

Later, the artists incorporated the finished head models onto generic bodies, then used a proprietary tool to scale the body to accurately match the player’s proportions. “Our artists have all the players’ stats, including their height and weight, as well as game footage and pictures to use as references,” says Hainey. The players were then fitted with the appropriate uniforms and equipment, created with Maya, Pixologic’s Zbrush, and Adobe’s Photoshop. An in-house cloth-simulation solution, coupled with Maya Cloth and Syflex’s cloth solution, allowed the clothing to move realistically on the model.

To make the player models themselves move realistically, Sony applied motion-captured data to both the bodies and faces of the characters. All told, the team acquired more than 10,000 different movements using a Vicon MX40 camera system during various sessions at Sony’s in-house mocap facility. Later, the crew applied the data to the models within Autodesk’s MotionBuilder, which was used to create all the game animation.

“The characters perform the appropriate motions while making a play in the game, and this gives the overall impression that this is a real game and that’s what an actual player would do in the same situation,” says Hainey. “We wanted it to look as if the game player were watching the action on television.” For some of the superstars, such as Boston Red Sox slugger David Ortiz, Sony was able to apply authentic mocap data of the individual, taken from the developer’s extensive motion-capture archives.

 

According to Brinkman, 2K Sports had a daunting task when creating this title: Not only did the group have to go for the extra bases in generating the imagery for the powerful new consoles, it had to create a title that could go the distance on all the major gaming platforms. This meant optimizing the graphics in each title to take advantage of the specific technologies of the different consoles, with the goal of creating a roster of the most realistic-looking athletes and building the most authentic-looking stadiums. “That’s what fans of our games want, and that’s what we’re going to deliver,” says Brinkman.

 The group is using approximately 750 player scans in its Major League Baseball 2K7 title. Although each season the developer unveils a new release of the game, this is the first time that 2K Sports has used the digital scans to create its models. But, it was not the developer’s first time at bat with scanning. 2K Sports, along with five other sports-game companies, attempted a similar endeavor a few years ago by pooling their money (see “Double Headers,” August 2002), “but it didn’t work out well,” he says. “The scanning technology just wasn’t there at the time. The cameras weren’t that high enough in quality, and the consoles weren’t very advanced [compared to the Xbox 360 and PS3].” At the time, the scans ended up being used by just a handful of developers, but only for a season or two until the data had to be retired because of the low quality.

That was the last time anyone had attempted to scan the players for Major League Baseball. What enticed MLB and the developers to take another swing were the capabilities of the Xbox 360 and PS3, coupled with the advances in the scanning technology. And with this current roster, “there was no way we could have made our player faces look this realistic without the scanning technology,” says Brinkman.

Brinkman notes that the Eyetronics information saved the art department a tremendous amount of time—artists were able to create a character model in 2.5 hours with the scans, compared to half a day when doing it manually. And, the results were far better. “Traditionally, we have several ‘likeness’ artists who do nothing but create player likenesses, and those people spend a lot of time doing just faces,” he points out.

However, 2K Sports, like Sony, cannot use the scan data directly in its title because the models—on average 300,000 triangles in size—are too large for a video game’s interactive environment. “But having the scans helps speed up the modeling process; it gets us to a level where we can work fairly easily,” says Brinkman. “The artist just has some QA work, to make sure that the data maps properly to the lower res model.”

Face Front

Eyetronics provided 2K Sports with the high-resolution scanned files in OBJ and TGA format, along with 11 high-resolution reference photos of the players—the latter having been a special request by 2K Sports. The scans aren’t always perfect, Brinkman explains, but having the photos allows the artists to fill in missing information if the need arises.

After the artists received those raw assets, they examined the files for issues, such as bad textures, blends, and seams. According to Brinkman, the team looks for distorted or corrupted geometry, which occurs if the player was chewing gum, for instance, at the time of the scan. Next, they checked the reference photos, particularly for the lighting, to make sure it was even, and the poses, to ensure they appeared natural. “The scans are not done in a studio environment; sometimes they are done inside a tent,” he adds. “That creates a challenging environment for scanning and can result in some quality issues. Also, some teams and players were not familiar with the process, which led to a few inconsistencies in the data. But when the players end up on our virtual fields, they have to look consistent.”

After examining the assets, the 2K Sports group began the feature-matching process, whereby the artists loaded their game models, created in Autodesk’s Maya, and matched the facial features of the relatively low-resolution in-game models to the high-resolution scans. After that 15-minute process, the team then used a proprietary tool to transfer UV data from the scan model to the game model, and created a tangent-space normal map. This surface-transfer tangent-space normal map contains the detail of the high-resolution scanned model for lighting calculations in the shader.

“The new generation of games is all about shaders,” Brinkman says. “You aren’t looking at polygons and poly counts anymore. The video game industry had been obsessed over that for the past few years, but now it’s all about shaders. We can do so much with them. 

“The systems are more efficient in the way they utilize data. We have a lot of accessories this year. We have a production assistant and artists who are dedicated to upgrading those items to lifelike versions,” Brinkman notes. “That alone is going to stand out and make the game more authentic-looking.”

In addition, the new consoles are better able to process HDRI lighting, making the textures look realistic. “The visual quality jumps out at you, and the imagery looks like something from a TV broadcast,” says Brinkman. “That’s the look we are after in our sim game.”

Major-League Modeling

Once the group completed the facial modeling, it spent approximately two hours per character cleaning up problem areas by hand. According to Brinkman, ears sometimes don’t capture well if the person’s hair gets in the way during the scan.

To animate the models, the artists used a combination of motion-capture technology and Maya. The mocap information was acquired using a Vicon system that is set up in-house at 2K Sports; in all, the group captured more than 3000 moves for this title alone during a one-year period. “We’re in the mocap studio a full month every four to five months [to acquire various motions for the ongoing collection]; when we are developing a game, we do nothing but capture animations over and over again,” says Brinkman.          

When the players finally take the field in the game, they look at home in their surroundings. That’s because the artists spend a lot of time accurately re-creating the various ballparks. “Baseball is a unique sport in which the stadiums and playing fields vary in size—one stadium may have 400-foot outfield walls and another 330-foot walls,” Brinkman says. “Those differences will affect the play. So we put a lot of effort into modeling the locations. It’s important.” To build the stadiums, the group dispatched photographers to each locale. Using between 4000 and 5000 images for reference, the group spent an average of one month constructing each venue inside Maya.

A Brand-New Game

According to Brinkman, every year when 2K Sports releases a game like Major League Baseball 2K7, there are some fans who post Web messages complaining that a certain game character does not look like its real-life counterpart for one reason or another. “With access to the scan data and the cooperation of the MLB and the players themselves, this becomes a much smaller issue,” says Brinkman. “You have never seen technology and quality like this in a video game.”

So, what are the limits to the technology? None, says Brinkman. “Everything had gone better than we had planned,” he recalls. “We thought it would take about the same time as our manual methods and we would just reap the benefit in the quality. But in the end, we got both speed and quality.” And, 2K Sports is getting more than a double play out of the deal—the developer is using the scans of 450 to 500 superstars in its upcoming title called The Bigs, an over-the-top lifestyle game that shows the players in a more heroic light. The more distant future looks bright, too: Brinkman expects that because of the high quality of the scan data, the developer will be able to use the scans for at least five to six years.