In a perfect world, effects artists compositing 3D imagery into live-action footage would receive all the data they needed to create a convincing scene. After each shot was carefully planned, the set would be surveyed, and when each shot was finished, the camera motion would be recorded. That data would then be turned over to the 3D artists so they could precisely replicate in their software's camera the location and movement of the live camera as it was shooting the footage. Such prep work, artists agree, would simplify and expedite the compositing process, eliminating the need to laboriously hand-track 3D elements in live footage frame by frame.
Of course, this isn't a perfect world. More often than not, little-if any-camera and set information is recorded and passed on to effects departments. This is where 3D matchmoving software comes into play.
Designed for use in 3D effects sequences in which the real camera is moving rather than locked in a fixed position, 3D matchmoving programs all work in much the same way. The software tracks the location of objects in a video or film clip, and mathematically determines the camera's movement based on the path derived from the tracking process. Artists then export that camera data to their 3D modeling and animation package, where they apply it to the software's camera. By reconstructing 3D camera-motion paths from live-action footage, effects artists can create realistic composites in which their 3D elements appear to be part of the original scene.
 |
| Double Negative used the matchmoving program Boujou to place CG smoke (below) into a shot from the movie Enemy at the Gates. The lack of detail and fixed features (see water and far-off land mass at upper left) to serve as reference points presented a tra |
According to many users, matchmoving tools simplify the content creation process, especially in cases where set and camera data hasn't been compiled.
This situation can occur for numerous reasons. Sometimes, depending on the location, it's impossible to take measurements of the set; one example would be a fast-paced shoot of a chase scene through the Australian outback.
At other times, the project's director might think that such planning interferes with creativity on the set. According to Pierre Raymond, president of Hybride Technologies (Quebec), that was the case with director Robert Rodriguez, with whom Hybride worked on the films The Faculty and Spy Kids. "Robert is the sort of person who doesn't like to feel trapped in a situation where technical obligations start to jeopardize spontaneity," says Raymond. "And he isn't alone. More and more directors feel this way today."
 |
| Using RealViz's MatchMover, animators at Giant Killer Robots tracked the movements of a bovine bartender's animatronic head, then applied the movements to a replacement created in Softimage XSI for the movie Monkey Bone. (Image courtesy Giant Killer Robot |
But the usual reason there is no set and camera data is that effects departments are handed visual effects shots that the director has added after the live shoot is complete. "Because CG technology has come so far, directors don't feel it's a big deal to add effects shots after the fact," says Jim Gorman, an animator at Pixel Magic (Burbank, CA).
"It's much easier to hand off the sequence to the effects department and say, 'OK, this is all we have. Make it perfect,'" adds Thad Beier, partner at Hammerhead Productions (Studio City, CA). "Asking the artists to fix it in post is a lot less expensive than re-shooting something because you changed your mind about the way your shot should look."
Luckily for effects artists, several 3D matchmoving programs are now commercially available that enable users to recover real camera movement so that they can incorporate 3D objects into scenes in a convincing manner.
Of the nine products covered on these pages, six are standalone applications. The two that have been around the longest are rastrack from Hammerhead Productions and 3D-Equalizer from Science-D-Visions.
Formed by ex-PDI employees in 1995, Hammerhead developed rastrack because there were no other matchmoving tools on the market at the time. "We started the company to do effects work for movies," recalls Beier, who won a Technical Achievement Award from the Academy of Motion Picture Arts and Sciences for rastrack in 1999. "We developed ras_track to help us in our work, and when other artists heard of it, we started selling it in 1996."
With ras_track, the process begins with the tracking phase, in which the user marks witness points on the first frame in the sequence. As with most matchmoving applications on the market, the user should choose 8 to 10 points on various static objects such as buildings, street signs, or parked cars. The points should be well distributed so that some are closer to the camera than others, and they should be spread across the frame rather than clustered in the middle.
 |
| For the film Dracula 2000, Computer Café used SynaMatch to matchmove a long aerial shot that the facility later had to convert to an airplane crash scene containing digital elements modeled, animated, and composited in NewTek's LightWave. |
Once a witness point is selected, it is tracked. If there is enough contrast between the witness point and the rest of the scene, rastrack's correlation-based feature tracks the sequence automatically. Otherwise, the user must follow each point manually. When all the witness points are tracked, the user enters whatever information he or she knows about the 3D positions of the points, and the system calculates an approximate 3D camera track. The user then tweaks the 3D track until it's consistent with its 2D counterpart. If no 3D positions are known, the user must provide a reasonable estimate of the 3D positions and the system will calculate an approximate camera track. Like some of its competitors, rastrack is capable of both camera and ob-ject tracking, so users can calculate how an object onto which a CG element has to be composited must move with respect to the moving camera.
According to Beier, rastrack is less automated than competing programs and is thus targeted at more experienced users. "We feel it offers more control than the other programs," he says. "It's like the difference between manual and automatic transmission. Manual is harder to use and fewer people use it, but it gives you a closer connection to the machine." Rastrack runs on SGI and Linux workstations and costs $1500.
Like rastrack, Science-D-Visions' 3D-Equalizer began shipping in 1996. But 3D-Equalizer is more automated than rastrack in that the resulting 3D track requires no user tweaking. Also, 3D-Equalizer differs from rastrack and other products in that the user must place points on two frames rather than one. "There must be at least six tracking points that are common in these two frames, and the camera position of each frame must be as different as possible," says Rolf Schneider, a company co-owner. "We do it this way because we feel the reconstructed camera path and the reconstructed 3D point model are more precise when you use two frames."
 |
| Virtual grandstands appear in a racing scene from the movie Driven with help from animators at Pixel Magic using MatchMover. (Images courtesy Pixel Magic.) |
According to Schneider, Science-D-Visions is working on a new release of 3D-Equalizer that will enable users to matchmove not just static points, but also individually moving points. "For instance, this would let you track points on an actor's face as the actor is speaking and the camera is moving, which is something you'd have to do by hand otherwise," he says.
This feature would also provide users with low-cost motion capture. "It would be good for users who don't need a high-end motion-capture system because they don't have a lot of motion capture shots to do on a regular basis," Schneider says. "We think this is the next logical step in matchmoving. According to Schneider, the new version will ship by the end of the year. The software runs on SGI, Windows NT/2000, and Linux workstations and sells for $10,000.
Another company that's been around for a while is SynaPix, which was formed in 1996 and began shipping its matchmoving application, SynaMatch, in early 2000. The software is now in Version 1.7, but SynaPix plans to release Version 2.0 shortly.
According to Marc Cajolet, director of product management, a highlight of SynaMatch is its ability to handle long, complex camera motions. "We've solved up into the thousands of frames, and we've worked with helicopter shots, and crane and boom shots, where the camera twists and turns radically," he says. Also, like rastrack, the software can do object and camera tracking.
 |
| The Mamoe program has a Render View (left), which shows the scene from a calculated camera position, and a Perspective View (middle), which displays the scene, including the calculated camera, from an imaginary camera position. The final result (right) in |
With Version 2.0, SynaMatch will include a feature called geometric constructs, which Cajolet says will vastly improve the recoverability of scene data. "With this feature our software will know, for instance, that the line on the sidewalk, the manhole cover, and the piece of trash that you used as tracking points are actually the ground plane. This will increase the quality and guarantee better registration because we're ensuring mathematically that those three points are co-planar." The Irix/Windows NT software sells for $7500.
One of the newest 3D matchmoving products on the market is 2d3's Boujou, which began shipping at the end of March. Boujou is different from all 3D matchmoving products shipping today in that it's entirely automatic.
After a user imports a film sequence into Boujou, the software analyzes it, then presents the user with the steps needed to process it. Essential steps such as importing the sequence or tracking the features are marked with three diamonds, while more optional ones such as providing known camera data are marked with fewer diamonds. Next, the software identifies image features to track by placing red crosses on them. Boujou identifies a large number of the qualifying features initially, then discards any that produce anomalous results.
As it works through the sequence, Boujou builds tracks from the features it identifies from frame to frame, all the while analyzing the track information and discarding tracks that are inconsistent with the overall calibration. When the sequence is finished, the software indicates the features that produced consistent tracks as 3D points in each frame. The user can then export the finished information for the sequence to a 3D animation package. At Siggraph, the company announced Boujou 1.3, which adds new tools for handling lens distortion, a batch-processing feature for hands-off matchmoving, new scene geometry capabilities, and support for Flame, Inferno, and Houdini export formats. The software, which runs on Windows NT/2000, sells for $10,000.
According to Chris Steele, CEO, Boujou can't track every sequence a director might shoot. For instance, it has difficulty tracking shots with a high degree of motion blur. "But we can do 80 percent of the shots that typically require 3D matchmoving," he says.
But if Boujou can track 80 percent of shots that are trackable, what's a user to do for a sequence that falls into the 20 percent that Boujou can't track? Not to worry; Imagica has just begun shipping Mamoe, a matchmoving application that can track points manually or automatically. And RealViz is incorporating a similar capability into the next version of its MatchMover software.
"With this capability, you'll be able to import a sequence and let the software track it automatically so that you don't have to manually select your tracking points," says Emmanuel Javal, co-founder of RealViz and president of the company's US division. "But if the automatic tracking doesn't provide you with a good track, you can go back and manually select your points. You'll be able to go automatic when you think it will work, or just try it if you're not sure if it will work and decide whether you want to fine-tune it manually."
The first two versions of Mamoe were available only in Asia. With Version 3, Imagica is making Mamoe available to US customers as well. In addition to automatically finding and extracting feature points that can be used as markers for tracking, or letting users place the tracking markers manually, Version 3 also features an expanded image cache that temporarily stores the images around the tracking markers in RAM. This, says product manager Yasushi Mishima, results in a much smoother operation, especially for long and high-resolution sequences.
Furthermore, the software boasts a repositionable sweet spot. "In the previous version of Mamoe, the sweet spot, which is the most important interest point, could only be located at the center of the tracking template," explains Mishima. "In the latest version, you can position the point anywhere you like in the tracking template." This is useful, he says, when the tracking point moves on to the edge of the picture frame.
Also new in Version 3 is a multi-path recalculation function, which ensures that the system calculates camera position and target data separately rather than at the same time. For example, if a tracking marker exits a frame during the sequence, the missing marker will cause a calculation error in the camera target data. With Mamoe, users can employ the software's Curve Editor to smooth the camera position path, then execute the second calculation with the camera position data locked, explains Mishima. The second path will maintain the smoothed camera position data and recalculate only the camera target data, resulting in a smooth camera path match to the CGI and the live footage. Running on SGI and NT 4.0 systems, Mamoe sells for $4000.
 |
| Phoenix Tools' 3D Camera Tracking program features a flexible interface that allows for tracking points to be selected automatically or manually. (Image courtesy Phoenix Tools.) |
MatchMover, available for the past 18 months and now in Version 1.7, is a member of RealViz's Image Processing Factory (IPF) family of image-based content creation tools. According to RealViz, one feature that sets MatchMover apart is its ability to process all external and internal camera parameters, including zoom. At Siggraph in August, the company launched MatchMover Version 2.0, which featured automatic tracking.
According to Javal, Version 2.0 also features a new 2D tracking capability that enables the software to track patterns that are rotating as well as zooming. MatchMover runs on NT and Irix platforms and sells for $2600 to $5000, based on export capability. Version 2.0 also supports Macintosh platforms.
While a majority of 3D matchmoving applications today are standalone products, three are plug-ins integrated with specific programs. The benefit of this kind of packaging, say the vendors, is that users don't have to leave their main 3D application to perform a matchmove, and they don't have to learn a whole new interface.
Designed for use with Discreet's 3ds max is Autonomous Effects' SceneGenie, which started shipping in 1999. "Our company's charter is in effects and robotics, or using robots to solve real-world problems," says Monnett Soldo, president. "SceneGenie evolved from our robotics work."
SceneGenie contains more than 50 different utilities for integrating effects into shots. Users can match lights, shadows, brightness, and colors. In addition to live shots, the software can analyze rendered images, adjusting them to match the original image, and can track moving objects. Soldo says the company will probably port SceneGenie to other applications in addition to 3ds max.
Alias|Wavefront's Maya Live, which also handles object and camera tracking, works in much the same way as SceneGenie, but from within Alias's Maya Unlimited, which sells for $16,000. First, the software identifies reference points that can be tracked throughout the sequence. Then it analyzes the live-action footage frame by frame to track the position of the reference points. Once a full set of 2D tracks has been calculated, the camera solver duplicates or extracts the camera motion from the original sequence. Maya Live then re-creates a virtual camera with the same position and orientation as the original physical camera.
According to Shai Hinitz, Maya Imaging product manager, Maya Live is best used by those who need to matchmove only occasionally. "Maya Live works well. But if all you're doing every day is matchmoving shots, a 3D package like Maya Unlimited is more than you need," he says. "You want to get something that's made specifically for matchmoving and that's targeted just for that workflow. All of the standalone matchmoving products out there can export to Maya."
Although all the vendors covered thus far offer tools that assure a match between 3D animated models and live-action footage, one of the most innovative is Phoenix Tools, which is working on technology to enable real-time 3D matchmoving.
At present, the company offers a 3D matchmoving plug-in called 3D Camera Tracking. Designed for use with Softimage 3D, 3D Camera Tracking is a utility within Origami, the company's set of plug-ins for extracting or reconstructing 3D information from a set of photos or a motion sequence.
Like Mamoe and MatchMover 2.0, 3D Camera Tracking automatically tracks key points in a sequence, or the user can choose the tracking points manually. Moreover, the software can estimate the distortion parameters of the real camera and automatically create and attach a shader inside Softimage to simulate the same distortion at rendering. "This means that the composited elements will be distorted in the same way as the real sequence," says Riccardo Savarè, Phoenix Tools' R&D director. The software also can automatically locate and track key patterns in a sequence. According to Savarè, the company is in the process of developing a complete, standalone product, code-named Icarus, which will combine the Origami plug-ins with Phoenix Tools' motion vector analysis-based TimeMaster software. Origami costs $2000 for a permanent license; users can rent the software for $500 per year.
In addition to its work on Origami and Icarus, Phoenix Tools is involved in a real-time matchmoving project in Europe for film and video applications. Although specific details were unavailable at press time, Savarè did say the technology would enable a director to look through a viewfinder or at a video assist and see what is being filmed, together with a low-resolution representation of the 3D objects that will be composited into the scene. At the same time, data pertaining to the real camera's movement will be fed to the 3D animation software in real time, thereby keeping the virtual objects anchored to the live scene.
"With this solution, a director could direct the actors better so that they really look like they're interacting with a CG character because he would see them composited with the CG as he's shooting the footage," Savarè says. Phoenix Tools' real-time technology is in the prototype stage at the moment, and Savarè doesn't expect that it will be ready to incorporate into a product for at least another year. But when it is ready, he plans to include it with Icarus.
According to Hinitz at Alias|Wavefront, other companies are working on similar real-time matchmoving solutions that will eventually become commercially available. However, due to confidentiality agreements, Hinitz was unable to divulge additional details. Nevertheless, he says it points to the direction in which this market is headed. "Our job is ultimately to bring the creative decisions into the hands of the creative people-the directors and cinematographers," he says. "The next generation of 3D matchmoving products will do that.
"Matchmoving has evolved a lot in a short period of time, and real-time matchmoving and in-camera effects do exist today," he concludes. "It's just a matter of time before the technology is incorporated into commercial products."
Audrey Doyle is a freelance writer based in Boston. She can be reached at audreyd@mediaone.net.
2d3 * www.2d3.com
Alias|Wavefront * www.aliaswavefront.com
Autonomous Effects * www.afx.com
Hammerhead Productions * www.hammerhead.com
Imagica * www.mamoe.com
Phoenix Tools * www.phoenixtools.com
RealVIZ * www.realviz.com
Science.D.Visions * www.sci-d-vis.com
SynaPix * www.synapix.com