Share and Share Alike

By Lucas Miller

Imagine the frustration of being under a tight deadline on a big project and not being able to communicate with those you are working with. The same situation can occur when digital artists from different studios or even groups within the same facility try to share shots or assets on a project. Alembic helps facilitate that communication for a smoother workflow.

Alembic is an open computer graphics interchange framework that distills complex, animated scenes into a non-procedural, application-independent set of baked geometric results. This distillation of scenes into baked geometry is exactly analogous to the distillation of lighting and rendering scenes into rendered image data.

Alembic started as a collaboration between Sony Pictures Imageworks and Industrial Light & Magic in 2009. At the time, Imageworks had just finished a refresh of its internal baked file format, but the industry was rapidly beginning to change. More than one studio was now working on a single project, so there was more of a need to share data. At the same time, ILM was just starting to update its internal geometry file format. The two companies decided to collaborate and release their code as open source in the hopes of increasing industry adoption.

Baked geometry means that the final point positions, transforms, normals, and so forth are stored to the file directly per sample. However, the recipe that helps contribute to the final calculations is most often not stored.

The baked requirement is important for a few reasons. The amount of time it takes to calculate the final asset is usually time prohibitive. Custom proprietary tools that a company does not want to share could also be part of what goes into calculating these final positions. The same functionality to create the asset is often not available in all content creation packages. Although the amount of disk space used can sometimes be greater in baked form, this form is much easier to deal with throughout modern VFX pipelines and between vendors.

Most baked geometry formats at the time were either too simple to represent the modern needs of computer graphics applications, inefficient for the vast amounts of geometry that were being created, or totally proprietary so that it was only supported in a handful of digital content creation packages. The collaboration allowed Imageworks and ILM to do a ground up rethink of what was needed and what could be improved, and to gather feedback from the open-source community.

Overcoming the Obstacles

One of the annoying constraints of many traditional baked file formats is that there is a file per asset per frame. This can become cumbersome when there are thousands of frames with exotic subframe sampling required for motion blur when rendering. That can lead to situations where you need to find all the files that are available but only read a small handful of them to get the samples you want. Alembic allows you to store all of your samples in one file. (Although, the classical approach is still possible.) Different shapes can have different sampling rates, as well.  This is often used to oversample cheaper objects to calculate like transforms, and more sparsely sample expensive things like heavy meshes with many points. 

One of the most novel ideas in Alembic compared with other file formats is how it can automatically share data written to it without needing to be specified by the user. Oftentimes with an asset, the same data indeed will be the same across multiple time samples as well as many different objects.

An example of this is a complex truck asset - four or more wheel shapes most likely will be very close to the same. They may share the same UVs and mesh topology, but have different point positions. The data doesn't even have to be of the same type to be shared; the sharing will automatically happen as long as the byte level representation is the same. This data sharing greatly reduces the file sizes compared to traditional file formats.

Another fairly unique feature of Alembic is that you only actually read data when you request it.  This makes it very fast to open the file and sparsely traverse a large hierarchy within it, reading only objects and time samples as requested. This allows you to very cheaply open the file and read just the asset's bounds at a particular frame, without needing to read anything else about the file.

Another popular example is finding out what cameras have been written to your Alembic file without needing to read any heavy geometry. Skipping over entire hierarchies is also possible, like in cases where you may not want to load certain shapes making up a character's costume.

As Alembic was mainly created as a geometry interchange format, defining a solid list of primitive types and what properties make up those primitive types was very important. Equally important was making sure additional properties could be created on those primitives to support custom workflows. Examples include extra UV sets that you may want to feed to a custom shader at render time, or special custom pipeline data, such as what version of a content creation application wrote this Alembic file.

Continuing the Progress

After the official 1.0 release of Alembic in 2011, support by many of the content creation and rendering vendors grew fairly quickly, and today support is nearly universal. Several of the popular game engines, such as Unity and Epic's Unreal, have also added support. This vendor support has helped Alembic grow beyond the initial film VFX industry in which it was first created and into other related industries, including television and games.

Since the 1.0 release, Alembic has been continuously updated and supported. Many new features and bug fixes have been added, with input from our open-source community. The Google group alembic discussion (alembic-discussion@googlegroups.com) is the primary place to ask questions. With several hundred members, it is made up of a wide variety of artists, engineers, and other professionals in the computer graphics community.

Alembic Milestones

There have been three major milestones that have added important functionality to Alembic.

In the 1.1 release, schemas for materials were added. Although interchanging materials between different packages is not yet a solved problem, there has been a lot of great work over the last few years toward this effort. Alembic's customizable data definition will be able to represent whatever the community eventually agrees on. Even without a universal solution, Alembic can still be used to store materials in a way that those applications, which support the specified target, can load it; those that don't support it can just skip it but still load everything else.

In Alembic 1.5, a new data backend named Ogawa was released. This was initially created to address issues that had been found with Alembic's read performance across multiple threads.  It actually helped all aspects of Alembic, as it led to smaller file sizes, moderately faster writes, and much faster reads.

Finally, in Alembic 1.7, layering was introduced. Layering was developed collaboratively between Blizzard Entertainment and Sony Pictures Imageworks and is a way for users to make sparse changes to an Alembic file by writing those changes into any number of other Alembic files, and then combining them all on read. This is useful for creating lightweight output from multiple departments and combining them together at render time. 

Modelers may create the initial static asset posed in a particular way. Texture painters may then need a specific set of UVs for the asset, and could create a sparse archive with just those UVs.  Animators can create just the animated points and transforms without needing to rewrite anything else. Look development can create a layer containing the materials and assignments for the asset. All these layers can be combined at render time to create the final complete performance.

Alembic has been an important component in modern visual effects pipelines, making it easier to share geometry between departments and between companies. Its read performance and scalability has eliminated a bottleneck and enabled the latest generation of tools to meet the complexity challenges of the latest animation and visual effects films. n