At PipelineFX, Qube users across the world are starting to move in this direction, which means we have been getting more questions about it – namely, how do I get started and how can I make this easier on myself?
Here are 9 tips that will help blend render farm management with GCP today.
1. Pick Your “Zone”
Knowing your zone can help you control how fast your data is transferred to and from the cloud, how much you pay for transfers, which equipment may be available to you, and even your environment’s stability.
Your data transfers more quickly when it’s closer to you just like driving your car from one spot to another. Since all cloud providers offer their resources by geographic region, you should start by looking at the region you are in or that is closest to you. Within Google regions, they offer zones, and you can select zones within a region that may be physically closer to you. The closer you can get, the faster your data will transfer and the less it will cost to complete those transfers.
And if you have to transfer data from one cloud resource to another, know that transferring between zones in a region is cheaper than going across regions.
Learn about the zones available to you as it can affect the hardware and services available to you. Certain zones have Ivy Bridge or Haswell processors where others don’t. In addition, zones may have different quotas on the available services like firewalls, IPs, image storage, etc.. Last but not least, for maximum uptime, Google highly recommends having instances spread out to several zones in the same region because zone maintenance and outages should be expected.
2. Pick Your Machine Type
Different machine types can offer a varying amount of processors or RAM which may render different applications more or less efficiently. It’s best to do a set of test renders of your typical work on several different types to find the sweet spot in terms of price and performance.
For scale, look at Instance Groups. You can define a template for your instance image, so adding more processors to your standard render farm image is as simple as increasing the number of instances for that template. You can even set the instances to scale up automatically as you need additional resources. Instance Groups automatically load balance between the resources in the group, and if an instance crashes, it will automatically reboot and pick up where it left off.
3. Leverage Pre-emptable Instances
If you are scheduling render jobs that can afford the possibility of being paused or stopped for a period of time then Preemptible VM instances are a great bargain, running at a much lower price. By using these instances, you can save, on average, between 3050% of the hourly cost for rendering instances. You can save quite a bit of money across one or more projects, but with some added risk since there is no guarantee that the instance will be not be preempted.
4. Preview Jobs First
Optimize your use of cloud nodes. If you are going to be paying to use off site resources, then make sure the jobs are correct before you start the meter.
If you are using our render management solution, Qube!, you can leverage our Preview Frames feature to automatically split a job into two parts: the first test renders a sample set of frames immediately and the second is the remaining frames are dispatched already blocked, waiting on user approval to continue. The Preview Frames are normally rendered locally with the remaining frames destined for the Cloud.
This way, you don’t start the meter on throw away work, and as you get information about the completed preview frames (render times, memory footprint, etc…), you can better determine how many cloud instances will be required to finish the project on time.
5. Don’t Keep the Meter Running
A best practice in minimizing cloud costs is to pause or stop the instances when they are not in use. It can mean the difference between a large monthly bill versus the kind of cost you were expecting.
When rendering, it’s best to automate. For example, Qube! offers the use of pre- and post-flight scripts that can be run at the start and end of every job. Write a script that starts your instance to
run every single time a new job is dispatched to a cloud resource, and do the same to shut them
down or pause them when your job completes. Your instances will only be up for rendering, and
you will not be paying for cloud resources that are sitting idle.
6. Let the Network Expert Get You Connected
Connecting your network to the Cloud requires some technical expertise that’s best handled by network engineers or system administrators. It’s not that it’s especially difficult, it’s just that it takes some time and testing. It’s getting better now that Google has added a more “out of the
box” VPN product and there is a growing online knowledge base where users are sharing their experiences.
7. Cache is King
In any network environment, whether it’s on-premise or remote or a mixture of the two, the quality of experience and overall throughput depends on the strength and speed of your available cache. It makes an even bigger difference when you are using cloud nodes.
Avere’s vXFT is an extremely easy way to initiate the transfer of data in and out of the Google Cloud.
Mount the Avere cluster along your file path, and your files will start transferring without changes to your pipeline. A standard three-node cluster can serve well over 100 cloud nodes, and the proprietary algorithms insure the most used files are available as the priority on the fastest disk.
8. Bill More Effectively
Because Google pricing is always online and available, you can forecast for a particular project how much rendering capacity you may need before you get started. And since pricing of cloud node usage is so much more straightforward than all of the baked in costs associated with on-premise rendering, you can estimate the work needed and make sure the actual, appropriate cost is passed on to your client and covered in the invoice.
You can improve cost forecasting accuracy by using previous show render data to estimate the type of work needed on a new project. Costs may include the instance, storage, caching, VPN, render license, and render management license. You’ll cut down on comments like “did I estimate correctly?” or “I need to go back to the client with additional charges” or “our numbers don’t add up we must be missing some cost for rendering.”
9. Test Now
Sign up for your free Google Cloud Platform trial now. You want to test the Cloud and figure out your integration before you have a desperate need for more render power. Once tested you’ll be ready when the need arises, so don’t wait!