One of the tricky things about getting Shotgun up and running smoothly, is getting your renderfarm environment configured correctly.
The Shotgun docs offer two approaches that are worth trying here and here but as these are specific to Nuke they may not do everything you need to get all your DCCs configured correctly.
Another approach to try is something that we have implemented here at HaloVFX; using Shotgun hooks to build/set your environments from scratch. This approach was developed largely to avoid the need to use 3rd party tools or scripts to set and update environment variables across the studio and renderfarm.
There are a number of benefits to this approach
- your environment is set entirely within the Shotgun ecosystem; there is no need to launch Shotgun from a custom in-house launcher.
- you can take a new workstation, install Shotgun and your DCCs, and without any further configuration or setting of a single environment variable, you can open Shotgun.
- the studio environment can be configured per-project.
- workstations/rendernodes require no environment configuration.
- your entire environment is included in the pipeline configuration for any given project facilitating easy backup and retrieval.
- your environments are entirely isolated per-project with no risk of changes causing un-expected problems across other projects.
In principle, this approach requires three steps.
- set studio-wide environment variables inside your core engine-init.py hook
- set project specific environment variables in your app-launch.py hook.
- update your deadline submissions scripts to include all required environment variables in the job file.
Update Shotgun ‘engine-init.py’
In your project pipeline config ‘core/hooks’ folder, make a copy of the ‘engine-init.py’ file from your core ‘install/core/hooks’ folder. (whether shared or localized)
You can add project-wide environment variables here by adding entries to the EngineInit ‘execute’ method.
class EngineInit(Hook):
def execute(self, engine, **kwargs):
os.environ['NUKE_PATH'] = some_path
os.environ['NUKE_FONT_PATH'] = some_path
os.environ['NUKE_LUT_PATH'] = some_path
Update Shotgun ‘app_launch.py’
In your project pipeline config ‘hooks’ folder, make a copy of the ‘app_launch.py’ file from your ‘config/hooks’ folder.
To help identify environment variables we want to keep track of, they are all prefixed with “HV_”. We will use this later in the deadline submission script to choose which environment variables to pass on to our rendernodes.
It’s up to you want variables you need to define for your pipeline, but here are some paths I found useful to have to use in other tools and apps.
import sys, os
def execute(self, app_path, app_args, version, **kwargs):
self.parent.log_info("\n\n######## app_launch.py #########")
system = sys.platform
self.path_sep = os.pathsep
engine = self.parent.get_setting("engine")
# Set some utility global environment vars
os.environ['HV_PROJECT'] = self.parent.tank.project_path
os.environ['HV_PROJECT_PATH'] = self.parent.tank.project_path
os.environ['HV_PROJECTCODE'] = os.path.split(self.parent.tank.project_path)[1]
You can define environment variables for specific applications by testing for the engine being called :
if engine in ["tk-houdini"]:
arnold_path = r'C:/solidangle_houdini/htoa-1.14.2_r8c78c1d_houdini-{0}'.format(version)
arnold_bin_path = r'C:/solidangle_houdini/htoa-1.14.2_r8c78c1d_houdini-{0}/scripts/bin'.format(version)
For Houdini, you’ll want to set a number of paths including the following.
os.environ['HOUDINI_PATH'] = some_path os.environ['HOUDINI_OTLSCAN_PATH'] = some_path
And if you need to add paths to your PYTHONPATH you can add the following :
os.environ['PYTHONPATH'] = some_other_path + os.pathsep + os.environ['PYTHONPATH']
Update Deadline submission scripts – submitNukeToDeadline.py
To pass your environment variables through to deadline, you need to edit the submission script and insert something like the following after the jobInfoFile has been opened and assigned to the fileHandle variable;
after :
fileHandle = open( jobInfoFile, "wb" ) fileHandle.write( EncodeAsUTF16String( "Plugin=Nuke\n" ) )
add something like this:
env_vars = ['an_environment_variable','another_environment_variable','etc'] # CREATE A LIST OF ENVIRONMENT VARIABLES YOU WANT TO PASS ON WITH YOUR JOB
env_index = 0 # WE NEED A COUNTER TO ENSURE THE ENVIRONMENT KEY VALUE KEY IS NUMBERED SEQUENTIALLY. DEADLINE DOES NOT HANDLE DUPLICATE OR MISSING NUMBERS GRACEFULLY
for env_var in env_vars: # LOOP THROUGH THE VARS AND ADD A LINE TO THE JOB FILE FOR EACH DEFINED ENVIRONMENT VARIABLE.
if os.environ.get(env_var) is not None: #CHECK THAT THE ENVIRONMENT VARIABLE HAS BEEN SET (WE DON'T WANT TO PASS 'NONE' TYPE VALUES TO DEADLINE AS THIS CAN CAUSE PROBLEMS)
fileHandle.write(
EncodeAsUTF16String("EnvironmentKeyValue{0}={1}={2!s}\n".format(env_index, env_var, os.environ.get(env_var)))
env_index+=1
else:
pass
of course, this example is specific to the deadline submission script for Nuke, but you should be able to adapt this code fairly easily for the submission scripts in other applications.
Once you have this implemented successfully, you should see in your ‘job details’ window of your Deadline monitor, the environment variables you defined. When the rendernode launches the job plugin, the environment of the rendernode will be updated with these environment variables.
It’s up to you how far you push this approach; you can theoretically recreate the entire environment of the machine/session submitting the job without having to worry about how the rendernode has been configured.
Conclusion
When implementing this approach. I half expected there to be a number of unexpected and unwanted consequences, but I’m glad to report that in production this has proven to be very robust. As an added bonus it also provided me with excellent information with which to debug problematic jobs on the farm!
Thanks for reading, and I’ll look forward to hearing your thoughts on how this approach can be further developed. If you have taken another approach to this then I’d love to hear about it!