Global backend of a visual effects pipeline

Presentation at PyCon Canada 2017.

Goes over an introduction to visual effects (VFX), the backend of a VFX pipeline, the challenges we face and the operational part of maintaining the backend.

Transcript

2. Introduction Lead Core Developer at Method Studios Based in Vancouver @method_studios
3. Presentation Overview 1. Visual Effects (VFX) 2. VFX Backend 3. VFX DevOps 4. Conclusion
4. Visual Effects? “Visual effects involve the integration of live-action footage and generated imagery to create environments which look realistic, but would be dangerous, expensive, impractical, or impossible to capture on film.” source: https://en.wikipedia.org/wiki/Visual_effects
5. All images are the property of their respective owners
6. VFX The Work The Process The Departments The Applications
7. Scope VFX studios turn the director’s imagination into a reality. AVATAR SUPERMAN INCEPTION OKJA Primary VFX by Weta Digital Primary VFX by MPC Primary VFX by Double Negative Primary VFX by Method Studios
8. The Work Breaking it down. “SHOW” – The Film, Commercial or TV Show “SEQUENCE” – A single set or environment “SHOT” – A single camera movement “TASK” – A unit of work for an artist /jobs/vfx_pycon/abc/abc1000/model /(ROOT)/(SHOW)/(SEQUENCE)/(SHOT)/(TASK)
9. The Process VFX Studio Product Input VFX by Method Studios
10. Breaking it down Product Input Paint Roto
11. Departments
13. Applications NUKE HOUDINI MAYA
14. Layout Primary VFX by Mackevision from a_library import search_assets houses = search_assets(type=“model”, level_of_detail=“low”, ext=“mb”)
15. FX 250,000 NODES (think python object!) 1,000,000,000 PARTICLES Primary VFX by Method Studios
16. FX
17. Compositing VFX by Method Studios (and others) def create_write_dir(): import nuke import os import errno file = nuke.filename(nuke.thisNode()) dir = os.path.dirname(file) osdir = nuke.callbacks.filenameFilter(dir) try: os.makedirs(osdir) except OSError, e: if e.errno != errno.EEXIST: raise nuke.addBeforeRender(create_write_dir)
18. Importing & Exporting
19. VFX Backend
20. Pipeline • We need to support the dynamic workflow of artists and productions. • Common in all VFX studios • Asset system; • Render farm; • Disk storage; • Software packages.
21. Asset system • All the data produced by artists needs to be tracked and versioned • A model, a texture, a render, a digital camera. • Metadata on assets • Who created it? • Who is using it downstream in the pipeline? • What are the assets needed to generate this asset? • What version was approved by the supervisor? • Where is it on disk?
22. Working with assets
23. Dependencies
24. Render farm
25. The basics • High-performance computing (HPC) • Many computers • Clusters • Render jobs • Tasks • Requirements • Memory • Cores • Priority • Farm manager • Server
26. Example
27. Qube! ArtistView
28. Challenging… Single Shot 120h per frame 27 years total Primary VFX by Method Studios 1,500 Nodes 56,000 Cores 150 TB RAM
29. Disk storage • Network File System (NFS) • Very high I/O • 300,000 IOPS • 82 GB/s throughput • Scalable to petabytes • 21 nodes in Vancouver • 4 PB of storage • 36 TB of SSD cache • Still easy to run out of disk space • Asset tracking is really important
30. More disk • Work is shared between studios • NFS over WAN?
31. Software management in pipeline • NFS • Different shows / shots / tasks use different versions of software • Just in time (JIT) resolving of the runtime environment
32. REZ
33. Abstracting the pipeline core in services • Libraries, libraries… • Runtime dependencies • Exposes too much • Different versions at same time • Services!
34. Services • Building blocks necessary for the backend of a pipeline • Asset service • Farm service • Disk service • Network calls • HTTP • AMQP
35. Challenges • Downtime isn’t an option • Limited internet access • On premises • Multi site
36. In one studio
37. Globally
38. Not everything can be an island Sharing work between sites
39. VFX DevOps
40. Ansible • Deployment pipeline • Development -> Staging -> Production • Environment variables • Common roles • Systems metrics • Centralized logging • Service discovery
41. Ansible
42. • Service discovery • Health checks • DNS • production.farm.service.van.consul • Round robin to a healthy Vancouver farm service. • production.farm.service.consul • In case the farm service in current datacenter is down go to next. • K/V store Consul
43. Consul
44. • Elasticsearch • Logstash • Kibana • Elasticsearch cluster per datacenter • Metrics and logging ELK
45. Kibana
46. • Zabbix agents • Installed on each Virtual Machine • Collects system metrics • Send periodically to Zabbix servers • Zabbix server • Storage • Alerts • Displayed through Grafana Zabbix
47. Grafana
48. Conclusion
49. What we want? Empower the Artists. Empower the Studio. Reduce Costs Automate,
 Automate, Automate!
50. Wrapping it up • VFX uses Python! • The future • Data analysis?
51. All images are property of their respective owners
52. Thank you! Feel free to reach me at andre.prado@methodstudios.com Special thanks to Mitchell Deeming for being awesome! VFX by Method Studios (and others)