Global backend of a visual effects pipeline

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Introduction Lead Core Developer at Method Studios Based in Vancouver @method_studios

Slide 3

Slide 3 text

Presentation Overview 1. Visual Effects (VFX) 2. VFX Backend 3. VFX DevOps 4. Conclusion

Slide 4

Slide 4 text

Visual Effects? “Visual effects involve the integration of live-action footage and generated imagery to create environments which look realistic, but would be dangerous, expensive, impractical, or impossible to capture on film.” source: https://en.wikipedia.org/wiki/Visual_effects

Slide 5

Slide 5 text

All images are the property of their respective owners

Slide 6

Slide 6 text

VFX The Work The Process The Departments The Applications

Slide 7

Slide 7 text

Scope VFX studios turn the director’s imagination into a reality. AVATAR SUPERMAN INCEPTION OKJA Primary VFX by Weta Digital Primary VFX by MPC Primary VFX by Double Negative Primary VFX by Method Studios

Slide 8

Slide 8 text

The Work Breaking it down. “SHOW” – The Film, Commercial or TV Show “SEQUENCE” – A single set or environment “SHOT” – A single camera movement “TASK” – A unit of work for an artist /jobs/vfx_pycon/abc/abc1000/model /(ROOT)/(SHOW)/(SEQUENCE)/(SHOT)/(TASK)

Slide 9

Slide 9 text

The Process VFX Studio Product Input VFX by Method Studios

Slide 10

Slide 10 text

Breaking it down Product Input Paint Roto

Slide 11

Slide 11 text

Departments

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Applications NUKE HOUDINI MAYA

Slide 14

Slide 14 text

Layout Primary VFX by Mackevision from a_library import search_assets houses = search_assets(type=“model”, level_of_detail=“low”, ext=“mb”)

Slide 15

Slide 15 text

FX 250,000 NODES (think python object!) 1,000,000,000 PARTICLES Primary VFX by Method Studios

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Compositing VFX by Method Studios (and others) def create_write_dir(): import nuke import os import errno file = nuke.filename(nuke.thisNode()) dir = os.path.dirname(file) osdir = nuke.callbacks.filenameFilter(dir) try: os.makedirs(osdir) except OSError, e: if e.errno != errno.EEXIST: raise nuke.addBeforeRender(create_write_dir)

Slide 18

Slide 18 text

Importing & Exporting

Slide 19

Slide 19 text

VFX Backend

Slide 20

Slide 20 text

Pipeline • We need to support the dynamic workflow of artists and productions. • Common in all VFX studios • Asset system; • Render farm; • Disk storage; • Software packages.

Slide 21

Slide 21 text

Asset system • All the data produced by artists needs to be tracked and versioned • A model, a texture, a render, a digital camera. • Metadata on assets • Who created it? • Who is using it downstream in the pipeline? • What are the assets needed to generate this asset? • What version was approved by the supervisor? • Where is it on disk?

Slide 22

Slide 22 text

Working with assets

Slide 23

Slide 23 text

Dependencies

Slide 24

Slide 24 text

Render farm

Slide 25

Slide 25 text

The basics • High-performance computing (HPC) • Many computers • Clusters • Render jobs • Tasks • Requirements • Memory • Cores • Priority • Farm manager • Server

Slide 26

Slide 26 text

Example

Slide 27

Slide 27 text

Qube! ArtistView

Slide 28

Slide 28 text

Challenging… Single Shot 120h per frame 27 years total Primary VFX by Method Studios 1,500 Nodes 56,000 Cores 150 TB RAM

Slide 29

Slide 29 text

Disk storage • Network File System (NFS) • Very high I/O • 300,000 IOPS • 82 GB/s throughput • Scalable to petabytes • 21 nodes in Vancouver • 4 PB of storage • 36 TB of SSD cache • Still easy to run out of disk space • Asset tracking is really important

Slide 30

Slide 30 text

More disk • Work is shared between studios • NFS over WAN?

Slide 31

Slide 31 text

Software management in pipeline • NFS • Different shows / shots / tasks use different versions of software • Just in time (JIT) resolving of the runtime environment

Slide 32

Slide 32 text

REZ

Slide 33

Slide 33 text

Abstracting the pipeline core in services • Libraries, libraries… • Runtime dependencies • Exposes too much • Different versions at same time • Services!

Slide 34

Slide 34 text

Services • Building blocks necessary for the backend of a pipeline • Asset service • Farm service • Disk service • Network calls • HTTP • AMQP

Slide 35

Slide 35 text

Challenges • Downtime isn’t an option • Limited internet access • On premises • Multi site

Slide 36

Slide 36 text

In one studio

Slide 37

Slide 37 text

Globally

Slide 38

Slide 38 text

Not everything can be an island Sharing work between sites

Slide 39

Slide 39 text

VFX DevOps

Slide 40

Slide 40 text

Ansible • Deployment pipeline • Development -> Staging -> Production • Environment variables • Common roles • Systems metrics • Centralized logging • Service discovery

Slide 41

Slide 41 text

Ansible

Slide 42

Slide 42 text

• Service discovery • Health checks • DNS • production.farm.service.van.consul • Round robin to a healthy Vancouver farm service. • production.farm.service.consul • In case the farm service in current datacenter is down go to next. • K/V store Consul

Slide 43

Slide 43 text

Consul

Slide 44

Slide 44 text

• Elasticsearch • Logstash • Kibana • Elasticsearch cluster per datacenter • Metrics and logging ELK

Slide 45

Slide 45 text

Kibana

Slide 46

Slide 46 text

• Zabbix agents • Installed on each Virtual Machine • Collects system metrics • Send periodically to Zabbix servers • Zabbix server • Storage • Alerts • Displayed through Grafana Zabbix

Slide 47

Slide 47 text

Grafana

Slide 48

Slide 48 text

Conclusion

Slide 49

Slide 49 text

What we want? Empower the Artists. Empower the Studio. Reduce Costs Automate,  Automate, Automate!

Slide 50

Slide 50 text

Wrapping it up • VFX uses Python! • The future • Data analysis?

Slide 51

Slide 51 text

All images are property of their respective owners

Slide 52

Slide 52 text

Thank you! Feel free to reach me at [email protected] Special thanks to Mitchell Deeming for being awesome! VFX by Method Studios (and others)