Building a Render Cloud

Slide 1

Slide 1 text

Building a Render Cloud Eliang @ PyCon Taiwan 2013

Slide 2

Slide 2 text

About Me 梁長宏 (Eliang)

Slide 3

Slide 3 text

Template: 1 Name: Amy Reed Photo: User Render

Slide 4

Slide 4 text

Tools We Used Non-Python ● Adobe After Effects ● FFmpeg ● RabbitMQ ● MySQL Python-related ● Django ● boto ● pika ● Pillow / PIL ● lxml ● ...

Slide 5

Slide 5 text

Requirements Scalable Parallel One render task is split into many subtasks, executed parallelly

Slide 6

Slide 6 text

Render Cloud Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 powered by Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2

Slide 7

Slide 7 text

Render Cloud Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 powered by Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Request

Slide 8

Slide 8 text

Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Queue Consumers Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2

Slide 19

Slide 19 text

Consumer Thread class ConsumerThread(threading.Thread): def __init__(self, queue): self.stop = False self.queue = queue def run(self): while not self.stop: msg = self.queue.get() process(msg)

Slide 20

Slide 20 text

Queueing System Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2

Slide 21

Slide 21 text

Queueing System powered by import json import pika # connect to RabbitMQ params = pika.URLParameters( 'amqp://guest:guest@localhost:5672/vhost') conn = pika.BlockingConnection(params) channel = conn.channel() # create a queue named 'WorkerQ' channel.queue_declare(queue='WorkerQ', durable=True)

Slide 22

Slide 22 text

Putting a Message msg = json.dumps({ 'type': 'render', 'texts': { 'name': 'Amy Reed', 'email': '[email protected]' }, ... }) channel.basic_publish( exchange='', routing_key='WorkerQ', body=msg, properties=pika.BasicProperties(delivery_mode=2) )

Slide 23

Slide 23 text

Getting a Message _, _, body = channel.basic_get(queue='WorkerQ') msg = json.loads(body) if msg.get('type') == 'render': do_render(msg)

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Replacing Text and Photo AEPX File (After Effects' XML format) AMY REED ....

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Automated Rendering in After Effects

Slide 29

Slide 29 text

Calling aerender in Python import subprocess subprocess.Popen([ 'aerender.exe', '-project', 'project.aepx', '-output', 'frmae.[#####].jpg', '-s', '100', # start frame '-e', '150', # end frame ... ])

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Merging Images import subprocess subprocess.Popen([ 'ffmpeg', '-i', '/path/frame.%05d.jpg', '-i', '/path/music.mp3', '-s', '720x480', '-vcodec', 'libx264', '-acodec', 'libaac', ..., '/path/output.mp4' ])

Slide 33

Slide 33 text

There's More ● Scaling ● Distributed Locking ● Estimating Render Time ● Render Watch

Slide 34

Slide 34 text

Scaling UP from boto.ec2.connection import EC2Connection conn = EC2Connection( 'access_key_id', 'secret_access_key') reservations = conn.get_all_instances() # find the first stopped EC2 instance and start it for res in reservations: for inst in res.instances: if inst.state == 'stopped': inst.start()

Slide 35

Slide 35 text

Scaling DOWN # Windows-only shutdown subprocess.call([ 'shutdown', '/f', # by force '/s', # shutdown '/t', '0' # after 0 seconds ])

Slide 36

Slide 36 text

There's More ● Scaling ● Distributed Locking ● Estimating Render Time ● Render Watch

Slide 37

Slide 37 text

Distributed Locking Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Need a lock here!

Slide 38

Slide 38 text

Distributed Locking - Lock def lock(lock_id, lock_timeout): lock_path = os.path.join('Z:\\locks', lock_id) while True: try: os.mkdir(lock_path) except OSError as err: if err.errno not in (errno.EEXIST, errno.EACCES): raise err try: time_locked = os.path.getctime(lock_path) if time.time() - time_locked < lock_timeout: time.sleep(1) # wait for it to be unlocked else: os.rmdir(lock_path) # lock expired except OSError: pass

Slide 39

Slide 39 text

def unlock(lock_id): lock_path = os.path.join('Z:\\locks', lock_id) try: os.rmdir(lock_path) except OSError as err: pass Distributed Locking - Unlock

Slide 40

Slide 40 text

There's More ● Scaling ● Distributed Locking ● Estimating Render Time ● Render Watch

Slide 41

Slide 41 text

Estimating Render Time num_workers render_time 2 100 6 x 8 50 Predict the future by reviewing the past: 6 - 2 x - 100 --------- = ------------ 8 - 2 50 - 100 => x = 66.7

Slide 42

Slide 42 text

There's More ● Scaling ● Distributed Locking ● Estimating Render Time ● Render Watch

Slide 43

Slide 43 text

Render Watch

Slide 44

Slide 44 text

Summary ● After Effects render cloud on AWS ● Worker queue ● Scalable and parallel ● Python and Django Thanks! Q & A