Building a Render Cloud

How Osube's render system was built. PyCon Taiwan 2013.

Transcript

1. Building a Render Cloud Eliang @ PyCon Taiwan 2013
2. About Me 梁長宏 (Eliang)
3. Template: 1 Name: Amy Reed Photo: User Render
4. Tools We Used Non-Python • Adobe After Effects • FFmpeg • RabbitMQ • MySQL Python-related • Django • boto • pika • Pillow / PIL • lxml • ...
5. Requirements Scalable Parallel One render task is split into many subtasks, executed parallelly
6. Render Cloud Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 powered by Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2
7. Render Cloud Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 powered by Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Request
8. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by
9. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by
10. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 After Effects Project and Asset Files Render Cloud powered by Subtasks
11. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Parallel rendering
12. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Rendered Images
13. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Rendered Images
14. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Rendered Images 'Merge' Message
15. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Merging Images
16. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Final Video 'Done' Message
17. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Final Video Download Video
18. Queue Consumers Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2
19. Consumer Thread class ConsumerThread(threading.Thread): def __init__(self, queue): self.stop = False self.queue = queue def run(self): while not self.stop: msg = self.queue.get() process(msg)
20. Queueing System Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2
21. Queueing System powered by import json import pika # connect to RabbitMQ params = pika.URLParameters( 'amqp://guest:guest@localhost:5672/vhost') conn = pika.BlockingConnection(params) channel = conn.channel() # create a queue named 'WorkerQ' channel.queue_declare(queue='WorkerQ', durable=True)
22. Putting a Message msg = json.dumps({ 'type': 'render', 'texts': { 'name': 'Amy Reed', 'email': 'amy@example.com' }, ... }) channel.basic_publish( exchange='', routing_key='WorkerQ', body=msg, properties=pika.BasicProperties(delivery_mode=2) )
23. Getting a Message _, _, body = channel.basic_get(queue='WorkerQ') msg = json.loads(body) if msg.get('type') == 'render': do_render(msg)
24. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 After Effects Project and Asset Files Preparing Assets
25. Replacing Text and Photo AEPX File (After Effects' XML format) <Layr> <string>AMY REED</string> .... <fileReference fullpath="C: \path\amy.png" ...>
26. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Rendering Parallel rendering
27. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Rendering Rendered Images
28. Automated Rendering in After Effects
29. Calling aerender in Python import subprocess subprocess.Popen([ 'aerender.exe', '-project', 'project.aepx', '-output', 'frmae.[#####].jpg', '-s', '100', # start frame '-e', '150', # end frame ... ])
30. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Merging Images
31. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Final Video 'Done' Message Merging Images
32. Merging Images import subprocess subprocess.Popen([ 'ffmpeg', '-i', '/path/frame.%05d.jpg', '-i', '/path/music.mp3', '-s', '720x480', '-vcodec', 'libx264', '-acodec', 'libaac', ..., '/path/output.mp4' ])
33. There's More • Scaling • Distributed Locking • Estimating Render Time • Render Watch
34. Scaling UP from boto.ec2.connection import EC2Connection conn = EC2Connection( 'access_key_id', 'secret_access_key') reservations = conn.get_all_instances() # find the first stopped EC2 instance and start it for res in reservations: for inst in res.instances: if inst.state == 'stopped': inst.start()
35. Scaling DOWN # Windows-only shutdown subprocess.call([ 'shutdown', '/f', # by force '/s', # shutdown '/t', '0' # after 0 seconds ])
36. There's More • Scaling • Distributed Locking • Estimating Render Time • Render Watch
37. Distributed Locking Queueing System RabbitMQ on AWS EC2 File Storage AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Need a lock here!
38. Distributed Locking - Lock def lock(lock_id, lock_timeout): lock_path = os.path.join('Z:\\locks', lock_id) while True: try: os.mkdir(lock_path) except OSError as err: if err.errno not in (errno.EEXIST, errno.EACCES): raise err try: time_locked = os.path.getctime(lock_path) if time.time() - time_locked < lock_timeout: time.sleep(1) # wait for it to be unlocked else: os.rmdir(lock_path) # lock expired except OSError: pass
39. def unlock(lock_id): lock_path = os.path.join('Z:\\locks', lock_id) try: os.rmdir(lock_path) except OSError as err: pass Distributed Locking - Unlock
40. There's More • Scaling • Distributed Locking • Estimating Render Time • Render Watch
41. Estimating Render Time num_workers render_time 2 100 6 x 8 50 Predict the future by reviewing the past: 6 - 2 x - 100 --------- = ------------ 8 - 2 50 - 100 => x = 66.7
42. There's More • Scaling • Distributed Locking • Estimating Render Time • Render Watch
43. Render Watch
44. Summary • After Effects render cloud on AWS • Worker queue • Scalable and parallel • Python and Django Thanks! Q & A