Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Render Cloud

Building a Render Cloud

How Osube's render system was built. PyCon Taiwan 2013.

267c8397349187890a5184eaab90278a?s=128

Chang-Hung Liang

May 26, 2013
Tweet

Transcript

  1. Building a Render Cloud Eliang @ PyCon Taiwan 2013

  2. About Me 梁長宏 (Eliang)

  3. Template: 1 Name: Amy Reed Photo: User Render

  4. Tools We Used Non-Python • Adobe After Effects • FFmpeg

    • RabbitMQ • MySQL Python-related • Django • boto • pika • Pillow / PIL • lxml • ...
  5. Requirements Scalable Parallel One render task is split into many

    subtasks, executed parallelly
  6. Render Cloud Queueing System RabbitMQ on AWS EC2 File Storage

    AWS S3 powered by Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2
  7. Render Cloud Queueing System RabbitMQ on AWS EC2 File Storage

    AWS S3 Render Worker 1 AWS EC2 powered by Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Request
  8. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by
  9. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by
  10. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 After Effects Project and Asset Files Render Cloud powered by Subtasks
  11. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Parallel rendering
  12. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Rendered Images
  13. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Rendered Images
  14. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Rendered Images 'Merge' Message
  15. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Merging Images
  16. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Final Video 'Done' Message
  17. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Render Cloud powered by Final Video Download Video
  18. Queue Consumers Queueing System RabbitMQ on AWS EC2 File Storage

    AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2
  19. Consumer Thread class ConsumerThread(threading.Thread): def __init__(self, queue): self.stop = False

    self.queue = queue def run(self): while not self.stop: msg = self.queue.get() process(msg)
  20. Queueing System Queueing System RabbitMQ on AWS EC2 File Storage

    AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2
  21. Queueing System powered by import json import pika # connect

    to RabbitMQ params = pika.URLParameters( 'amqp://guest:guest@localhost:5672/vhost') conn = pika.BlockingConnection(params) channel = conn.channel() # create a queue named 'WorkerQ' channel.queue_declare(queue='WorkerQ', durable=True)
  22. Putting a Message msg = json.dumps({ 'type': 'render', 'texts': {

    'name': 'Amy Reed', 'email': 'amy@example.com' }, ... }) channel.basic_publish( exchange='', routing_key='WorkerQ', body=msg, properties=pika.BasicProperties(delivery_mode=2) )
  23. Getting a Message _, _, body = channel.basic_get(queue='WorkerQ') msg =

    json.loads(body) if msg.get('type') == 'render': do_render(msg)
  24. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 After Effects Project and Asset Files Preparing Assets
  25. Replacing Text and Photo AEPX File (After Effects' XML format)

    <Layr> <string>AMY REED</string> .... <fileReference fullpath="C: \path\amy.png" ...>
  26. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Rendering Parallel rendering
  27. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Rendering Rendered Images
  28. Automated Rendering in After Effects

  29. Calling aerender in Python import subprocess subprocess.Popen([ 'aerender.exe', '-project', 'project.aepx',

    '-output', 'frmae.[#####].jpg', '-s', '100', # start frame '-e', '150', # end frame ... ])
  30. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Merging Images
  31. Queueing System RabbitMQ on AWS EC2 File Storage AWS S3

    Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Final Video 'Done' Message Merging Images
  32. Merging Images import subprocess subprocess.Popen([ 'ffmpeg', '-i', '/path/frame.%05d.jpg', '-i', '/path/music.mp3',

    '-s', '720x480', '-vcodec', 'libx264', '-acodec', 'libaac', ..., '/path/output.mp4' ])
  33. There's More • Scaling • Distributed Locking • Estimating Render

    Time • Render Watch
  34. Scaling UP from boto.ec2.connection import EC2Connection conn = EC2Connection( 'access_key_id',

    'secret_access_key') reservations = conn.get_all_instances() # find the first stopped EC2 instance and start it for res in reservations: for inst in res.instances: if inst.state == 'stopped': inst.start()
  35. Scaling DOWN # Windows-only shutdown subprocess.call([ 'shutdown', '/f', # by

    force '/s', # shutdown '/t', '0' # after 0 seconds ])
  36. There's More • Scaling • Distributed Locking • Estimating Render

    Time • Render Watch
  37. Distributed Locking Queueing System RabbitMQ on AWS EC2 File Storage

    AWS S3 Render Worker 1 AWS EC2 Render Worker 2 AWS EC2 Render Worker 3 AWS EC2 Render Manager AWS EC2 Shared Disk AWS EBS MySQL Web Server AWS EC2 Need a lock here!
  38. Distributed Locking - Lock def lock(lock_id, lock_timeout): lock_path = os.path.join('Z:\\locks',

    lock_id) while True: try: os.mkdir(lock_path) except OSError as err: if err.errno not in (errno.EEXIST, errno.EACCES): raise err try: time_locked = os.path.getctime(lock_path) if time.time() - time_locked < lock_timeout: time.sleep(1) # wait for it to be unlocked else: os.rmdir(lock_path) # lock expired except OSError: pass
  39. def unlock(lock_id): lock_path = os.path.join('Z:\\locks', lock_id) try: os.rmdir(lock_path) except OSError

    as err: pass Distributed Locking - Unlock
  40. There's More • Scaling • Distributed Locking • Estimating Render

    Time • Render Watch
  41. Estimating Render Time num_workers render_time 2 100 6 x 8

    50 Predict the future by reviewing the past: 6 - 2 x - 100 --------- = ------------ 8 - 2 50 - 100 => x = 66.7
  42. There's More • Scaling • Distributed Locking • Estimating Render

    Time • Render Watch
  43. Render Watch

  44. Summary • After Effects render cloud on AWS • Worker

    queue • Scalable and parallel • Python and Django Thanks! Q & A