Tweet
Tweet

Slide 1

Slide 1 text

Docker containers at scale with Apache Mesos Distributed Systems Engineer Apache Mesos PMC Timothy Chen @tnachen [email protected]

Slide 2

Slide 2 text

 Mesoswhat?  Marathon  Chronos  Demo!

Slide 3

Slide 3 text

Status quo is static partitioning and use of virtual machines

Slide 4

Slide 4 text

Add some virtual machines

Slide 5

Slide 5 text

Provision Hadoop

Slide 6

Slide 6 text

Provision a web service

Slide 7

Slide 7 text

Moar data, moar Hadoop

Slide 8

Slide 8 text

Mesos let us treat a cluster of nodes...

Slide 9

Slide 9 text

As one big computer

Slide 10

Slide 10 text

 Not as individual machines  Not as VMs

Slide 11

Slide 11 text

But as computational resources like cores, memory, disks, etc.

Slide 12

Slide 12 text

240 million monthly active users 500 million tweets per day Up to 150k tweets per second More than 100TB per day of compressed data

Slide 13

Slide 13 text

“Mesos is the cornerstone of our elastic compute infrastructure — it's how we build all our new services and is critical for Twitter's continued success at scale. It's one of the primary keys to our data center efficiency.” — Chris Fry, SVP of Engineering at Twitter

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

Containerization in Mesos, a brief history

Slide 19

Slide 19 text

Mesos is...  A top-level Apache project  A cluster resource negotiator  Scalable to 10,000s of nodes  Fault-tolerant, battle-tested  An SDK for distributed apps

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Mesos for all the things

Slide 22

Slide 22 text

 Marathon  Mesoswhat?  Chronos  Demo!

Slide 23

Slide 23 text

a self-serve interface to your cluster

Slide 24

Slide 24 text

distributed "init" for long-running services

Slide 25

Slide 25 text

a private fault-tolerant PaaS

Slide 26

Slide 26 text

Marathon does it! Start, stop, scale, update apps Nice web interface, API Highly available, no SPoF Native Docker support Fully featured REST API Pluggable event bus Rolling deploy / restart Application health checks Artifact staging

Slide 27

Slide 27 text

 Chronos  Mesoswhat?  Marathon  Demo!

Slide 28

Slide 28 text

Introducing Chronos a scheduler for batch and one-off jobs

Slide 29

Slide 29 text

Distribute a graph of jobs Dependency graph for execution

Slide 30

Slide 30 text

Features Distributed job scheduler Web interface, API Highly available, no SPoF Native Docker support Easy scheduling with repeating intervals

Slide 31

Slide 31 text

Chronos  REST P U T c h r o n o s - n o d e : 8 0 8 0 / s c h e d u l e r / j o b / j o b 1 G E T c h r o n o s - n o d e : 8 0 8 0 / s c h e d u l e r / j o b s D E L E T E c h r o n o s - n o d e : 8 0 8 0 / s c h e d u l e r / t a s k / k i l l / j o b 2

Slide 32

Slide 32 text

 Demo!  Mesoswhat?  Marathon  Chronos

Slide 33

Slide 33 text

Thanks! Come and talk to me P.S., we're hiring!