Herding Cattle (Or: Service Discovery Done Right)

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

© 2015 Mesosphere, Inc. All Rights Reserved. WHAT  IS  THIS  ALL ABOUT? 3 Pets vs Cattle Pets are individuals that you give names and when they get ill you nurse them back to health. Cattle are anonymous, identical to other cattle you assign numbers and when they get ill → next one. http://www.theregister.co.uk/2013/03/18/servers_pets_or_cattle_cern/

Slide 4

Slide 4 text

© 2015 Mesosphere, Inc. All Rights Reserved. CONSEQUENCES  OF GOING  ALL-IN WITH  CATTLE APPROACH 4 Pets vs Cattle • scale out on commodity hardware • elasticity • 'cheap' & 'simple' • R U on pager duty? Just sleep through! • social ≫ technology challenge • where the f#@k is my app? http://www.theregister.co.uk/2013/03/18/servers_pets_or_cattle_cern/

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

© 2015 Mesosphere, Inc. All Rights Reserved. SERVICE DISCOVERY CHALLENGES 10 • Remember the location, that is IP:PORT of each app across re-launches throughout the cluster • Potentially exclude unhealthy hosts • Others/related • strong consistency vs. highly available  (or: the CAP theorem strikes, again) • how about load balancing?

Slide 11

Slide 11 text

Slide 12

Slide 12 text

© 2015 Mesosphere, Inc. All Rights Reserved. SERVICE DISCOVERY  OPTIONS 12 Name Consistency Language Options etcd strong Go sidekick+client registration Eureka eventual Java client registration Consul strong Go auto | Traeﬁk Mesos-DNS strong Go auto | Traeﬁk (Marathon) SkyDNS strong Go client registration Weave-DNS strong Go auto Zookeeper strong Java client registration http://programmableinfrastructure.com/components/service-discovery/

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

© 2015 Mesosphere, Inc. All Rights Reserved. DCOS IS A DISTRIBUTED OPERATING SYSTEM 16 • kernel (Apache Mesos, written in C++) scales to 10,000 of nodes • fault-tolerant in all components, rolling upgrades throughout • containers ﬁrst class citizens (LXC, Docker) • local OS per node (+container enabled) • scheduling (long-lived, batch) • service discovery, monitoring, logging, debugging

Slide 17

Slide 17 text

Slide 18

Slide 18 text

© 2015 Mesosphere, Inc. All Rights Reserved. BENEFITS 18 DCOS • Run stateless services such as Web servers, app servers (via Marathon) and stateful services like Crate, Kafka, HDFS, Cassandra, ArangoDB etc. together on one cluster • Dynamic partitioning of your cluster, depending on your needs (business requirements) • Increased utilization (10% → 80% an more)

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

© 2015 Mesosphere, Inc. All Rights Reserved. SERVICE DISCOVERY  EXAMPLE 2 21 { "id": "webserver", "cmd": "python3 -m http.server 8080", "cpus": 0.5, "mem": 32.0, "container": { "type": "DOCKER", "docker": { "image": "python:3", "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0 } ] } }, "acceptedResourceRoles": [ "slave_public" ], "constraints": [ [ "hostname", "UNIQUE" ] ] }

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

© 2015 Mesosphere, Inc. All Rights Reserved. Q & A 24 • @mhausenblas • mhausenblas.info • @mesosphere • mesosphere.io/product Kudos for material in this talk go out to:  The folks who wrote 'Docker in Production' and my colleagues from p24e.io http://shop.oreilly.com/product/9781939902184.do http://p24e.io