Tweet
Tweet

Slide 1

Slide 1 text

© 2016 Mesosphere, Inc. All Rights Reserved. USING DC/OS 1 Michael Hausenblas, Developer & Cloud Advocate | 2016-06-03, codecentric, Solingen

Slide 2

Slide 2 text

© 2016 Mesosphere, Inc. All Rights Reserved. sys admin devops developer architect data engineer data scientist

Slide 3

Slide 3 text

© 2016 Mesosphere, Inc. All Rights Reserved. SOME ADMINISTRATIVE THINGS … 3 • Clusters: go to http://301.sh/cct2016 • We have clusters prepared → team up in groups of 3-4 people • One person should drive, others help/comment/plan • Join DC/OS Slack Community • http://chat.dcos.io/ • will invite you to private channel #cc2016 • Proctors are around: Bernd, Jens, Tobi

Slide 4

Slide 4 text

© 2016 Mesosphere, Inc. All Rights Reserved. TIME FOR A NEW KIND OF OPERATING SYSTEM 4

Slide 5

Slide 5 text

© 2016 Mesosphere, Inc. All Rights Reserved. SINGLE MACHINE APPLICATION 5 hardware OS app

Slide 6

Slide 6 text

© 2016 Mesosphere, Inc. All Rights Reserved. DISTRIBUTED APPLICATION 6 hardware OS app hardware OS hardware OS hardware OS hardware OS hardware OS hardware OS

Slide 7

Slide 7 text

© 2016 Mesosphere, Inc. All Rights Reserved. DISTRIBUTED OS + DISTRIBUTED APP 7 hardware OS app hardware OS hardware OS hardware OS hardware OS hardware OS hardware OS distributed OS

Slide 8

Slide 8 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS IS A DISTRIBUTED OS 8

Slide 9

Slide 9 text

© 2016 Mesosphere, Inc. All Rights Reserved. LOCAL OS
 VS
 DISTRIBUTED OS 9

Slide 10

Slide 10 text

© 2015 Mesosphere, Inc. All Rights Reserved. MEET THE DATACENTER OPERATING SYSTEM 10

Slide 11

Slide 11 text

© 2016 Mesosphere, Inc. 11 https://dcos.io

Slide 12

Slide 12 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS ARCHITECTURE 12

Slide 13

Slide 13 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS BENEFITS 13 • One cluster for • stateless services such as Web servers & app servers (via Marathon) • stateful services like PostgreSQL, MemSQL, Kafka, Cassandra, etc. • elastic data processing via Spark, Akka, etc. • CI/CD, for example Jenkins+Marathon • Dynamic partitioning of your cluster, depending on your needs • Increased utilization (10% → 80%+)

Slide 14

Slide 14 text

© 2016 Mesosphere, Inc. All Rights Reserved. 14

Slide 15

Slide 15 text

© 2016 Mesosphere, Inc. All Rights Reserved. 15

Slide 16

Slide 16 text

© 2016 Mesosphere, Inc. All Rights Reserved. APACHE MESOS & MARATHON 16

Slide 17

Slide 17 text

© 2016 Mesosphere, Inc. All Rights Reserved. APACHE MESOS 17 • A top-level ASF project • A cluster resource negotiator • Scalable to 10,000s of nodes but also useful for a handful of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support mesos.apache.org

Slide 18

Slide 18 text

© 2016 Mesosphere, Inc. All Rights Reserved. MESOS
 ARCHITECTURE 18 http://mesos.berkeley.edu/mesos_tech_report.pdf Marathon
 scheduler Spark
 scheduler Spark
 executor Marathon
 executor Spark
 executor Marathon
 executor Mesos Agent Mesos Agent Mesos Agent Mesos
 Master Standby
 Master Standby
 Master Framework

Slide 19

Slide 19 text

© 2016 Mesosphere, Inc. All Rights Reserved. MESOS
 RESOURCES 19 • resource—anything a task consumes to carry out its work • standard resources cpu
 mem
 disk
 ports • to guarantee fair allocation across resource types —Dominant Resource Fairness (DRF) algorithm

Slide 20

Slide 20 text

© 2016 Mesosphere, Inc. 20

Slide 21

Slide 21 text

© 2016 Mesosphere, Inc. 21

Slide 22

Slide 22 text

© 2016 Mesosphere, Inc. 22

Slide 23

Slide 23 text

© 2016 Mesosphere, Inc. 23

Slide 24

Slide 24 text

© 2016 Mesosphere, Inc. 24

Slide 25

Slide 25 text

© 2016 Mesosphere, Inc. 25

Slide 26

Slide 26 text

© 2016 Mesosphere, Inc. 26

Slide 27

Slide 27 text

© 2016 Mesosphere, Inc. 27

Slide 28

Slide 28 text

© 2016 Mesosphere, Inc. 28

Slide 29

Slide 29 text

© 2016 Mesosphere, Inc. 29

Slide 30

Slide 30 text

© 2016 Mesosphere, Inc. 30

Slide 31

Slide 31 text

© 2016 Mesosphere, Inc. 31

Slide 32

Slide 32 text

© 2016 Mesosphere, Inc. 32

Slide 33

Slide 33 text

© 2016 Mesosphere, Inc. 33

Slide 34

Slide 34 text

© 2016 Mesosphere, Inc. 34

Slide 35

Slide 35 text

© 2016 Mesosphere, Inc. 35

Slide 36

Slide 36 text

© 2016 Mesosphere, Inc. All Rights Reserved. MARATHON 36 Also acts as the distributed init system for DC/OS • starts instances of a long-running services • restarts the instances if they crash • supports health checks • supports multitude of upgrade strategies • HA built in

Slide 37

Slide 37 text

© 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 ORGANIZATIONAL PRIMITIVES 37 • groups can contain one or more apps/groups • good for dependency management/scaling • labels → good for non-hierarchical organization https://mesosphere.com/blog/2015/06/21/web-application-analytics-using-docker-and-marathon/

Slide 38

Slide 38 text

© 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 APP SPEC 38 { "id": "/webserver", "instances": 5, "cpus": 0.1, "mem": 128, "container": { "type": "DOCKER", "docker": { "image": "nginx:1.9.14", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0 } ] } }, "upgradeStrategy": { "minimumHealthCapacity": 0.9 }, "acceptedResourceRoles": [ "slave_public" ] }

Slide 39

Slide 39 text

© 2016 Mesosphere, Inc. All Rights Reserved. MARATHON UI/UX 39

Slide 40

Slide 40 text

© 2016 Mesosphere, Inc. All Rights Reserved. MARATHON HTTP API 40 https://mesosphere.github.io/marathon/docs/generated/api.html

Slide 41

Slide 41 text

© 2015 Mesosphere, Inc. Hands-on … 41

Slide 42

Slide 42 text

© 2015 Mesosphere, Inc. All Rights Reserved. ELASTIC DATA PIPELINES 42

Slide 43

Slide 43 text

© 2016 Mesosphere, Inc. All Rights Reserved. ELASTIC DATA PIPELINES WITH DC/OS 43 https://mesosphere.com/infinity/

Slide 44

Slide 44 text

© 2015 Mesosphere, Inc. All Rights Reserved. 44 EXAMPLE 1 github.com/mesosphere/cassandra-kairosdb-tutorial

Slide 45

Slide 45 text

© 2015 Mesosphere, Inc. All Rights Reserved. 45 EXAMPLE 2 mesosphere.com/blog/2015/11/18/dcos-time-series-demo/

Slide 46

Slide 46 text

© 2015 Mesosphere, Inc. Hands-on … 46

Slide 47

Slide 47 text

© 2015 Mesosphere, Inc. All Rights Reserved. CONTAINER OPERATIONS: CI/CD & DEPLOYMENT STRATEGIES 47

Slide 48

Slide 48 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 48 The why and the what: • Containers vs VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & security

Slide 49

Slide 49 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 49 • namespaces (isolation) • Isolate PIDs between processes • Isolate process to network resources • Isolate the hostname to fake it out (UTS) • Isolate the filesystem mount points (chroot) • Isolate inter process communication (IPC) • Isolate specific users to specific processes • cgroups (limiting & accounting)
 https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups

Slide 50

Slide 50 text

© 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 50

Slide 51

Slide 51 text

© 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 51 Registries • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • Run your own
 https://docs.docker.com/registry/deploying/

Slide 52

Slide 52 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER ARTIFACTS LAYER DIAGRAM 52

Slide 53

Slide 53 text

© 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 53

Slide 54

Slide 54 text

© 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 54 release

Slide 55

Slide 55 text

© 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 55

Slide 56

Slide 56 text

© 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 56 https://github.com/mesosphere/training/tree/master/dcos-bootcamp-04-2016/ci-cd

Slide 57

Slide 57 text

© 2015 Mesosphere, Inc. Hands-on … 57

Slide 58

Slide 58 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS SERVICE DISCOVERY 58 DNS-based easy to integrate SRV records no health checks TTL Proxy-based no port conflicts fast failover no UDP management of VIPs (Minuteman) or service ports (Marathon-lb) Application-aware developer fully in control and full-feature implementation effort requires distributed state management (ZK, etcd or Consul) examples:
 Mesos-DNS,Consul examples:
 Minuteman, Marathon-lb examples:
 Roll-your-own, Finagle

Slide 59

Slide 59 text

© 2016 Mesosphere, Inc. All Rights Reserved. LOAD BALANCING TERMINOLOGY 59 rolling deployment bring up N instances of new app & terminate N instances of old app until all old instances are gone goal: minimize capacity requirements blue-green deployment launch a new stack and switch traffic from old to new when the new instances are healthy goal: minimize impact of regressions, friction, delays, and allow easy rollbacks canary deployment bring up a new stack, start by routing a small portion of traffic to the new app, and slowly increase goal: test production traffic slowly & safely

Slide 60

Slide 60 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS LOAD BALANCING 60 • Cluster-internal: Minuteman, a L4 distributed LB, usage via VIP in Marathon • Internal or edge: Marathon-lb, dynamically updates HAProxy, usage via package+service ports in Marathon • External, for example Azure's offerings

Slide 61

Slide 61 text

© 2016 Mesosphere, Inc. All Rights Reserved. DEPLOYMENT STRATEGIES 61 • Based on health checks • Policy via • minimumHealthCapacity float value between 0—1, specifies % of app instances to maintain healthy while performing deployment • maximumOverCapacity float value between 0 — 1, specifies the maximum % of instances that can be over capacity during deployment

Slide 62

Slide 62 text

© 2015 Mesosphere, Inc. Hands-on … 62

Slide 63

Slide 63 text

© 2015 Mesosphere, Inc. All Rights Reserved. MICROSERVICES ON DC/OS 63

Slide 64

Slide 64 text

© 2016 Mesosphere, Inc. All Rights Reserved. EXAMPLE 64 github.com/mhausenblas/marvin

Slide 65

Slide 65 text

© 2015 Mesosphere, Inc. Hands-on … 65

Slide 66

Slide 66 text

© 2015 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 66

Slide 67

Slide 67 text

© 2016 Mesosphere, Inc. All Rights Reserved. 67 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/9781939902184.do 67 http://shop.oreilly.com/product/0636920035671.do

Slide 68

Slide 68 text

© 2016 Mesosphere, Inc. All Rights Reserved. 68 WHERE CAN
 I LEARN MORE? 68 https://www.nginx.com/resources/library/docker-networking/

Slide 69

Slide 69 text

© 2016 Mesosphere, Inc. All Rights Reserved. 69 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action 69

Slide 70

Slide 70 text

© 2016 Mesosphere, Inc. All Rights Reserved. Q & A 70 • @mhausenblas • mhausenblas.info • [email protected] https://dcos.io