codecentric training day

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. USING DC/OS 1 Michael Hausenblas, Developer & Cloud Advocate | 2016-06-03, codecentric, Solingen
2. © 2016 Mesosphere, Inc. All Rights Reserved. sys admin devops developer architect data engineer data scientist
3. © 2016 Mesosphere, Inc. All Rights Reserved. SOME ADMINISTRATIVE THINGS … 3 • Clusters: go to http://301.sh/cct2016 • We have clusters prepared → team up in groups of 3-4 people • One person should drive, others help/comment/plan • Join DC/OS Slack Community • http://chat.dcos.io/ • will invite you to private channel #cc2016 • Proctors are around: Bernd, Jens, Tobi
4. © 2016 Mesosphere, Inc. All Rights Reserved. TIME FOR A NEW KIND OF OPERATING SYSTEM 4
5. © 2016 Mesosphere, Inc. All Rights Reserved. SINGLE MACHINE APPLICATION 5 hardware OS app
6. © 2016 Mesosphere, Inc. All Rights Reserved. DISTRIBUTED APPLICATION 6 hardware OS app hardware OS hardware OS hardware OS hardware OS hardware OS hardware OS
7. © 2016 Mesosphere, Inc. All Rights Reserved. DISTRIBUTED OS + DISTRIBUTED APP 7 hardware OS app hardware OS hardware OS hardware OS hardware OS hardware OS hardware OS distributed OS
8. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS IS A DISTRIBUTED OS 8
9. © 2016 Mesosphere, Inc. All Rights Reserved. LOCAL OS
 VS
 DISTRIBUTED OS 9
10. © 2015 Mesosphere, Inc. All Rights Reserved. MEET THE DATACENTER OPERATING SYSTEM 10
11. © 2016 Mesosphere, Inc. 11 https://dcos.io
12. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS ARCHITECTURE 12
13. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS BENEFITS 13 • One cluster for • stateless services such as Web servers & app servers (via Marathon) • stateful services like PostgreSQL, MemSQL, Kafka, Cassandra, etc. • elastic data processing via Spark, Akka, etc. • CI/CD, for example Jenkins+Marathon • Dynamic partitioning of your cluster, depending on your needs • Increased utilization (10% → 80%+)
14. © 2016 Mesosphere, Inc. All Rights Reserved. 14
15. © 2016 Mesosphere, Inc. All Rights Reserved. 15
16. © 2016 Mesosphere, Inc. All Rights Reserved. APACHE MESOS & MARATHON 16
17. © 2016 Mesosphere, Inc. All Rights Reserved. APACHE MESOS 17 • A top-level ASF project • A cluster resource negotiator • Scalable to 10,000s of nodes but also useful for a handful of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support mesos.apache.org
18. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS
 ARCHITECTURE 18 http://mesos.berkeley.edu/mesos_tech_report.pdf Marathon
 scheduler Spark
 scheduler Spark
 executor Marathon
 executor Spark
 executor Marathon
 executor Mesos Agent Mesos Agent Mesos Agent Mesos
 Master Standby
 Master Standby
 Master Framework
19. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS
 RESOURCES 19 • resource—anything a task consumes to carry out its work • standard resources cpu
 mem
 disk
 ports • to guarantee fair allocation across resource types —Dominant Resource Fairness (DRF) algorithm
20. © 2016 Mesosphere, Inc. 20
21. © 2016 Mesosphere, Inc. 21
22. © 2016 Mesosphere, Inc. 22
23. © 2016 Mesosphere, Inc. 23
24. © 2016 Mesosphere, Inc. 24
25. © 2016 Mesosphere, Inc. 25
26. © 2016 Mesosphere, Inc. 26
27. © 2016 Mesosphere, Inc. 27
28. © 2016 Mesosphere, Inc. 28
29. © 2016 Mesosphere, Inc. 29
30. © 2016 Mesosphere, Inc. 30
31. © 2016 Mesosphere, Inc. 31
32. © 2016 Mesosphere, Inc. 32
33. © 2016 Mesosphere, Inc. 33
34. © 2016 Mesosphere, Inc. 34
35. © 2016 Mesosphere, Inc. 35
36. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON 36 Also acts as the distributed init system for DC/OS • starts instances of a long-running services • restarts the instances if they crash • supports health checks • supports multitude of upgrade strategies • HA built in
37. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 ORGANIZATIONAL PRIMITIVES 37 • groups can contain one or more apps/groups • good for dependency management/scaling • labels → good for non-hierarchical organization https://mesosphere.com/blog/2015/06/21/web-application-analytics-using-docker-and-marathon/
38. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 APP SPEC 38 { "id": "/webserver", "instances": 5, "cpus": 0.1, "mem": 128, "container": { "type": "DOCKER", "docker": { "image": "nginx:1.9.14", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0 } ] } }, "upgradeStrategy": { "minimumHealthCapacity": 0.9 }, "acceptedResourceRoles": [ "slave_public" ] }
39. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON UI/UX 39
40. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON HTTP API 40 https://mesosphere.github.io/marathon/docs/generated/api.html
41. © 2015 Mesosphere, Inc. Hands-on … 41
42. © 2015 Mesosphere, Inc. All Rights Reserved. ELASTIC DATA PIPELINES 42
43. © 2016 Mesosphere, Inc. All Rights Reserved. ELASTIC DATA PIPELINES WITH DC/OS 43 https://mesosphere.com/infinity/
44. © 2015 Mesosphere, Inc. All Rights Reserved. 44 EXAMPLE 1 github.com/mesosphere/cassandra-kairosdb-tutorial
45. © 2015 Mesosphere, Inc. All Rights Reserved. 45 EXAMPLE 2 mesosphere.com/blog/2015/11/18/dcos-time-series-demo/
46. © 2015 Mesosphere, Inc. Hands-on … 46
47. © 2015 Mesosphere, Inc. All Rights Reserved. CONTAINER OPERATIONS: CI/CD & DEPLOYMENT STRATEGIES 47
48. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 48 The why and the what: • Containers vs VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & security
49. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 49 • namespaces (isolation) • Isolate PIDs between processes • Isolate process to network resources • Isolate the hostname to fake it out (UTS) • Isolate the filesystem mount points (chroot) • Isolate inter process communication (IPC) • Isolate specific users to specific processes • cgroups (limiting & accounting)
 https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups
50. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 50
51. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 51 Registries • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • Run your own
 https://docs.docker.com/registry/deploying/
52. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER ARTIFACTS LAYER DIAGRAM 52
53. © 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 53
54. © 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 54 release
55. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 55
56. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD PIPELINES 56 https://github.com/mesosphere/training/tree/master/dcos-bootcamp-04-2016/ci-cd
57. © 2015 Mesosphere, Inc. Hands-on … 57
58. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS SERVICE DISCOVERY 58 DNS-based easy to integrate SRV records no health checks TTL Proxy-based no port conflicts fast failover no UDP management of VIPs (Minuteman) or service ports (Marathon-lb) Application-aware developer fully in control and full-feature implementation effort requires distributed state management (ZK, etcd or Consul) examples:
 Mesos-DNS,Consul examples:
 Minuteman, Marathon-lb examples:
 Roll-your-own, Finagle
59. © 2016 Mesosphere, Inc. All Rights Reserved. LOAD BALANCING TERMINOLOGY 59 rolling deployment bring up N instances of new app & terminate N instances of old app until all old instances are gone goal: minimize capacity requirements blue-green deployment launch a new stack and switch traffic from old to new when the new instances are healthy goal: minimize impact of regressions, friction, delays, and allow easy rollbacks canary deployment bring up a new stack, start by routing a small portion of traffic to the new app, and slowly increase goal: test production traffic slowly & safely
60. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS LOAD BALANCING 60 • Cluster-internal: Minuteman, a L4 distributed LB, usage via VIP in Marathon • Internal or edge: Marathon-lb, dynamically updates HAProxy, usage via package+service ports in Marathon • External, for example Azure's offerings
61. © 2016 Mesosphere, Inc. All Rights Reserved. DEPLOYMENT STRATEGIES 61 • Based on health checks • Policy via • minimumHealthCapacity float value between 0—1, specifies % of app instances to maintain healthy while performing deployment • maximumOverCapacity float value between 0 — 1, specifies the maximum % of instances that can be over capacity during deployment
62. © 2015 Mesosphere, Inc. Hands-on … 62
63. © 2015 Mesosphere, Inc. All Rights Reserved. MICROSERVICES ON DC/OS 63
64. © 2016 Mesosphere, Inc. All Rights Reserved. EXAMPLE 64 github.com/mhausenblas/marvin
65. © 2015 Mesosphere, Inc. Hands-on … 65
66. © 2015 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 66
67. © 2016 Mesosphere, Inc. All Rights Reserved. 67 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/9781939902184.do 67 http://shop.oreilly.com/product/0636920035671.do
68. © 2016 Mesosphere, Inc. All Rights Reserved. 68 WHERE CAN
 I LEARN MORE? 68 https://www.nginx.com/resources/library/docker-networking/
69. © 2016 Mesosphere, Inc. All Rights Reserved. 69 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action 69
70. © 2016 Mesosphere, Inc. All Rights Reserved. Q & A 70 • @mhausenblas • mhausenblas.info • michael@dcos.io https://dcos.io