Slide 1

Slide 1 text

© 2016 Mesosphere, Inc. All Rights Reserved. HOW CONTAINERS ARE MADE
 A LOOK INTO THE
 SAUSAGE FACTORY 1 Michael Hausenblas, Distributed Systems Jester | 2016-06-08 | ContainerSched, London @mhausenblas

Slide 2

Slide 2 text

© 2016 Mesosphere, Inc. All Rights Reserved. sys admin/SRE appops developer architect QA/test engineer data engineer

Slide 3

Slide 3 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER OPERATIONS 3

Slide 4

Slide 4 text

© 2016 Mesosphere, Inc. All Rights Reserved. WTF IS CONTAINER OPERATIONS? 4 containerization runtime
 specification deployment day 2
 operations

Slide 5

Slide 5 text

© 2016 Mesosphere, Inc. All Rights Reserved. WTF IS CONTAINER OPERATIONS? 5 examples layer source artifacts .scala, .go, .py, .sh, etc.
 and config files packaging Dockerfile, runc runtime Marathon app spec, Kubernetes RC, Docker compose

Slide 6

Slide 6 text

© 2016 Mesosphere, Inc. All Rights Reserved. 6 appops The person who writes an app is also the person responsible for operating the app in prod.

Slide 7

Slide 7 text

© 2016 Mesosphere, Inc. All Rights Reserved. 7 It's not about provisioning
 a VM or installing a DC/OS cluster or replacing a faulty HDD …
 
 … this would be on the infrastructure team. appops

Slide 8

Slide 8 text

© 2016 Mesosphere, Inc. All Rights Reserved. 8 appops speakerdeck.com/charity/devops-for-developers-building-an-effective-ops-org-1

Slide 9

Slide 9 text

© 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 9

Slide 10

Slide 10 text

© 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 10 dev CI/CD CI/CD time QA prod dev QA prod release

Slide 11

Slide 11 text

© 2016 Mesosphere, Inc. All Rights Reserved. PICK YOUR POISON 11 templated
 serverless IFTTT,
 Microsoft Flow free-style 
 serverless AWS Lambda, Microsoft Services PaaS Heroku,
 Google App Engine containerized Docker/rkt
 with Marathon
 or Kubernetes traditional
 multi-tier monolith
 on bare metal
 or VM agility level of control

Slide 12

Slide 12 text

© 2016 Mesosphere, Inc. All Rights Reserved. PICK YOUR POISON 12 serverless microservices monolith nanoservices function container container machine unit: env: maintenance utilization costs/unit

Slide 13

Slide 13 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINERIZATION 13

Slide 14

Slide 14 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 14 The why and the what: • Containers vs VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & security containerization runtime
 specification deployment day 2
 operations

Slide 15

Slide 15 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 15 • namespaces (isolation) • Isolate PIDs between processes • Isolate process to network resources • Isolate the hostname to fake it out (UTS) • Isolate the filesystem mount points (chroot) • Isolate inter process communication (IPC) • Isolate specific users to specific processes • cgroups (limiting & accounting)
 https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups containerization runtime
 specification deployment day 2
 operations

Slide 16

Slide 16 text

© 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 16 containerization runtime
 specification deployment day 2
 operations

Slide 17

Slide 17 text

© 2016 Mesosphere, Inc. All Rights Reserved. REGISTRIES 17 • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • CoreOS
 https://quay.io/ • SUSE Portus
 http://port.us.org/ • JFrog Artifactory
 https://www.jfrog.com/artifactory/ • Run your own
 https://docs.docker.com/registry/deploying/ containerization runtime
 specification deployment day 2
 operations

Slide 18

Slide 18 text

© 2016 Mesosphere, Inc. All Rights Reserved. SECRETS 18 Please, don't bake credentials into images … rather do: $ docker run -d -e API_TOKEN=SECRET somedatabase $ docker run -d -v $(pwd):/fsecret:/fsecret:ro somedatabase Even better: use key-value in-memory stores such as Square's KeyWhiz, HashiCorp's Vault, or Crypt or native solutions such as Kubernetes Secrets containerization runtime
 specification deployment day 2
 operations

Slide 19

Slide 19 text

© 2016 Mesosphere, Inc. All Rights Reserved. A WORD OF CAUTION 19 containerize != Docker image containerization runtime
 specification deployment day 2
 operations github.com/mhausenblas/marvin

Slide 20

Slide 20 text

© 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER ORCHESTRATION 20

Slide 21

Slide 21 text

© 2015 Mesosphere, Inc. All Rights Reserved. WHAT IS
 CONTAINER ORCHESTRATION? 21 containerization runtime
 specification deployment day 2
 operations

Slide 22

Slide 22 text

© 2016 Mesosphere, Inc. All Rights Reserved. WHAT DOES A SCHEDULER DO? 22 containerization runtime
 specification deployment day 2
 operations

Slide 23

Slide 23 text

© 2015 Mesosphere, Inc. All Rights Reserved. DOCKER SWARM 23 http://www.slideshare.net/snrism/swarm-container-cluster-service containerization runtime
 specification deployment day 2
 operations

Slide 24

Slide 24 text

© 2015 Mesosphere, Inc. All Rights Reserved. KUBERNETES 24 http://k8s.info/cs.html containerization runtime
 specification deployment day 2
 operations

Slide 25

Slide 25 text

© 2015 Mesosphere, Inc. All Rights Reserved. KUBERNETES 25 http://k8s.info/cs.html containerization runtime
 specification deployment day 2
 operations

Slide 26

Slide 26 text

© 2015 Mesosphere, Inc. All Rights Reserved. NOMAD 26 https://www.nomadproject.io/docs/internals/architecture.html containerization runtime
 specification deployment day 2
 operations

Slide 27

Slide 27 text

© 2015 Mesosphere, Inc. All Rights Reserved. NOMAD 27 https://www.nomadproject.io/docs/internals/scheduling.html containerization runtime
 specification deployment day 2
 operations

Slide 28

Slide 28 text

© 2015 Mesosphere, Inc. All Rights Reserved. FIRMAMENT 28 http://www.firmament.io/ containerization runtime
 specification deployment day 2
 operations

Slide 29

Slide 29 text

© 2015 Mesosphere, Inc. All Rights Reserved. FIRMAMENT 29 http://www.firmament.io/ containerization runtime
 specification deployment day 2
 operations

Slide 30

Slide 30 text

© 2015 Mesosphere, Inc. All Rights Reserved. APACHE MESOS + MARATHON 30 http://mesos.berkeley.edu/mesos_tech_report.pdf Marathon
 scheduler Spark
 scheduler Spark
 executor Marathon
 executor Spark
 executor Marathon
 executor Mesos Agent Mesos Agent Mesos Agent Mesos
 Master Standby
 Master Standby
 Master Framework containerization runtime
 specification deployment day 2
 operations

Slide 31

Slide 31 text

© 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 APP SPEC 31 { "id": "/webserver", "instances": 5, "cpus": 0.1, "mem": 128, "container": { "type": "DOCKER", "docker": { "image": "nginx:1.9.14", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0 } ] } }, "upgradeStrategy": { "minimumHealthCapacity": 0.9 }, "acceptedResourceRoles": [ "slave_public" ] } containerization runtime
 specification deployment day 2
 operations

Slide 32

Slide 32 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS 32 https://dcos.io containerization runtime
 specification deployment day 2
 operations

Slide 33

Slide 33 text

© 2016 Mesosphere, Inc. All Rights Reserved. INTROSPECTION 33 weave.works/products/weave-scope/ containerization runtime
 specification deployment day 2
 operations

Slide 34

Slide 34 text

© 2016 Mesosphere, Inc. All Rights Reserved. DEPLOYMENT STRATEGIES 34

Slide 35

Slide 35 text

© 2016 Mesosphere, Inc. All Rights Reserved. DC/OS SERVICE DISCOVERY 35 DNS-based easy to integrate SRV records no health checks TTL Proxy-based no port conflicts fast failover no UDP management of VIPs (Minuteman) or service ports (Marathon-lb) Application-aware developer fully in control and full-feature implementation effort requires distributed state management (ZK, etcd or Consul) examples:
 Mesos-DNS,Consul examples:
 Minuteman, Marathon-lb examples:
 roll-your-own, Finagle containerization runtime
 specification deployment day 2
 operations

Slide 36

Slide 36 text

© 2016 Mesosphere, Inc. All Rights Reserved. ZERO-DOWNTIME DEPLOYMENTS 36 rolling deployment bring up N instances of new app & terminate N instances of old app until all old instances are gone goal: minimize capacity requirements blue-green deployment launch a new stack and switch traffic from old to new when the new instances are healthy goal: minimize impact of regressions, friction, delays, and allow easy rollbacks canary deployment bring up a new stack, start by routing a small portion of traffic to the new app, and slowly increase goal: test production traffic slowly & safely containerization runtime
 specification deployment day 2
 operations

Slide 37

Slide 37 text

© 2016 Mesosphere, Inc. All Rights Reserved. DEPLOYMENT STRATEGIES (DC/OS) 37 • Based on health checks • Policy via • minimumHealthCapacity float value between 0—1, specifies % of app instances to maintain healthy while performing deployment • maximumOverCapacity float value between 0 — 1, specifies the maximum % of instances that can be over capacity during deployment containerization runtime
 specification deployment day 2
 operations

Slide 38

Slide 38 text

© 2016 Mesosphere, Inc. All Rights Reserved. DAY 2 OPERATIONS 38

Slide 39

Slide 39 text

© 2016 Mesosphere, Inc. All Rights Reserved. DAY 2
 OPERATIONS 39 • Logging • Monitoring and alerting • Sysdig • Prometheus • DataDog • Auditing • Capacity planning, auto-scaling, oversubscription containerization runtime
 specification deployment day 2
 operations

Slide 40

Slide 40 text

© 2016 Mesosphere, Inc. All Rights Reserved. HANDS ON … 40

Slide 41

Slide 41 text

© 2016 Mesosphere, Inc. All Rights Reserved. A CI/CD PIPELINE FOR CONTAINERS 41 https://github.com/mesosphere/training/tree/master/dcos-bootcamp-04-2016/ci-cd

Slide 42

Slide 42 text

© 2016 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 42

Slide 43

Slide 43 text

© 2016 Mesosphere, Inc. All Rights Reserved. 43 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/9781939902184.do 43 http://shop.oreilly.com/product/0636920035671.do

Slide 44

Slide 44 text

© 2016 Mesosphere, Inc. All Rights Reserved. 44 WHERE CAN
 I LEARN MORE? 44 http://301.sh/ora2016-dnsd http://301.sh/ora2016-dnsd

Slide 45

Slide 45 text

© 2016 Mesosphere, Inc. All Rights Reserved. 45 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action 45

Slide 46

Slide 46 text

© 2016 Mesosphere, Inc. All Rights Reserved. Q & A 46 • @mhausenblas • mhausenblas.info • [email protected] https://dcos.io