How Containers Are Made

Keynote at ContainerSched 2016, see https://skillsmatter.com/conferences/7429-containersched-2016-the-container-and-scheduler-conference

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. HOW CONTAINERS ARE MADE
 A LOOK INTO THE
 SAUSAGE FACTORY 1 Michael Hausenblas, Distributed Systems Jester | 2016-06-08 | ContainerSched, London @mhausenblas
2. © 2016 Mesosphere, Inc. All Rights Reserved. sys admin/SRE appops developer architect QA/test engineer data engineer
3. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER OPERATIONS 3
4. © 2016 Mesosphere, Inc. All Rights Reserved. WTF IS CONTAINER OPERATIONS? 4 containerization runtime
 specification deployment day 2
 operations
5. © 2016 Mesosphere, Inc. All Rights Reserved. WTF IS CONTAINER OPERATIONS? 5 examples layer source artifacts .scala, .go, .py, .sh, etc.
 and config files packaging Dockerfile, runc runtime Marathon app spec, Kubernetes RC, Docker compose
6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 appops The person who writes an app is also the person responsible for operating the app in prod.
7. © 2016 Mesosphere, Inc. All Rights Reserved. 7 It's not about provisioning
 a VM or installing a DC/OS cluster or replacing a faulty HDD …
 
 … this would be on the infrastructure team. appops
8. © 2016 Mesosphere, Inc. All Rights Reserved. 8 appops speakerdeck.com/charity/devops-for-developers-building-an-effective-ops-org-1
9. © 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 9
10. © 2016 Mesosphere, Inc. All Rights Reserved. APPOPS
 LIFE CYCLE 10 dev CI/CD CI/CD time QA prod dev QA prod release
11. © 2016 Mesosphere, Inc. All Rights Reserved. PICK YOUR POISON 11 templated
 serverless IFTTT,
 Microsoft Flow free-style 
 serverless AWS Lambda, Microsoft Services PaaS Heroku,
 Google App Engine containerized Docker/rkt
 with Marathon
 or Kubernetes traditional
 multi-tier monolith
 on bare metal
 or VM agility level of control
12. © 2016 Mesosphere, Inc. All Rights Reserved. PICK YOUR POISON 12 serverless microservices monolith nanoservices function container container machine unit: env: maintenance utilization costs/unit
13. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINERIZATION 13
14. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 14 The why and the what: • Containers vs VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & security containerization runtime
 specification deployment day 2
 operations
15. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER
 101 15 • namespaces (isolation) • Isolate PIDs between processes • Isolate process to network resources • Isolate the hostname to fake it out (UTS) • Isolate the filesystem mount points (chroot) • Isolate inter process communication (IPC) • Isolate specific users to specific processes • cgroups (limiting & accounting)
 https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups containerization runtime
 specification deployment day 2
 operations
16. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 16 containerization runtime
 specification deployment day 2
 operations
17. © 2016 Mesosphere, Inc. All Rights Reserved. REGISTRIES 17 • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • CoreOS
 https://quay.io/ • SUSE Portus
 http://port.us.org/ • JFrog Artifactory
 https://www.jfrog.com/artifactory/ • Run your own
 https://docs.docker.com/registry/deploying/ containerization runtime
 specification deployment day 2
 operations
18. © 2016 Mesosphere, Inc. All Rights Reserved. SECRETS 18 Please, don't bake credentials into images … rather do: $ docker run -d -e API_TOKEN=SECRET somedatabase $ docker run -d -v $(pwd):/fsecret:/fsecret:ro somedatabase Even better: use key-value in-memory stores such as Square's KeyWhiz, HashiCorp's Vault, or Crypt or native solutions such as Kubernetes Secrets containerization runtime
 specification deployment day 2
 operations
19. © 2016 Mesosphere, Inc. All Rights Reserved. A WORD OF CAUTION 19 containerize != Docker image containerization runtime
 specification deployment day 2
 operations github.com/mhausenblas/marvin
20. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER ORCHESTRATION 20
21. © 2015 Mesosphere, Inc. All Rights Reserved. WHAT IS
 CONTAINER ORCHESTRATION? 21 containerization runtime
 specification deployment day 2
 operations
22. © 2016 Mesosphere, Inc. All Rights Reserved. WHAT DOES A SCHEDULER DO? 22 containerization runtime
 specification deployment day 2
 operations
23. © 2015 Mesosphere, Inc. All Rights Reserved. DOCKER SWARM 23 http://www.slideshare.net/snrism/swarm-container-cluster-service containerization runtime
 specification deployment day 2
 operations
24. © 2015 Mesosphere, Inc. All Rights Reserved. KUBERNETES 24 http://k8s.info/cs.html containerization runtime
 specification deployment day 2
 operations
25. © 2015 Mesosphere, Inc. All Rights Reserved. KUBERNETES 25 http://k8s.info/cs.html containerization runtime
 specification deployment day 2
 operations
26. © 2015 Mesosphere, Inc. All Rights Reserved. NOMAD 26 https://www.nomadproject.io/docs/internals/architecture.html containerization runtime
 specification deployment day 2
 operations
27. © 2015 Mesosphere, Inc. All Rights Reserved. NOMAD 27 https://www.nomadproject.io/docs/internals/scheduling.html containerization runtime
 specification deployment day 2
 operations
28. © 2015 Mesosphere, Inc. All Rights Reserved. FIRMAMENT 28 http://www.firmament.io/ containerization runtime
 specification deployment day 2
 operations
29. © 2015 Mesosphere, Inc. All Rights Reserved. FIRMAMENT 29 http://www.firmament.io/ containerization runtime
 specification deployment day 2
 operations
30. © 2015 Mesosphere, Inc. All Rights Reserved. APACHE MESOS + MARATHON 30 http://mesos.berkeley.edu/mesos_tech_report.pdf Marathon
 scheduler Spark
 scheduler Spark
 executor Marathon
 executor Spark
 executor Marathon
 executor Mesos Agent Mesos Agent Mesos Agent Mesos
 Master Standby
 Master Standby
 Master Framework containerization runtime
 specification deployment day 2
 operations
31. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 APP SPEC 31 { "id": "/webserver", "instances": 5, "cpus": 0.1, "mem": 128, "container": { "type": "DOCKER", "docker": { "image": "nginx:1.9.14", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0 } ] } }, "upgradeStrategy": { "minimumHealthCapacity": 0.9 }, "acceptedResourceRoles": [ "slave_public" ] } containerization runtime
 specification deployment day 2
 operations
32. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS 32 https://dcos.io containerization runtime
 specification deployment day 2
 operations
33. © 2016 Mesosphere, Inc. All Rights Reserved. INTROSPECTION 33 weave.works/products/weave-scope/ containerization runtime
 specification deployment day 2
 operations
34. © 2016 Mesosphere, Inc. All Rights Reserved. DEPLOYMENT STRATEGIES 34
35. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS SERVICE DISCOVERY 35 DNS-based easy to integrate SRV records no health checks TTL Proxy-based no port conflicts fast failover no UDP management of VIPs (Minuteman) or service ports (Marathon-lb) Application-aware developer fully in control and full-feature implementation effort requires distributed state management (ZK, etcd or Consul) examples:
 Mesos-DNS,Consul examples:
 Minuteman, Marathon-lb examples:
 roll-your-own, Finagle containerization runtime
 specification deployment day 2
 operations
36. © 2016 Mesosphere, Inc. All Rights Reserved. ZERO-DOWNTIME DEPLOYMENTS 36 rolling deployment bring up N instances of new app & terminate N instances of old app until all old instances are gone goal: minimize capacity requirements blue-green deployment launch a new stack and switch traffic from old to new when the new instances are healthy goal: minimize impact of regressions, friction, delays, and allow easy rollbacks canary deployment bring up a new stack, start by routing a small portion of traffic to the new app, and slowly increase goal: test production traffic slowly & safely containerization runtime
 specification deployment day 2
 operations
37. © 2016 Mesosphere, Inc. All Rights Reserved. DEPLOYMENT STRATEGIES (DC/OS) 37 • Based on health checks • Policy via • minimumHealthCapacity float value between 0—1, specifies % of app instances to maintain healthy while performing deployment • maximumOverCapacity float value between 0 — 1, specifies the maximum % of instances that can be over capacity during deployment containerization runtime
 specification deployment day 2
 operations
38. © 2016 Mesosphere, Inc. All Rights Reserved. DAY 2 OPERATIONS 38
39. © 2016 Mesosphere, Inc. All Rights Reserved. DAY 2
 OPERATIONS 39 • Logging • Monitoring and alerting • Sysdig • Prometheus • DataDog • Auditing • Capacity planning, auto-scaling, oversubscription containerization runtime
 specification deployment day 2
 operations
40. © 2016 Mesosphere, Inc. All Rights Reserved. HANDS ON … 40
41. © 2016 Mesosphere, Inc. All Rights Reserved. A CI/CD PIPELINE FOR CONTAINERS 41 https://github.com/mesosphere/training/tree/master/dcos-bootcamp-04-2016/ci-cd
42. © 2016 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 42
43. © 2016 Mesosphere, Inc. All Rights Reserved. 43 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/9781939902184.do 43 http://shop.oreilly.com/product/0636920035671.do
44. © 2016 Mesosphere, Inc. All Rights Reserved. 44 WHERE CAN
 I LEARN MORE? 44 http://301.sh/ora2016-dnsd http://301.sh/ora2016-dnsd
45. © 2016 Mesosphere, Inc. All Rights Reserved. 45 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action 45
46. © 2016 Mesosphere, Inc. All Rights Reserved. Q & A 46 • @mhausenblas • mhausenblas.info • michael@dcos.io https://dcos.io