Containers, FTW!

Talk at ProgSCon 2016 in London, see also http://progscon.co.uk/program

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINERS, FTW! 1 Michael Hausenblas | 2016-04-22 | ProgSCon, London
2. © 2015 Mesosphere, Inc. All Rights Reserved. sys admin devops developer architect QA/testing other
3. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER 101 3
4. © 2016 Mesosphere, Inc. All Rights Reserved. LINUX
 CONTAINERS 4 The why and the what: • Containers vs VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & security
5. © 2016 Mesosphere, Inc. All Rights Reserved. LINUX
 CONTAINERS 5 • namespaces • Isolate PIDs between processes • Isolate process to network resources • Isolate the hostname to fake it out (UTS) • Isolate the filesystem mount points (chroot) • Isolate inter process communication (IPC) • Isolate specific users to specific processes • cgroups
 https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups
6. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 6
7. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 7 Registries • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • Run your own
 https://docs.docker.com/registry/deploying/
8. © 2015 Mesosphere, Inc. All Rights Reserved. ARE CONTAINERS SECURE? 8
9. © 2015 Mesosphere, Inc. All Rights Reserved. ARE CONTAINERS SECURE? 9 • containers share same kernel (!) • namespaces (user NS: UID 0 recently introduced) • lock down networking (ICC, for example) • apply common sense when handling credentials
10. © 2015 Mesosphere, Inc. All Rights Reserved. ARE CONTAINERS SECURE? 10 Please, don't bake credentials into images … rather do: $ docker run -d -e API_TOKEN=SECRET somedatabase $ docker run -d -v $(pwd):/fsecret:/fsecret:ro somedatabase → even better: use a key-value in-memory store such as Square's KeyWhiz, HashiCorp's Vault, or Crypt or native solutions such as Kubernetes Secrets for credentials.
11. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER OPERATIONS 11
12. © 2016 Mesosphere, Inc. 12 https://dcos.io
13. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS 101 13 Any Service on DC/OS Any Infrastructure DC/OS Your favorite services, container formats, and those yet to come Build apps once on DCOS, and run it anywhere Runs distributed apps anywhere 
 as simply as running apps on your laptop
14. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS ARCHITECTURE 14
15. © 2016 Mesosphere, Inc. All Rights Reserved. LOCAL OS
 VS
 DISTRIBUTED OS 15
16. © 2015 Mesosphere, Inc. All Rights Reserved. QUICK INTERLUDE: APACHE MESOS 16
17. © 2016 Mesosphere, Inc. All Rights Reserved. APACHE MESOS 17 • A top-level ASF project • A cluster resource negotiator • Scalable to 10,000s of nodes but also useful for a handful of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support mesos.apache.org
18. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS
 ARCHITECTURE 18 http://mesos.berkeley.edu/mesos_tech_report.pdf Marathon
 scheduler Spark
 scheduler Spark
 executor Marathon
 executor Spark
 executor Marathon
 executor Mesos Agent Mesos Agent Mesos Agent Mesos
 Master Standby
 Master Standby
 Master Framework
19. © 2015 Mesosphere, Inc. All Rights Reserved. RESOURCES 19 • resource: anything a task consumes to do its work • standard resources: cpu
 mem
 disk
 ports • Dominant Resource Fairness (DRF) algorithm guarantees fair allocation across resource types
20. © 2015 Mesosphere, Inc. All Rights Reserved. … BACK TO DC/OS 20
21. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS BENEFITS 21 • One cluster for • stateless services such as Web servers & app servers (via Marathon) • stateful services like PostgreSQL, MemSQL, Kafka, Cassandra, etc. • elastic data processing via Spark, Akka, etc. • CI/CD, for example Jenkins+Marathon • Dynamic partitioning of your cluster, depending on your needs • Increased utilization (10% → 80%+)
22. © 2015 Mesosphere, Inc. All Rights Reserved. 22
23. © 2015 Mesosphere, Inc. All Rights Reserved. 23
24. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON 24 The DC/OS 'init system' • starts instances of a long-running services • restarts the instances if they crash • provides composition primitives • supports health checks • supports rolling upgrades
25. © 2015 Mesosphere, Inc. All Rights Reserved. MARATHON
 APP SPEC 25 { "id": "/webserver", "instances": 5, "cpus": 0.1, "mem": 128, "container": { "type": "DOCKER", "docker": { "image": "nginx:1.9.14", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0 } ] } }, "upgradeStrategy": { "minimumHealthCapacity": 0.9, "maximumOverCapacity": 0.2 }, "acceptedResourceRoles": [ "slave_public" ] }
26. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 GROUPS 26 • Groups can contain one or more apps/groups • Dependency management • Scaling https://mesosphere.com/blog/2015/06/21/web-application-analytics-using-docker-and-marathon/
27. © 2015 Mesosphere, Inc. All Rights Reserved. MARATHON UI/UX 27
28. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD AND DEPLOYMENT STRATEGIES 28
29. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER LIFE CYCLE 29
30. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD WITH DC/OS 30
31. © 2016 Mesosphere, Inc. All Rights Reserved. HANDS-ON 31
32. © 2016 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 32
33. © 2015 Mesosphere, Inc. All Rights Reserved. 33 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/9781939902184.do 33 http://shop.oreilly.com/product/0636920035671.do
34. © 2015 Mesosphere, Inc. All Rights Reserved. 34 WHERE CAN
 I LEARN MORE? 34 https://www.nginx.com/resources/library/docker-networking/
35. © 2015 Mesosphere, Inc. All Rights Reserved. 35 WHERE CAN
 I LEARN MORE? http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action 35
36. © 2016 Mesosphere, Inc. All Rights Reserved. Q & A 36 • @mhausenblas • mhausenblas.info • michael.hausenblas@dcos.io https://dcos.io