Containers, FTW!

Containers, FTW!

Talk at ProgSCon 2016 in London, see also http://progscon.co.uk/program

5c3807aaaf0ffefe6c75e3dbbb8588b5?s=128

Michael Hausenblas

April 22, 2016
Tweet

Transcript

  1. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINERS, FTW! 1

    Michael Hausenblas | 2016-04-22 | ProgSCon, London
  2. © 2015 Mesosphere, Inc. All Rights Reserved. sys admin devops

    developer architect QA/testing other
  3. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER 101 3

  4. © 2016 Mesosphere, Inc. All Rights Reserved. LINUX
 CONTAINERS 4

    The why and the what: • Containers vs VMs • app-level dependency management • lightweight (startup time, footprint, average runtime) • isolation & security
  5. © 2016 Mesosphere, Inc. All Rights Reserved. LINUX
 CONTAINERS 5

    • namespaces • Isolate PIDs between processes • Isolate process to network resources • Isolate the hostname to fake it out (UTS) • Isolate the filesystem mount points (chroot) • Isolate inter process communication (IPC) • Isolate specific users to specific processes • cgroups
 https://sysadmincasts.com/episodes/14-introduction-to-linux-control-groups-cgroups
  6. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 6

  7. © 2016 Mesosphere, Inc. All Rights Reserved. DOCKER 7 Registries

    • Docker Hub
 https://hub.docker.com/ • Google Cloud
 https://cloud.google.com/tools/container-registry/ • AWS
 https://aws.amazon.com/ecr/ • Run your own
 https://docs.docker.com/registry/deploying/
  8. © 2015 Mesosphere, Inc. All Rights Reserved. ARE CONTAINERS SECURE?

    8
  9. © 2015 Mesosphere, Inc. All Rights Reserved. ARE CONTAINERS SECURE?

    9 • containers share same kernel (!) • namespaces (user NS: UID 0 recently introduced) • lock down networking (ICC, for example) • apply common sense when handling credentials
  10. © 2015 Mesosphere, Inc. All Rights Reserved. ARE CONTAINERS SECURE?

    10 Please, don't bake credentials into images … rather do: $ docker run -d -e API_TOKEN=SECRET somedatabase $ docker run -d -v $(pwd):/fsecret:/fsecret:ro somedatabase → even better: use a key-value in-memory store such as Square's KeyWhiz, HashiCorp's Vault, or Crypt or native solutions such as Kubernetes Secrets for credentials.
  11. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER OPERATIONS 11

  12. © 2016 Mesosphere, Inc. 12 https://dcos.io

  13. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS 101 13

    Any Service on DC/OS Any Infrastructure DC/OS Your favorite services, container formats, and those yet to come Build apps once on DCOS, and run it anywhere Runs distributed apps anywhere 
 as simply as running apps on your laptop
  14. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS ARCHITECTURE 14

  15. © 2016 Mesosphere, Inc. All Rights Reserved. LOCAL OS
 VS


    DISTRIBUTED OS 15
  16. © 2015 Mesosphere, Inc. All Rights Reserved. QUICK INTERLUDE: APACHE

    MESOS 16
  17. © 2016 Mesosphere, Inc. All Rights Reserved. APACHE MESOS 17

    • A top-level ASF project • A cluster resource negotiator • Scalable to 10,000s of nodes but also useful for a handful of nodes • Fault-tolerant, battle-tested • An SDK for distributed apps • Native Docker support mesos.apache.org
  18. © 2016 Mesosphere, Inc. All Rights Reserved. MESOS
 ARCHITECTURE 18

    http://mesos.berkeley.edu/mesos_tech_report.pdf Marathon
 scheduler Spark
 scheduler Spark
 executor Marathon
 executor Spark
 executor Marathon
 executor Mesos Agent Mesos Agent Mesos Agent Mesos
 Master Standby
 Master Standby
 Master Framework
  19. © 2015 Mesosphere, Inc. All Rights Reserved. RESOURCES 19 •

    resource: anything a task consumes to do its work • standard resources: cpu
 mem
 disk
 ports • Dominant Resource Fairness (DRF) algorithm guarantees fair allocation across resource types
  20. © 2015 Mesosphere, Inc. All Rights Reserved. … BACK TO

    DC/OS 20
  21. © 2016 Mesosphere, Inc. All Rights Reserved. DC/OS BENEFITS 21

    • One cluster for • stateless services such as Web servers & app servers (via Marathon) • stateful services like PostgreSQL, MemSQL, Kafka, Cassandra, etc. • elastic data processing via Spark, Akka, etc. • CI/CD, for example Jenkins+Marathon • Dynamic partitioning of your cluster, depending on your needs • Increased utilization (10% → 80%+)
  22. © 2015 Mesosphere, Inc. All Rights Reserved. 22

  23. © 2015 Mesosphere, Inc. All Rights Reserved. 23

  24. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON 24 The

    DC/OS 'init system' • starts instances of a long-running services • restarts the instances if they crash • provides composition primitives • supports health checks • supports rolling upgrades
  25. © 2015 Mesosphere, Inc. All Rights Reserved. MARATHON
 APP SPEC

    25 { "id": "/webserver", "instances": 5, "cpus": 0.1, "mem": 128, "container": { "type": "DOCKER", "docker": { "image": "nginx:1.9.14", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0 } ] } }, "upgradeStrategy": { "minimumHealthCapacity": 0.9, "maximumOverCapacity": 0.2 }, "acceptedResourceRoles": [ "slave_public" ] }
  26. © 2016 Mesosphere, Inc. All Rights Reserved. MARATHON
 GROUPS 26

    • Groups can contain one or more apps/groups • Dependency management • Scaling https://mesosphere.com/blog/2015/06/21/web-application-analytics-using-docker-and-marathon/
  27. © 2015 Mesosphere, Inc. All Rights Reserved. MARATHON UI/UX 27

  28. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD AND DEPLOYMENT

    STRATEGIES 28
  29. © 2016 Mesosphere, Inc. All Rights Reserved. CONTAINER LIFE CYCLE

    29
  30. © 2016 Mesosphere, Inc. All Rights Reserved. CI/CD WITH DC/OS

    30
  31. © 2016 Mesosphere, Inc. All Rights Reserved. HANDS-ON 31

  32. © 2016 Mesosphere, Inc. All Rights Reserved. LEARNING RESOURCES 32

  33. © 2015 Mesosphere, Inc. All Rights Reserved. 33 WHERE CAN


    I LEARN MORE? http://shop.oreilly.com/product/9781939902184.do 33 http://shop.oreilly.com/product/0636920035671.do
  34. © 2015 Mesosphere, Inc. All Rights Reserved. 34 WHERE CAN


    I LEARN MORE? 34 https://www.nginx.com/resources/library/docker-networking/
  35. © 2015 Mesosphere, Inc. All Rights Reserved. 35 WHERE CAN


    I LEARN MORE? http://shop.oreilly.com/product/0636920039952.do https://manning.com/books/mesos-in-action 35
  36. © 2016 Mesosphere, Inc. All Rights Reserved. Q & A

    36 • @mhausenblas • mhausenblas.info • michael.hausenblas@dcos.io https://dcos.io