Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Adopting Docker

Adopting Docker

Talk from Pragmatic Docker, all about use cases for adopting docker from an operations perspective.


Gareth Rushgrove

April 21, 2015


  1. Adopting Docker Puppet Labs Gareth Rushgrove Common use cases, from

    getting started to production
  2. @garethr

  3. Gareth Rushgrove

  4. Gareth Rushgrove

  5. This talk

  6. Solving local development problems Gareth Rushgrove

  7. Improving continuous integration Gareth Rushgrove

  8. First steps into production Gareth Rushgrove

  9. Going full container Gareth Rushgrove

  10. Local development dependencies

  11. Developers want to run different operating systems, and manage them

    adhoc Gareth Rushgrove
  12. Local operating systems are often different to the target production

    operating system Gareth Rushgrove
  13. Modern software stacks are complex Gareth Rushgrove

  14. Not everyone can know how to manage every component Gareth

  15. Being able to run everything locally is incredibly handy Gareth

  16. Local virtual machines help, but still have a maintenance cost

    Gareth Rushgrove
  17. Gareth Rushgrove application code dependent services a computer

  18. Gareth Rushgrove application code dependent services a shared computer application

    code application code
  19. Gareth Rushgrove application code services in virtual machines a computer

  20. Gareth Rushgrove application code dependent services in containers a computer

    container runtime in a virtual machine
  21. Gareth Rushgrove application code services in containers a computer with

    a native container runtime
  22. Gareth Rushgrove Docker Machine

  23. $ docker-machine create -d virtualbox dev INFO[0000] Creating SSH key...

    INFO[0000] Creating VirtualBox VM... INFO[0007] Starting VirtualBox VM... INFO[0007] Waiting for VM to start... INFO[0041] "dev" has been created and is now the active machine. Gareth Rushgrove Create local container runtimes
  24. Gareth Rushgrove $ docker-machine ls NAME ACTIVE DRIVER STATE URL

    dev * virtualbox Running tcp://
  25. Gareth Rushgrove $ docker-machine create -d digitalocean remote INFO[0000] Creating

    SSH key... INFO[0001] Creating Digital Ocean droplet... INFO[0002] Waiting for SSH... INFO[0070] Configuring Machine... INFO[0109] "remote" has been created and is now the active machine. Create remote container runtimes
  26. Gareth Rushgrove $ docker-machine ls NAME ACTIVE DRIVER STATE URL

    dev virtualbox Running tcp:// remote * digitalocean Running tcp://
  27. Gareth Rushgrove Docker Compose

  28. Gareth Rushgrove $ cat docker-compose.yml web: build: . links: -

    db - redis ports: - "8000:8000" db: image: postgres image: redis Share workspace definitions
  29. Gareth Rushgrove $ cat docker-compose.yml web: build: . links: -

    db - redis - elasticsearch ports: - "8000:8000" db: image: postgres image: redis image: elasticsearch Easily add new services
  30. Continuous Integration

  31. Everyone wants faster build times Gareth Rushgrove

  32. Isolated builds are essential to avoid unrelated failures due to

    environment pollution Gareth Rushgrove
  33. Virtual machines solve the isolation, but add overhead to a

    test run Gareth Rushgrove
  34. Lots of hardware capacity for peak load can lead to

    utilisation issues Gareth Rushgrove
  35. Docker provides just enough isolation for many test use cases

    Gareth Rushgrove
  36. And a scheduler like Mesos can result in best possible

    utilisation Gareth Rushgrove
  37. Gareth Rushgrove Docker Jenkins plugin

  38. The aim of the docker plugin is to be able

    to use a docker host to dynamically provision a slave, run a single build, then tear-down that slave. Gareth Rushgrove https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
  39. Gareth Rushgrove Mesos Jenkins plugin

  40. The mesos-jenkins plugin allows Jenkins to dynamically launch Jenkins slaves

    on a Mesos cluster depending on the workload! Gareth Rushgrove https://wiki.jenkins-ci.org/display/JENKINS/Mesos+Plugin
  41. Gareth Rushgrove jenkins slave jenkins slave jenkins slave jenkins master

  42. Gareth Rushgrove jenkins slave jenkins slave jenkins slave jenkins master

  43. Gareth Rushgrove jenkins slave jenkins slave jenkins slave jenkins master

    Job Job Job one job per slave is no problem
  44. Gareth Rushgrove jenkins slave jenkins slave jenkins slave jenkins master

    Job Job Job but with multiple jobs per slave we can run into dependency issues
  45. Gareth Rushgrove mesos slave mesos slave mesos slave jenkins master

    Job Job Job each container is a short lived jenkins slave
  46. Gareth Rushgrove eBay CI Solution

  47. Docker is an implementation detail. You’re not directly using the

    Docker interfaces. Gareth Rushgrove
  48. A package format and a runtime

  49. 1 container per VM pattern Gareth Rushgrove

  50. Gareth Rushgrove

  51. It’s all about security domains Gareth Rushgrove

  52. Take advantage of containers in production now, while minimising downsides

    Gareth Rushgrove
  53. Operating containers requires new knowledge and new tools Gareth Rushgrove

  54. Visibility for operations Gareth Rushgrove

  55. Problems with ps Gareth Rushgrove

  56. Gareth Rushgrove $ ps aux USER PID %CPU %MEM VSZ

    RSS TTY STAT START TIME COMMAND ... 999 1807 0.2 11.4 867624 464572 ? Ssl 09:38 0:21 mysqld Is this process in a container?
  57. Gareth Rushgrove $ ps -eo ucmd,cgroup COMMAND CGROUP ... mysqld

    9:perf_event:/docker/61e76d2c39121282474ff895b9b3ba2addd775cdea6d2ba89ce76c28 Which container is that?
  58. Gareth Rushgrove Sysdig

  59. Provides a Kernel module, which hooks into cgroups and namespaces

    Gareth Rushgrove (Prediction: We’ll see more of this kind of thing)
  60. Gareth Rushgrove $ sudo sysdig -c topcontainers_cpu CPU% container.name -----------------------------------------------------------------------

    90.13% mysql 15.93% wordpress1 7.27% haproxy 3.46% wordpress2 CPU usage acros containers
  61. Gareth Rushgrove $ sudo sysdig -pc -c topprocs_cpu container.name=client CPU%

    Process container.name ---------------------------------------------- 02.69% bash client 31.04% curl client 0.74% sleep client CPU usage in a single container
  62. Gareth Rushgrove $ sudo sysdig -pc -c topprocs_net Bytes Process

    Host_pid Container_pid container.name --------------------------------------------------------------- 72.06KB haproxy 7385 13 haproxy 56.96KB docker.io 1775 7039 host 44.45KB mysqld 6995 91 mysql 44.45KB mysqld 6995 99 mysql 29.36KB apache2 7893 124 wordpress1 29.36KB apache2 26895 126 wordpress4 29.36KB apache2 26622 131 wordpress2 29.36KB apache2 27935 132 wordpress3 29.36KB apache2 27306 125 wordpress4 22.23KB mysqld 6995 90 mysqlclient Network bandwidth
  63. Gareth Rushgrove $ sudo sysdig -pc -A -c echo_fds "fd.ip=

    and fd.ip=" ------ Write 103B to [haproxy] [d468ee81543a]> (hapr GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: Accept: */* X-Forwarded-For: ------ Read 103B from [wordpress1] [12b8c6a04031]> ( GET / HTTP/1.1 User-Agent: curl/7.35.0 Host: Accept: */* X-Forwarded-For: ------ Write 346B to [wordpress1] [12b8c6a04031]> (a HTTP/1.1 302 Found Date: Sat, 21 Feb 2015 22:19:18 GMT Traffic between containers
  64. Metadata Gareth Rushgrove

  65. Gareth Rushgrove

  66. Metadata like name, version, build time, build host, dependencies, descriptions,

    licence, signature and urls Built in logic like pre/post install scripts An API to interact with this – the rpm or apt/deb commands Gareth Rushgrove - - - https://www.devco.net/archives/2015/03/30/some-thoughts-on-operating-containers.php
  67. Gareth Rushgrove $ docker exec -ti mycontainer container --metadata|json_reformat {

    "validate_method": "/srv/support/bin/validate.sh", "start_method": "/srv/support/bin/start.sh", "update_method": "/srv/support/bin/update.sh" "validate": true, "build_cause": "TIMERTRIGGER", "build_tag": "jenkins-docker rbldnsd-55", "ci": true, "image_tag_names": [ "hub.my.net/ripienaar/rbldnsd" ], "project": "rbldnsd", "build_time": "2015-03-30 06:02:10", "build_time_stamp": 1427691730, "image_name": "ripienaar/rbldnsd", "gitref": "e1b0a445744fec5e584919711cafd8f4cebdee0e", }
  68. $ docker exec -ti rbldnsd container --examine Container first started

    at 2015-03-30 05:02:37 +0000 (1427691757) Names: Project Name: centos_base Image Name: ripienaar/centos_base Image Tag Names: hub.my.net/ripienaar/centos_base Build Info: CI Run: true Git Hash: fcb5f3c664b293c7a196c9809a33714427804d40 Build Cause: TIMERTRIGGER Build Time: 2015-03-24 03:25:01 (1427167501) Build Tag: jenkins-docker centos_base-20 Actions: START: not set UPDATE: not set VALIDATE: not set Gareth Rushgrove
  69. Community standards here would be awesome Gareth Rushgrove

  70. Base image Gareth Rushgrove

  71. Scratch images Minimal distros like busybox Full OS like Ubuntu

    or Debian Full OS with working init system Gareth Rushgrove - - - -
  72. No one right answer. Depends on context; existing tooling, application

    runtime, etc. Gareth Rushgrove
  73. Managing configuration Gareth Rushgrove

  74. Separate configuration from containers Gareth Rushgrove

  75. Retain advantages of declarative configuration management tools Gareth Rushgrove

  76. Gareth Rushgrove

  77. Gareth Rushgrove

  78. Clusters of containers

  79. Warning Beyond here lies the future Gareth Rushgrove

  80. Higher level interfaces Multi-host scheduling Health checks Load balancing Gareth

    Rushgrove - - - -
  81. Gareth Rushgrove Borg

  82. Gareth Rushgrove Kubernetes

  83. Gareth Rushgrove Mesos

  84. Gareth Rushgrove CoreOS

  85. Gareth Rushgrove

  86. Gareth Rushgrove Lattice

  87. Gareth Rushgrove Docker Swarm

  88. Gareth Rushgrove Amazon ECS

  89. A quest for an API Gareth Rushgrove

  90. Current lack of tools build atop those APIs Gareth Rushgrove

  91. Conclusions

  92. We should talk about container use cases more than about

    containers Gareth Rushgrove
  93. Different use cases probably require different tooling Gareth Rushgrove

  94. Different use cases will have different best practices Gareth Rushgrove

  95. You don’t have to use containers everywhere to see benefits

    Gareth Rushgrove
  96. Questions? And thanks for listening Gareth Rushgrove