[TrueNorth PHP 2015] Deploy Anything, Control Everything, Scale Effortlessly: Deis

Fee39f0c0ffb29d9ac21607ed188be6b?s=47 Davey Shafik
November 05, 2015

[TrueNorth PHP 2015] Deploy Anything, Control Everything, Scale Effortlessly: Deis

Deis is an open source platform for Docker. Built around the 12-factor app methodology, Deis allows you to create your own Docker container based platform for application deployment.

In this talk we will look at the 12-factor architecture and how to setup and deploy using Docker and Deis.


Davey Shafik

November 05, 2015


  1. D E P LOY I N G D O C

    K E R TO T H E C LO U D : D E I S CC-BY 2.0:
  2. D AV E Y S H A F I K

    • Author of Zend PHP 5 Certification Study Guide, Sitepoints PHP Anthology: 101 Essential Tips, Tricks & Hacks & PHP Master: Write Cutting Edge Code • A contributor to Zend Framework 1 & 2, phpdoc, & PHP internals • Original creator of PHAR/ PHP_Archive • @dshafik
  3. h tt p : / /d e v e l

    o p e r. a ka m a i .co m
  4. Let’s&start&a&conversation&about& mental&health&in&tech mhprompt.org/support

  5. AG E N DA CC-BY 2.0:

  6. D E P LOY I N G D O C

    K E R TO T H E C LO U D : D E I S • ~3 hour tutorial (hopefully) • Docker intro/overview • Hands on: Build your own Docker images • Hands on: Deploy Deis to AWS • Hands on: Deploy applications to Deis • 12-Factor App Methodology Review
  7. R E Q U I R E M E N

    TS CC-BY-SA 2.0:
  8. I N T E R N E T

  9. A N A M A ZO N A W S

    ACCO U N T ( W I T H C LO U D F O R M AT I O N A CC E SS )
  10. A C O M P U T E R

  11. P L E A S E A S K Q

    U E ST I O N S
  12. D O C K E R : CO N TA

    I N A L L T H E T H I N G S CC-BY 2.0:
  13. W H AT I S D O C K E

    R ?
  14. “Docker is a tool that can package an application and

    its dependencies in a virtual container that can run on any Linux server. This helps enable flexibility and portability on where the application can run, whether on premise, public cloud, private cloud, bare metal, etc.” S O U R C E : 4 5 1 R E S E A R C H ( E M P H A S I S M I N E )
  15. S O U R C E : W I K

    I P E D I A “[Docker] automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating-system-level virtualization on Linux.”
  16. W H AT I S D O C K E

    R ? • Docker is not the container technology • Docker is an abstraction and automation framework for deploying applications on Linux containers (LXC) • Provides process isolation (sandboxing) • Does not require a virtualized environment, runs on the host OS
  17. W H AT I S D O C K E

    R ?
  18. W H AT I S D O C K E

    R ? Server (Real or Virtual)
  19. W H AT I S D O C K E

    R ? Host OS (Linux) Server (Real or Virtual)
  20. W H AT I S D O C K E

    R ? Host OS (Linux) Server (Real or Virtual) Docker Daemon
  21. W H AT I S D O C K E

    R ? Container Host OS (Linux) Server (Real or Virtual) Docker Daemon binaries/libs Container binaries/libs Container binaries/libs Container binaries/libs
  22. D O C K E R O N M A

    C O S X / W I N D O W S • docker machine • Lightweight Linux distro for running Docker in a VM • 27MB
  23. D O C K E R I M A G

    E S L I K E A N O N I O N : I T H A S L A Y E R S CC-BY-SA:
  24. U N I O N FS : L AY E

    R E D I M A G E S references parent image { readonly writable
  25. E XT E N D I N G I M

    A G E S • You can build an image from scratch: don’t • Extend from a base image • Ubuntu, Debian • CentOS, RHEL, Fedora • ArchLinux • OpenSUSE • Gentoo • CoreOS
  26. CO R E O S CC-BY-ND 2.0:

  27. CO R E O S • Minimal Distro (based on

    Gentoo) • Automatic Updates (Atomic + Rollbacks) • Container Support • Cluster Management (fleet) • Service Discovery (etcd) • Everything is a service, accessed via an API
  28. F L E E T • Manages Container • Systemd

    for the cluster • Schedules tasks automatically • Resolving conflicts • Automatically handles machine failure
  29. E TC D • Key-Value Store • Handles service discovery

    • Configuration Storage • Guaranteed Consistency • Useful for implementing things like distributed locking
  30. OT H E R TO O LS • Flannel: Container

    Networking Layer • Rkt: CoreOS backed container format (alternative to Docker) • Locksmith: Reboot Manager, allows you to smartly reboot segments of a cluster and ensure zero interruptions • Many more…
  31. B U I L D I N G A N

    I M A G E CC-BY 2.0:
  32. B U I L D I N G A N

    I M A G E • Create a Dockerfile • Have Docker Hub build it for you by linking to a Github/Bitbucket repo • Build it locally • Build it on deploy with Deis
  33. D O C K E R H U B CC-BY

  34. D O C K E R H U B •

    Github for Docker Images • Sign up with Github (or with bespoke credentials) • Supports organizations • Private images (one free) • Automatic builds on push to Github/Bitbucket • Images: <username or organization>/<image>
  35. D O C K E R F I L E

    E X A M P L E : M E M C A C H E D FROM ubuntu:wily MAINTAINER Davey Shafik <dshafik@akamai.com> RUN apt-get update -qq RUN apt-get install -q -y memcached CMD ["memcached", "-u", "daemon"] EXPOSE 11211
  36. D O C K E R F I L E

    E X A M P L E : M E M C A C H E D • Must start with FROM (first non-comment), defines the base image • Creates images after each step as required • Caches and will re-use any step that it can • The container will continue running for as long as the CMD is running the foreground. Will only run the last CMD • With Deis, you may only EXPOSE one port
  37. D E M O CC-BY 2.0:

  38. None
  39. None
  40. B U I L D CO N T E XT

    CC-BY-SA 2.0:
  41. CO N T E XT • The entire CWD is

    available to the Dockerfile: This is the build Context • Use .dockerignore file to ignore files in the CWD. Users Go’s filepath.Match pattern matching • Use WORKDIR to change CWD • Use ADD to add additional files, directories, or remote files • ADD <src> <dest> • # Required for paths with whitespace
 ADD ["src", "dest"] • Supports wildcards
  42. R U N N I N G CO M M

    A N D S CC-BY-SA 2.0:
  43. R U N N I N G CO M M

    A N D S • RUN: Run commands to build the final container image • CMD: The default process, or arguments the container is going to run when run • ENTRYPOINT: A default command to which default arguments from CMD, or those passed in via docker run, are passed. • Relative to the WORKDIR • Runs as root unless changed with USER
  44. R U N N I N G CO M M

    A N D S • All three take two forms (at least): • exec form: • shell form: • CMD also takes just arguments to pass to the ENTRYPOINT: • • exec and param form do not perform shell interpolation of params (e.g. $USER or `hostname`) ["executable", "param1", "param…"] executable param1 param… ["param1", "param…"]
  45. R U N N I N G CO M M

    A N D S : R U N RUN apt-get install -y memcached RUN ["apt-get", "install", "-y", "memcached"] # This is NOT the same: 
 RUN ["apt-get", "install -y memcached"]
  46. R U N N I N G CO M M

    A N D S : C M D CMD memcached -u daemon CMD ["memcached", "-u", "daemon"]
  47. R U N N I N G CO M M

    A N D S : E N T R Y P O I N T ENTRYPOINT memcached CMD ["-u", "daemon"] ENTRYPOINT memcached $ docker run -u daemon $ docker exec -u daemon -p 11212
  48. D E F E R R E D CO M

    M A N D S
  49. D E F E R R E D CO M

    M A N D S • Commands to run when using the image as the base for another image • Allows you to call any other Dockerfile instruction (some may not make sense however) • For example: the base ubuntu image could ensure that apt-get update is always run whenever you build upon that base image. • ONBUILD RUN apt-get update -qq
  50. CO PY I N G F I L E S

  51. CO PY I N G F I L E S

    • Similar to ADD but instead of adding files to the context, it copies it from the context into the resulting image • Two syntaxes: • COPY <src> <dest> • COPY ["src", "dest"] • Supports wildcards • Relative to the WORKDIR
  52. S H A R I N G F I L

    E S CC-BY 2.0:
  53. S H A R I N G F I L

    E S • Volumes create a mount point within the container • Volumes are shared with the host, or other containers • Set at runtime • Files created within the VOLUME path prior to running are copied over to the mounted share at runtime
  54. M E TA - DATA CC-BY:

  55. M E TA - D ATA • Associate meta-data using

    LABEL • Each LABEL creates a new image! • LABEL version="1.0" • Read meta-data using docker inspect
  56. R U N N I N G A CO N

    TA I N E R CC-BY 2.0:
  57. D E M O CC-BY-SA 2.0:

  58. D E M O • docker run -d -p 11211:11211

    dshafik/memcached • -d: daemonizes the container • -p: bind container and host port • <image>: the image to launch • docker ps: shows currently running containers • telnet <host> 11211: telnet to the mecached daemon • docker stop <hash or name>: stop the container
  59. None
  60. D E M O CC-BY-SA 2.0:

  61. D E M O • docker run -d -P dshafik/memcached

    • -P: bind all container ports to random host ports • docker port <hash>
  62. P O RTS CC-BY-SA 2.0:

  63. P O RTS • EXPOSE: In the Dockerfile • --expose

    with docker run (useful for with custom run commands) • Bind to host: • -p: bind host port to container port: -p <host>:<container> • -P: bind all exposed ports to a random ports on the host • Find ports: docker port <container> <container port>
  64. L I N K I N G CO N TA

    I N E RS CC-BY 2.0:
  65. L I N K I N G CO N TA

    I N E RS • Intra-Container Communication (TCP and/or UDP) • Linked by container name • Sets ENVironment variables and • Updates /etc/hosts file • Doesn’t require ports be exposed to the outside (e.g. using -p or - P)
  66. L I N K I N G CO N TA

    I N E RS $ docker run -d -P --name <name> <image> $ docker run -d -P --link <name>:<alias> <image>
  67. L I N K I N G CO N TA

    I N E RS • Exposes all ENV vars from source container • Creates ENV vars: - <alias>_PORT_<port>_<protocol>_ADDR = <IP> - <alias>_PORT_<port>_<protocol>_PORT=<port> - <alias>_PORT_<port>_<protocol>_PROTO=<protocol> - <alias>_PORT=<first EXPOSEd port> - <alias>_ENV_<environment vars> = <value> • Add <alias> to hosts file: ping <alias>: <container IP>
  68. S H A R I N G I M AG

    E S CC-BY-SA 2.0:
  69. S H A R I N G I M A

    G E S • Using docker hub • docker push <image> • docker pull <image> • Without docker hub • docker save -o <image>.tar <image> • docker load -i <image>.tar
  70. D E I S

  71. D E I S • Build Your Own Heroku •

    Supports Heroku Buildpacks • Docker-based PaaS management • 100% Open Source • Community First Project • Maintained by Engine Yard
  72. h tt p : / /d s h a f

    i k . g i t h u b . i o /d e i s - d o c ke r- w o r k s h o p
  73. T W E LV E FACTO R : A P

    P S FO R T H E C LO U D 1 2 CC-BY 2.0:
  74. W H AT I S 1 2 FA CTO R

    ? CC-BY 2.0:
  75. F R O M T H E F I N

    E FO L K S AT
  76. 1 2 FA CTO R A P P S •

    Declarative Setup and Configuration • Clean Contracts Between Services • Scalability • Minimal Divergence Between Environments
  77. P U T A N OT H E R W

    AY… • Easy to Deploy Anywhere • Easy to Scale • Easy to Maintain • Easy to Develop
  78. S H A R E D N O T H

    I N G
  79. S H A R E D N OT H I

    N G • Any time you share resources, scaling is hard • Shared Nothing tries to eliminate any shared resources • At worst, it puts shared resources in appropriate, easily scaled data stores • e.g. sessions in memcached, search in ElasticSearch
  80. 1 . CO D E B A S E

  81. Ƙ Codebase Dev Staging QA Production Deploys

  82. 1 . CO D E B A S E •

    One codebase, many deployments • Dev, Staging, QA, Production • Multiple codebases = distributed system • Shared code should be in re-usable libraries • Micro-services architecture good • Each service must itself adhere to 12-factor! • Environments should be functionally identical
  83. 2 . D E P E N D E N

    C I E S CC-BY 2.0:
  84. 2 . D E P E N D E N

    C I E S • Never rely on system-wide packages (e.g. gems) • Always vendor dependencies, including system tools (e.g. imagick) • Lock files are your friend • Package Managers: • Ruby: bundler • PHP: composer • Node.js: npm • Python: pip
  85. 3 . CO N F I G CC-BY 2.0:

  86. 3 . CO N F I G • Configuration is

    variables that will change between deployments (dev/qa/staging/prod) • Database credentials • Service credentials • Service locations • Stored in Environment Variables • Language/OS agnostic • Easy to change • You won’t accidentally commit things like passwords into RCS
  87. 4 . B A C K I N G S

    E R V I C E S CC-BY-ND 2.0:
  88. Ȑ

  89.  Database Ȑ

  90.   Database Akamai NetStorage Ȑ

  91.    Database Akamai NetStorage Twitter Ȑ

  92. 4 . B A C K I N G S

    E R V I C E S • All resources are treated as services • Treat local and third-party services identically • All resources should be accessed via an API, making them an implementation detail that can be switched out • Service Locations and Credentials stored in the shared config
  93. 5 . B U I L D , R E

    L E A S E , R U N CC-BY 2.0:
  94. Ƙ Codebase Build Config + =

  95. Ƙ Codebase Build Config + Release =

  96. 5 . B U I L D , R E

    L E A S E , R U N • The Build performs tasks to create a distributable version of the code base • Bundle dependencies • Compiles binaries • Builds assets • The Release combines the Build with the Config • The Run is the actual execution of the release in it’s intended environment
  97. 6 . P R O C E SS E S

    CC-BY 2.0:
  98. 6 . P R O C E SS E S

    • Execute the app as one or more stateless processes • Any data that needs to persist must be stored in a stateful backing service • Memory and local FS may be used as a brief, single transaction cache (such as handling large files) • Dependencies are incorporated during the build, rather than on deploy • Sticky sessions are a violation of 12-factor. Use a shared session store.
  99. 7 . P O RT B I N D I

    N G CC-BY 2.0:
  100. 7 . P O RT B I N D I

    N G • All applications should expose themselves via userland HTTP • This allows for a completely self-contained application • I strongly disagree with this! • Great for dev, often terrible for production (particularly in PHP) • Doesn’t work with middle-ware type stacks, like WSGI and PSR-7
  101. CC-BY: Alosh Bennett 8 . CO N CU R R

    E N C Y
  102. processor.1 translator.1 web.1 web.2 processor.2 translator.2 translator.3 translator.4 Scale

    processes) Workload Diversity (process types) web.3
  103. 8 . CO N CU R R E N C

    Y • Applications should scale by spawning more processes • This enables easy horizontal scaling • Shared-nothing makes this easy • By using micro services and different process types we can easily scale different workloads with different resource needs
  104. 9 . D I S P O S A B

    I L I TY CC-BY 2.0:
  105. 9 . D I S P O S A B

    I L I TY • All processes should be disposable — this helps with rapid deployment • All processes should gracefully handle SIGTERM (e.g. return job to the queue) • Handling of unexpected death should be handled when possible • All jobs should ideally be idempotent, and must be reentrant — running them again after death should be possible • Processes should minimize startup time
  106. 1 0 . D E V/ P R O D

    U CT I O N PA R I TY CC-BY-SA 2.0:
  107. 1 0 . D E V/ P R O D

    U CT I O N PA R I TY • Minimize differences between development and production environments • Time: Quick deploys ensures that the codebases diverge minimally • Personnel: Code authors should also be deploying it • Tools: Use the same tools in both systems, do not use lightweight alternatives (e.g. SQLite) or different OSes (use a VM if necessary)
  108. 1 1 . LO G S CC-BY-SA 2.0:

  109. 1 1 . LO G S • Logs should be

    treated as an event stream • All processes should emit logs to STDOUT • During development, a developer should have these visible in the foreground • In other deploys the environment is responsible for collating these all together into a single cohesive event stream • Routed to one or more final destinations (file, network storage, splunk)
  110. 1 2 . A D M I N P R

    O C E SS E S CC-BY-SA 2.0:
  111. 1 2 . A D M I N P R

    O C E SS E S • Admin and management processes (such as DB migrations) should be run as one-off processes. • Run in a duplicate environment with the same config (so that it points to the same data sources) • Run against a specific release • Admin processes must ship the release code alongside to avoid sync issues
  112. F E E D B AC K & Q U

    E ST I O N S Feedback: Twitter: Email: Slides: https://joind.in/ @dshafik dshafik@akamai.com http://daveyshafik.com/slides 15739