[TrueNorth PHP 2015] Deploy Anything, Control Everything, Scale Effortlessly: Deis

Deis is an open source platform for Docker. Built around the 12-factor app methodology, Deis allows you to create your own Docker container based platform for application deployment.

In this talk we will look at the 12-factor architecture and how to setup and deploy using Docker and Deis.

Transcript

1. D E P LOY I N G D O C K E R TO T H E C LO U D : D E I S CC-BY 2.0:
2. D AV E Y S H A F I K • Author of Zend PHP 5 Certification Study Guide, Sitepoints PHP Anthology: 101 Essential Tips, Tricks & Hacks & PHP Master: Write Cutting Edge Code • A contributor to Zend Framework 1 & 2, phpdoc, & PHP internals • Original creator of PHAR/ PHP_Archive • @dshafik
3. h tt p : / /d e v e l o p e r. a ka m a i .co m
4. Let’s&start&a&conversation&about& mental&health&in&tech mhprompt.org/support
5. AG E N DA CC-BY 2.0:
6. D E P LOY I N G D O C K E R TO T H E C LO U D : D E I S • ~3 hour tutorial (hopefully) • Docker intro/overview • Hands on: Build your own Docker images • Hands on: Deploy Deis to AWS • Hands on: Deploy applications to Deis • 12-Factor App Methodology Review
7. R E Q U I R E M E N TS CC-BY-SA 2.0:
8. I N T E R N E T
9. A N A M A ZO N A W S ACCO U N T ( W I T H C LO U D F O R M AT I O N A CC E SS )
10. A C O M P U T E R
11. P L E A S E A S K Q U E ST I O N S
12. D O C K E R : CO N TA I N A L L T H E T H I N G S CC-BY 2.0:
13. W H AT I S D O C K E R ?
14. “Docker is a tool that can package an application and its dependencies in a virtual container that can run on any Linux server. This helps enable flexibility and portability on where the application can run, whether on premise, public cloud, private cloud, bare metal, etc.” S O U R C E : 4 5 1 R E S E A R C H ( E M P H A S I S M I N E )
15. S O U R C E : W I K I P E D I A “[Docker] automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating-system-level virtualization on Linux.”
16. W H AT I S D O C K E R ? • Docker is not the container technology • Docker is an abstraction and automation framework for deploying applications on Linux containers (LXC) • Provides process isolation (sandboxing) • Does not require a virtualized environment, runs on the host OS
17. W H AT I S D O C K E R ?
18. W H AT I S D O C K E R ? Server (Real or Virtual)
19. W H AT I S D O C K E R ? Host OS (Linux) Server (Real or Virtual)
20. W H AT I S D O C K E R ? Host OS (Linux) Server (Real or Virtual) Docker Daemon
21. W H AT I S D O C K E R ? Container Host OS (Linux) Server (Real or Virtual) Docker Daemon binaries/libs Container binaries/libs Container binaries/libs Container binaries/libs
22. D O C K E R O N M A C O S X / W I N D O W S • docker machine • Lightweight Linux distro for running Docker in a VM • 27MB
23. D O C K E R I M A G E S L I K E A N O N I O N : I T H A S L A Y E R S CC-BY-SA:
24. U N I O N FS : L AY E R E D I M A G E S references parent image { readonly writable
25. E XT E N D I N G I M A G E S • You can build an image from scratch: don’t • Extend from a base image • Ubuntu, Debian • CentOS, RHEL, Fedora • ArchLinux • OpenSUSE • Gentoo • CoreOS
26. CO R E O S CC-BY-ND 2.0:
27. CO R E O S • Minimal Distro (based on Gentoo) • Automatic Updates (Atomic + Rollbacks) • Container Support • Cluster Management (fleet) • Service Discovery (etcd) • Everything is a service, accessed via an API
28. F L E E T • Manages Container • Systemd for the cluster • Schedules tasks automatically • Resolving conflicts • Automatically handles machine failure
29. E TC D • Key-Value Store • Handles service discovery • Configuration Storage • Guaranteed Consistency • Useful for implementing things like distributed locking
30. OT H E R TO O LS • Flannel: Container Networking Layer • Rkt: CoreOS backed container format (alternative to Docker) • Locksmith: Reboot Manager, allows you to smartly reboot segments of a cluster and ensure zero interruptions • Many more…
31. B U I L D I N G A N I M A G E CC-BY 2.0:
32. B U I L D I N G A N I M A G E • Create a Dockerfile • Have Docker Hub build it for you by linking to a Github/Bitbucket repo • Build it locally • Build it on deploy with Deis
33. D O C K E R H U B CC-BY 2.0:
34. D O C K E R H U B • Github for Docker Images • Sign up with Github (or with bespoke credentials) • Supports organizations • Private images (one free) • Automatic builds on push to Github/Bitbucket • Images: <username or organization>/<image>
35. D O C K E R F I L E E X A M P L E : M E M C A C H E D FROM ubuntu:wily MAINTAINER Davey Shafik <dshafik@akamai.com> RUN apt-get update -qq RUN apt-get install -q -y memcached CMD ["memcached", "-u", "daemon"] EXPOSE 11211
36. D O C K E R F I L E E X A M P L E : M E M C A C H E D • Must start with FROM (first non-comment), defines the base image • Creates images after each step as required • Caches and will re-use any step that it can • The container will continue running for as long as the CMD is running the foreground. Will only run the last CMD • With Deis, you may only EXPOSE one port
37. D E M O CC-BY 2.0:
40. B U I L D CO N T E XT CC-BY-SA 2.0:
41. CO N T E XT • The entire CWD is available to the Dockerfile: This is the build Context • Use .dockerignore file to ignore files in the CWD. Users Go’s filepath.Match pattern matching • Use WORKDIR to change CWD • Use ADD to add additional files, directories, or remote files • ADD <src> <dest> • # Required for paths with whitespace
 ADD ["src", "dest"] • Supports wildcards
42. R U N N I N G CO M M A N D S CC-BY-SA 2.0:
43. R U N N I N G CO M M A N D S • RUN: Run commands to build the final container image • CMD: The default process, or arguments the container is going to run when run • ENTRYPOINT: A default command to which default arguments from CMD, or those passed in via docker run, are passed. • Relative to the WORKDIR • Runs as root unless changed with USER
44. R U N N I N G CO M M A N D S • All three take two forms (at least): • exec form: • shell form: • CMD also takes just arguments to pass to the ENTRYPOINT: • • exec and param form do not perform shell interpolation of params (e.g. $USER or `hostname`) ["executable", "param1", "param…"] executable param1 param… ["param1", "param…"]
45. R U N N I N G CO M M A N D S : R U N RUN apt-get install -y memcached RUN ["apt-get", "install", "-y", "memcached"] # This is NOT the same: 
 RUN ["apt-get", "install -y memcached"]
46. R U N N I N G CO M M A N D S : C M D CMD memcached -u daemon CMD ["memcached", "-u", "daemon"]
47. R U N N I N G CO M M A N D S : E N T R Y P O I N T ENTRYPOINT memcached CMD ["-u", "daemon"] ENTRYPOINT memcached $ docker run -u daemon $ docker exec -u daemon -p 11212
48. D E F E R R E D CO M M A N D S
49. D E F E R R E D CO M M A N D S • Commands to run when using the image as the base for another image • Allows you to call any other Dockerfile instruction (some may not make sense however) • For example: the base ubuntu image could ensure that apt-get update is always run whenever you build upon that base image. • ONBUILD RUN apt-get update -qq
50. CO PY I N G F I L E S CC-BY-SA:
51. CO PY I N G F I L E S • Similar to ADD but instead of adding files to the context, it copies it from the context into the resulting image • Two syntaxes: • COPY <src> <dest> • COPY ["src", "dest"] • Supports wildcards • Relative to the WORKDIR
52. S H A R I N G F I L E S CC-BY 2.0:
53. S H A R I N G F I L E S • Volumes create a mount point within the container • Volumes are shared with the host, or other containers • Set at runtime • Files created within the VOLUME path prior to running are copied over to the mounted share at runtime
54. M E TA - DATA CC-BY:
55. M E TA - D ATA • Associate meta-data using LABEL • Each LABEL creates a new image! • LABEL version="1.0" • Read meta-data using docker inspect
56. R U N N I N G A CO N TA I N E R CC-BY 2.0:
57. D E M O CC-BY-SA 2.0:
58. D E M O • docker run -d -p 11211:11211 dshafik/memcached • -d: daemonizes the container • -p: bind container and host port • <image>: the image to launch • docker ps: shows currently running containers • telnet <host> 11211: telnet to the mecached daemon • docker stop <hash or name>: stop the container
60. D E M O CC-BY-SA 2.0:
61. D E M O • docker run -d -P dshafik/memcached • -P: bind all container ports to random host ports • docker port <hash>
62. P O RTS CC-BY-SA 2.0:
63. P O RTS • EXPOSE: In the Dockerfile • --expose with docker run (useful for with custom run commands) • Bind to host: • -p: bind host port to container port: -p <host>:<container> • -P: bind all exposed ports to a random ports on the host • Find ports: docker port <container> <container port>
64. L I N K I N G CO N TA I N E RS CC-BY 2.0:
65. L I N K I N G CO N TA I N E RS • Intra-Container Communication (TCP and/or UDP) • Linked by container name • Sets ENVironment variables and • Updates /etc/hosts file • Doesn’t require ports be exposed to the outside (e.g. using -p or - P)
66. L I N K I N G CO N TA I N E RS $ docker run -d -P --name <name> <image> $ docker run -d -P --link <name>:<alias> <image>
67. L I N K I N G CO N TA I N E RS • Exposes all ENV vars from source container • Creates ENV vars: - <alias>_PORT_<port>_<protocol>_ADDR = <IP> - <alias>_PORT_<port>_<protocol>_PORT=<port> - <alias>_PORT_<port>_<protocol>_PROTO=<protocol> - <alias>_PORT=<first EXPOSEd port> - <alias>_ENV_<environment vars> = <value> • Add <alias> to hosts file: ping <alias>: <container IP>
68. S H A R I N G I M AG E S CC-BY-SA 2.0:
69. S H A R I N G I M A G E S • Using docker hub • docker push <image> • docker pull <image> • Without docker hub • docker save -o <image>.tar <image> • docker load -i <image>.tar
70. D E I S
71. D E I S • Build Your Own Heroku • Supports Heroku Buildpacks • Docker-based PaaS management • 100% Open Source • Community First Project • Maintained by Engine Yard
72. h tt p : / /d s h a f i k . g i t h u b . i o /d e i s - d o c ke r- w o r k s h o p
73. T W E LV E FACTO R : A P P S FO R T H E C LO U D 1 2 CC-BY 2.0:
74. W H AT I S 1 2 FA CTO R ? CC-BY 2.0:
75. F R O M T H E F I N E FO L K S AT
76. 1 2 FA CTO R A P P S • Declarative Setup and Configuration • Clean Contracts Between Services • Scalability • Minimal Divergence Between Environments
77. P U T A N OT H E R W AY… • Easy to Deploy Anywhere • Easy to Scale • Easy to Maintain • Easy to Develop
78. S H A R E D N O T H I N G
79. S H A R E D N OT H I N G • Any time you share resources, scaling is hard • Shared Nothing tries to eliminate any shared resources • At worst, it puts shared resources in appropriate, easily scaled data stores • e.g. sessions in memcached, search in ElasticSearch
80. 1 . CO D E B A S E
81. Ƙ Codebase Dev Staging QA Production Deploys
82. 1 . CO D E B A S E • One codebase, many deployments • Dev, Staging, QA, Production • Multiple codebases = distributed system • Shared code should be in re-usable libraries • Micro-services architecture good • Each service must itself adhere to 12-factor! • Environments should be functionally identical
83. 2 . D E P E N D E N C I E S CC-BY 2.0:
84. 2 . D E P E N D E N C I E S • Never rely on system-wide packages (e.g. gems) • Always vendor dependencies, including system tools (e.g. imagick) • Lock files are your friend • Package Managers: • Ruby: bundler • PHP: composer • Node.js: npm • Python: pip
85. 3 . CO N F I G CC-BY 2.0:
86. 3 . CO N F I G • Configuration is variables that will change between deployments (dev/qa/staging/prod) • Database credentials • Service credentials • Service locations • Stored in Environment Variables • Language/OS agnostic • Easy to change • You won’t accidentally commit things like passwords into RCS
87. 4 . B A C K I N G S E R V I C E S CC-BY-ND 2.0:
88. Ȑ
89.  Database Ȑ
90.   Database Akamai NetStorage Ȑ
91.    Database Akamai NetStorage Twitter Ȑ
92. 4 . B A C K I N G S E R V I C E S • All resources are treated as services • Treat local and third-party services identically • All resources should be accessed via an API, making them an implementation detail that can be switched out • Service Locations and Credentials stored in the shared config
93. 5 . B U I L D , R E L E A S E , R U N CC-BY 2.0:
94. Ƙ Codebase Build Config + =
95. Ƙ Codebase Build Config + Release =
96. 5 . B U I L D , R E L E A S E , R U N • The Build performs tasks to create a distributable version of the code base • Bundle dependencies • Compiles binaries • Builds assets • The Release combines the Build with the Config • The Run is the actual execution of the release in it’s intended environment
97. 6 . P R O C E SS E S CC-BY 2.0:
98. 6 . P R O C E SS E S • Execute the app as one or more stateless processes • Any data that needs to persist must be stored in a stateful backing service • Memory and local FS may be used as a brief, single transaction cache (such as handling large files) • Dependencies are incorporated during the build, rather than on deploy • Sticky sessions are a violation of 12-factor. Use a shared session store.
99. 7 . P O RT B I N D I N G CC-BY 2.0:
100. 7 . P O RT B I N D I N G • All applications should expose themselves via userland HTTP • This allows for a completely self-contained application • I strongly disagree with this! • Great for dev, often terrible for production (particularly in PHP) • Doesn’t work with middle-ware type stacks, like WSGI and PSR-7
101. CC-BY: Alosh Bennett 8 . CO N CU R R E N C Y
102. processor.1 translator.1 web.1 web.2 processor.2 translator.2 translator.3 translator.4 Scale
 (running processes) Workload Diversity (process types) web.3
103. 8 . CO N CU R R E N C Y • Applications should scale by spawning more processes • This enables easy horizontal scaling • Shared-nothing makes this easy • By using micro services and different process types we can easily scale different workloads with different resource needs
104. 9 . D I S P O S A B I L I TY CC-BY 2.0:
105. 9 . D I S P O S A B I L I TY • All processes should be disposable — this helps with rapid deployment • All processes should gracefully handle SIGTERM (e.g. return job to the queue) • Handling of unexpected death should be handled when possible • All jobs should ideally be idempotent, and must be reentrant — running them again after death should be possible • Processes should minimize startup time
106. 1 0 . D E V/ P R O D U CT I O N PA R I TY CC-BY-SA 2.0:
107. 1 0 . D E V/ P R O D U CT I O N PA R I TY • Minimize differences between development and production environments • Time: Quick deploys ensures that the codebases diverge minimally • Personnel: Code authors should also be deploying it • Tools: Use the same tools in both systems, do not use lightweight alternatives (e.g. SQLite) or different OSes (use a VM if necessary)
108. 1 1 . LO G S CC-BY-SA 2.0:
109. 1 1 . LO G S • Logs should be treated as an event stream • All processes should emit logs to STDOUT • During development, a developer should have these visible in the foreground • In other deploys the environment is responsible for collating these all together into a single cohesive event stream • Routed to one or more final destinations (file, network storage, splunk)
110. 1 2 . A D M I N P R O C E SS E S CC-BY-SA 2.0:
111. 1 2 . A D M I N P R O C E SS E S • Admin and management processes (such as DB migrations) should be run as one-off processes. • Run in a duplicate environment with the same config (so that it points to the same data sources) • Run against a specific release • Admin processes must ship the release code alongside to avoid sync issues
112. F E E D B AC K & Q U E ST I O N S Feedback: Twitter: Email: Slides: https://joind.in/ @dshafik dshafik@akamai.com http://daveyshafik.com/slides 15739