[TrueNorth PHP 2015] Deploy Anything, Control Everything, Scale Effortlessly: Deis

Slide 1

Slide 1 text

D E P LOY I N G D O C K E R TO T H E C LO U D : D E I S CC-BY 2.0:

Slide 2

Slide 2 text

D AV E Y S H A F I K • Author of Zend PHP 5 Certification Study Guide, Sitepoints PHP Anthology: 101 Essential Tips, Tricks & Hacks & PHP Master: Write Cutting Edge Code • A contributor to Zend Framework 1 & 2, phpdoc, & PHP internals • Original creator of PHAR/ PHP_Archive • @dshafik

Slide 3

Slide 3 text

h tt p : / /d e v e l o p e r. a ka m a i .co m

Slide 4

Slide 4 text

Let’s&start&a&conversation&about& mental&health&in&tech mhprompt.org/support

Slide 5

Slide 5 text

AG E N DA CC-BY 2.0:

Slide 6

Slide 6 text

D E P LOY I N G D O C K E R TO T H E C LO U D : D E I S • ~3 hour tutorial (hopefully) • Docker intro/overview • Hands on: Build your own Docker images • Hands on: Deploy Deis to AWS • Hands on: Deploy applications to Deis • 12-Factor App Methodology Review

Slide 7

Slide 7 text

R E Q U I R E M E N TS CC-BY-SA 2.0:

Slide 8

Slide 8 text

I N T E R N E T

Slide 9

Slide 9 text

A N A M A ZO N A W S ACCO U N T ( W I T H C LO U D F O R M AT I O N A CC E SS )

Slide 10

Slide 10 text

A C O M P U T E R

Slide 11

Slide 11 text

P L E A S E A S K Q U E ST I O N S

Slide 12

Slide 12 text

D O C K E R : CO N TA I N A L L T H E T H I N G S CC-BY 2.0:

Slide 13

Slide 13 text

W H AT I S D O C K E R ?

Slide 14

Slide 14 text

“Docker is a tool that can package an application and its dependencies in a virtual container that can run on any Linux server. This helps enable flexibility and portability on where the application can run, whether on premise, public cloud, private cloud, bare metal, etc.” S O U R C E : 4 5 1 R E S E A R C H ( E M P H A S I S M I N E )

Slide 15

Slide 15 text

S O U R C E : W I K I P E D I A “[Docker] automates the deployment of applications inside software containers, by providing an additional layer of abstraction and automation of operating-system-level virtualization on Linux.”

Slide 16

Slide 16 text

W H AT I S D O C K E R ? • Docker is not the container technology • Docker is an abstraction and automation framework for deploying applications on Linux containers (LXC) • Provides process isolation (sandboxing) • Does not require a virtualized environment, runs on the host OS

Slide 17

Slide 17 text

W H AT I S D O C K E R ?

Slide 18

Slide 18 text

W H AT I S D O C K E R ? Server (Real or Virtual)

Slide 19

Slide 19 text

W H AT I S D O C K E R ? Host OS (Linux) Server (Real or Virtual)

Slide 20

Slide 20 text

W H AT I S D O C K E R ? Host OS (Linux) Server (Real or Virtual) Docker Daemon

Slide 21

Slide 21 text

W H AT I S D O C K E R ? Container Host OS (Linux) Server (Real or Virtual) Docker Daemon binaries/libs Container binaries/libs Container binaries/libs Container binaries/libs

Slide 22

Slide 22 text

D O C K E R O N M A C O S X / W I N D O W S • docker machine • Lightweight Linux distro for running Docker in a VM • 27MB

Slide 23

Slide 23 text

D O C K E R I M A G E S L I K E A N O N I O N : I T H A S L A Y E R S CC-BY-SA:

Slide 24

Slide 24 text

U N I O N FS : L AY E R E D I M A G E S references parent image { readonly writable

Slide 25

Slide 25 text

E XT E N D I N G I M A G E S • You can build an image from scratch: don’t • Extend from a base image • Ubuntu, Debian • CentOS, RHEL, Fedora • ArchLinux • OpenSUSE • Gentoo • CoreOS

Slide 26

Slide 26 text

CO R E O S CC-BY-ND 2.0:

Slide 27

Slide 27 text

CO R E O S • Minimal Distro (based on Gentoo) • Automatic Updates (Atomic + Rollbacks) • Container Support • Cluster Management (fleet) • Service Discovery (etcd) • Everything is a service, accessed via an API

Slide 28

Slide 28 text

F L E E T • Manages Container • Systemd for the cluster • Schedules tasks automatically • Resolving conflicts • Automatically handles machine failure

Slide 29

Slide 29 text

E TC D • Key-Value Store • Handles service discovery • Configuration Storage • Guaranteed Consistency • Useful for implementing things like distributed locking

Slide 30

Slide 30 text

OT H E R TO O LS • Flannel: Container Networking Layer • Rkt: CoreOS backed container format (alternative to Docker) • Locksmith: Reboot Manager, allows you to smartly reboot segments of a cluster and ensure zero interruptions • Many more…

Slide 31

Slide 31 text

B U I L D I N G A N I M A G E CC-BY 2.0:

Slide 32

Slide 32 text

B U I L D I N G A N I M A G E • Create a Dockerfile • Have Docker Hub build it for you by linking to a Github/Bitbucket repo • Build it locally • Build it on deploy with Deis

Slide 33

Slide 33 text

D O C K E R H U B CC-BY 2.0:

Slide 34

Slide 34 text

D O C K E R H U B • Github for Docker Images • Sign up with Github (or with bespoke credentials) • Supports organizations • Private images (one free) • Automatic builds on push to Github/Bitbucket • Images: /

Slide 35

Slide 35 text

D O C K E R F I L E E X A M P L E : M E M C A C H E D FROM ubuntu:wily MAINTAINER Davey Shafik RUN apt-get update -qq RUN apt-get install -q -y memcached CMD ["memcached", "-u", "daemon"] EXPOSE 11211

Slide 36

Slide 36 text

D O C K E R F I L E E X A M P L E : M E M C A C H E D • Must start with FROM (first non-comment), defines the base image • Creates images after each step as required • Caches and will re-use any step that it can • The container will continue running for as long as the CMD is running the foreground. Will only run the last CMD • With Deis, you may only EXPOSE one port

Slide 37

Slide 37 text

D E M O CC-BY 2.0:

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

B U I L D CO N T E XT CC-BY-SA 2.0:

Slide 41

Slide 41 text

CO N T E XT • The entire CWD is available to the Dockerfile: This is the build Context • Use .dockerignore file to ignore files in the CWD. Users Go’s filepath.Match pattern matching • Use WORKDIR to change CWD • Use ADD to add additional files, directories, or remote files • ADD • # Required for paths with whitespace  ADD ["src", "dest"] • Supports wildcards

Slide 42

Slide 42 text

R U N N I N G CO M M A N D S CC-BY-SA 2.0:

Slide 43

Slide 43 text

R U N N I N G CO M M A N D S • RUN: Run commands to build the final container image • CMD: The default process, or arguments the container is going to run when run • ENTRYPOINT: A default command to which default arguments from CMD, or those passed in via docker run, are passed. • Relative to the WORKDIR • Runs as root unless changed with USER

Slide 44

Slide 44 text

R U N N I N G CO M M A N D S • All three take two forms (at least): • exec form: • shell form: • CMD also takes just arguments to pass to the ENTRYPOINT: • • exec and param form do not perform shell interpolation of params (e.g. $USER or `hostname`) ["executable", "param1", "param…"] executable param1 param… ["param1", "param…"]

Slide 45

Slide 45 text

R U N N I N G CO M M A N D S : R U N RUN apt-get install -y memcached RUN ["apt-get", "install", "-y", "memcached"] # This is NOT the same:   RUN ["apt-get", "install -y memcached"]

Slide 46

Slide 46 text

R U N N I N G CO M M A N D S : C M D CMD memcached -u daemon CMD ["memcached", "-u", "daemon"]

Slide 47

Slide 47 text

R U N N I N G CO M M A N D S : E N T R Y P O I N T ENTRYPOINT memcached CMD ["-u", "daemon"] ENTRYPOINT memcached $ docker run -u daemon $ docker exec -u daemon -p 11212

Slide 48

Slide 48 text

D E F E R R E D CO M M A N D S

Slide 49

Slide 49 text

D E F E R R E D CO M M A N D S • Commands to run when using the image as the base for another image • Allows you to call any other Dockerfile instruction (some may not make sense however) • For example: the base ubuntu image could ensure that apt-get update is always run whenever you build upon that base image. • ONBUILD RUN apt-get update -qq

Slide 50

Slide 50 text

CO PY I N G F I L E S CC-BY-SA:

Slide 51

Slide 51 text

CO PY I N G F I L E S • Similar to ADD but instead of adding files to the context, it copies it from the context into the resulting image • Two syntaxes: • COPY • COPY ["src", "dest"] • Supports wildcards • Relative to the WORKDIR

Slide 52

Slide 52 text

S H A R I N G F I L E S CC-BY 2.0:

Slide 53

Slide 53 text

S H A R I N G F I L E S • Volumes create a mount point within the container • Volumes are shared with the host, or other containers • Set at runtime • Files created within the VOLUME path prior to running are copied over to the mounted share at runtime

Slide 54

Slide 54 text

M E TA - DATA CC-BY:

Slide 55

Slide 55 text

M E TA - D ATA • Associate meta-data using LABEL • Each LABEL creates a new image! • LABEL version="1.0" • Read meta-data using docker inspect

Slide 56

Slide 56 text

R U N N I N G A CO N TA I N E R CC-BY 2.0:

Slide 57

Slide 57 text

D E M O CC-BY-SA 2.0:

Slide 58

Slide 58 text

D E M O • docker run -d -p 11211:11211 dshafik/memcached • -d: daemonizes the container • -p: bind container and host port • : the image to launch • docker ps: shows currently running containers • telnet 11211: telnet to the mecached daemon • docker stop : stop the container

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

D E M O CC-BY-SA 2.0:

Slide 61

Slide 61 text

D E M O • docker run -d -P dshafik/memcached • -P: bind all container ports to random host ports • docker port

Slide 62

Slide 62 text

P O RTS CC-BY-SA 2.0:

Slide 63

Slide 63 text

P O RTS • EXPOSE: In the Dockerfile • --expose with docker run (useful for with custom run commands) • Bind to host: • -p: bind host port to container port: -p : • -P: bind all exposed ports to a random ports on the host • Find ports: docker port

Slide 64

Slide 64 text

L I N K I N G CO N TA I N E RS CC-BY 2.0:

Slide 65

Slide 65 text

L I N K I N G CO N TA I N E RS • Intra-Container Communication (TCP and/or UDP) • Linked by container name • Sets ENVironment variables and • Updates /etc/hosts file • Doesn’t require ports be exposed to the outside (e.g. using -p or - P)

Slide 66

Slide 66 text

L I N K I N G CO N TA I N E RS $ docker run -d -P --name $ docker run -d -P --link :

Slide 67

Slide 67 text

L I N K I N G CO N TA I N E RS • Exposes all ENV vars from source container • Creates ENV vars: - _PORT___ADDR = - _PORT___PORT= - _PORT___PROTO= - _PORT= - _ENV_ = • Add to hosts file: ping :

Slide 68

Slide 68 text

S H A R I N G I M AG E S CC-BY-SA 2.0:

Slide 69

Slide 69 text

S H A R I N G I M A G E S • Using docker hub • docker push • docker pull • Without docker hub • docker save -o .tar • docker load -i .tar

Slide 70

Slide 70 text

D E I S

Slide 71

Slide 71 text

D E I S • Build Your Own Heroku • Supports Heroku Buildpacks • Docker-based PaaS management • 100% Open Source • Community First Project • Maintained by Engine Yard

Slide 72

Slide 72 text

h tt p : / /d s h a f i k . g i t h u b . i o /d e i s - d o c ke r- w o r k s h o p

Slide 73

Slide 73 text

T W E LV E FACTO R : A P P S FO R T H E C LO U D 1 2 CC-BY 2.0:

Slide 74

Slide 74 text

W H AT I S 1 2 FA CTO R ? CC-BY 2.0:

Slide 75

Slide 75 text

F R O M T H E F I N E FO L K S AT

Slide 76

Slide 76 text

1 2 FA CTO R A P P S • Declarative Setup and Configuration • Clean Contracts Between Services • Scalability • Minimal Divergence Between Environments

Slide 77

Slide 77 text

P U T A N OT H E R W AY… • Easy to Deploy Anywhere • Easy to Scale • Easy to Maintain • Easy to Develop

Slide 78

Slide 78 text

S H A R E D N O T H I N G

Slide 79

Slide 79 text

S H A R E D N OT H I N G • Any time you share resources, scaling is hard • Shared Nothing tries to eliminate any shared resources • At worst, it puts shared resources in appropriate, easily scaled data stores • e.g. sessions in memcached, search in ElasticSearch

Slide 80

Slide 80 text

1 . CO D E B A S E

Slide 81

Slide 81 text

Ƙ Codebase Dev Staging QA Production Deploys

Slide 82

Slide 82 text

1 . CO D E B A S E • One codebase, many deployments • Dev, Staging, QA, Production • Multiple codebases = distributed system • Shared code should be in re-usable libraries • Micro-services architecture good • Each service must itself adhere to 12-factor! • Environments should be functionally identical

Slide 83

Slide 83 text

2 . D E P E N D E N C I E S CC-BY 2.0:

Slide 84

Slide 84 text

2 . D E P E N D E N C I E S • Never rely on system-wide packages (e.g. gems) • Always vendor dependencies, including system tools (e.g. imagick) • Lock files are your friend • Package Managers: • Ruby: bundler • PHP: composer • Node.js: npm • Python: pip

Slide 85

Slide 85 text

3 . CO N F I G CC-BY 2.0:

Slide 86

Slide 86 text

3 . CO N F I G • Configuration is variables that will change between deployments (dev/qa/staging/prod) • Database credentials • Service credentials • Service locations • Stored in Environment Variables • Language/OS agnostic • Easy to change • You won’t accidentally commit things like passwords into RCS

Slide 87

Slide 87 text

4 . B A C K I N G S E R V I C E S CC-BY-ND 2.0:

Slide 88

Slide 88 text

Slide 89

Slide 89 text

 Database Ȑ

Slide 90

Slide 90 text

  Database Akamai NetStorage Ȑ

Slide 91

Slide 91 text

   Database Akamai NetStorage Twitter Ȑ

Slide 92

Slide 92 text

4 . B A C K I N G S E R V I C E S • All resources are treated as services • Treat local and third-party services identically • All resources should be accessed via an API, making them an implementation detail that can be switched out • Service Locations and Credentials stored in the shared config

Slide 93

Slide 93 text

5 . B U I L D , R E L E A S E , R U N CC-BY 2.0:

Slide 94

Slide 94 text

Ƙ Codebase Build Config + =

Slide 95

Slide 95 text

Ƙ Codebase Build Config + Release =

Slide 96

Slide 96 text

5 . B U I L D , R E L E A S E , R U N • The Build performs tasks to create a distributable version of the code base • Bundle dependencies • Compiles binaries • Builds assets • The Release combines the Build with the Config • The Run is the actual execution of the release in it’s intended environment

Slide 97

Slide 97 text

6 . P R O C E SS E S CC-BY 2.0:

Slide 98

Slide 98 text

6 . P R O C E SS E S • Execute the app as one or more stateless processes • Any data that needs to persist must be stored in a stateful backing service • Memory and local FS may be used as a brief, single transaction cache (such as handling large files) • Dependencies are incorporated during the build, rather than on deploy • Sticky sessions are a violation of 12-factor. Use a shared session store.

Slide 99

Slide 99 text

7 . P O RT B I N D I N G CC-BY 2.0:

Slide 100

Slide 100 text

7 . P O RT B I N D I N G • All applications should expose themselves via userland HTTP • This allows for a completely self-contained application • I strongly disagree with this! • Great for dev, often terrible for production (particularly in PHP) • Doesn’t work with middle-ware type stacks, like WSGI and PSR-7

Slide 101

Slide 101 text

CC-BY: Alosh Bennett 8 . CO N CU R R E N C Y

Slide 102

Slide 102 text

processor.1 translator.1 web.1 web.2 processor.2 translator.2 translator.3 translator.4 Scale  (running processes) Workload Diversity (process types) web.3

Slide 103

Slide 103 text

8 . CO N CU R R E N C Y • Applications should scale by spawning more processes • This enables easy horizontal scaling • Shared-nothing makes this easy • By using micro services and different process types we can easily scale different workloads with different resource needs

Slide 104

Slide 104 text

9 . D I S P O S A B I L I TY CC-BY 2.0:

Slide 105

Slide 105 text

9 . D I S P O S A B I L I TY • All processes should be disposable — this helps with rapid deployment • All processes should gracefully handle SIGTERM (e.g. return job to the queue) • Handling of unexpected death should be handled when possible • All jobs should ideally be idempotent, and must be reentrant — running them again after death should be possible • Processes should minimize startup time

Slide 106

Slide 106 text

1 0 . D E V/ P R O D U CT I O N PA R I TY CC-BY-SA 2.0:

Slide 107

Slide 107 text

1 0 . D E V/ P R O D U CT I O N PA R I TY • Minimize differences between development and production environments • Time: Quick deploys ensures that the codebases diverge minimally • Personnel: Code authors should also be deploying it • Tools: Use the same tools in both systems, do not use lightweight alternatives (e.g. SQLite) or different OSes (use a VM if necessary)

Slide 108

Slide 108 text

1 1 . LO G S CC-BY-SA 2.0:

Slide 109

Slide 109 text

1 1 . LO G S • Logs should be treated as an event stream • All processes should emit logs to STDOUT • During development, a developer should have these visible in the foreground • In other deploys the environment is responsible for collating these all together into a single cohesive event stream • Routed to one or more final destinations (file, network storage, splunk)

Slide 110

Slide 110 text

1 2 . A D M I N P R O C E SS E S CC-BY-SA 2.0:

Slide 111

Slide 111 text

1 2 . A D M I N P R O C E SS E S • Admin and management processes (such as DB migrations) should be run as one-off processes. • Run in a duplicate environment with the same config (so that it points to the same data sources) • Run against a specific release • Admin processes must ship the release code alongside to avoid sync issues

Slide 112

Slide 112 text

F E E D B AC K & Q U E ST I O N S Feedback: Twitter: Email: Slides: https://joind.in/ @dshafik [email protected] http://daveyshafik.com/slides 15739