[TrueNorth PHP 2015] Deploy Anything, Control Everything, Scale Effortlessly: Deis

Deis is an open source platform for Docker. Built around the 12-factor app methodology, Deis allows you to create your own Docker container based platform for application deployment.

In this talk we will look at the 12-factor architecture and how to setup and deploy using Docker and Deis.

Transcript

1. D E P LOY I N G D O C K E R TO T H E C LO U D : D E I S
   CC-BY 2.0:
   

   View Slide

2. D AV E Y S H A F I K
   • Author of Zend PHP 5 Certification
   Study Guide, Sitepoints PHP
   Anthology: 101 Essential Tips, Tricks &
   Hacks & PHP Master: Write Cutting
   Edge Code
   • A contributor to Zend Framework 1 &
   2, phpdoc, & PHP internals
   • Original creator of PHAR/
   PHP_Archive
   • @dshafik
   

   View Slide

3. h tt p : / /d e v e l o p e r. a ka m a i .co m
   

   View Slide

4. Let’s&start&a&conversation&about&
   mental&health&in&tech
   mhprompt.org/support
   

   View Slide

5. AG E N DA
   CC-BY 2.0:
   

   View Slide

6. D E P LOY I N G D O C K E R TO T H E C LO U D : D E I S
   • ~3 hour tutorial (hopefully)
   • Docker intro/overview
   • Hands on: Build your own Docker images
   • Hands on: Deploy Deis to AWS
   • Hands on: Deploy applications to Deis
   • 12-Factor App Methodology Review
   

   View Slide

7. R E Q U I R E M E N TS
   CC-BY-SA 2.0:
   

   View Slide

8. I N T E R N E T
   

   View Slide

9. A N A M A ZO N A W S ACCO U N T
   ( W I T H C LO U D F O R M AT I O N A CC E SS )
   

   View Slide

10. A C O M P U T E R
    

    View Slide

11. P L E A S E A S K Q U E ST I O N S
    

    View Slide

12. D O C K E R : CO N TA I N A L L T H E T H I N G S
    CC-BY 2.0:
    

    View Slide

13. W H AT I S D O C K E R ?
    

    View Slide

14. “Docker is a tool that can package an application and its dependencies in
    a virtual container that can run on any Linux server. This helps enable
    flexibility and portability on where the application can run, whether on
    premise, public cloud, private cloud, bare metal, etc.”
    S O U R C E : 4 5 1 R E S E A R C H
    ( E M P H A S I S M I N E )
    

    View Slide

15. S O U R C E : W I K I P E D I A
    “[Docker] automates the deployment of applications inside software
    containers, by providing an additional layer of abstraction and
    automation of operating-system-level virtualization on Linux.”
    

    View Slide

16. W H AT I S D O C K E R ?
    • Docker is not the container technology
    • Docker is an abstraction and automation framework for deploying
    applications on Linux containers (LXC)
    • Provides process isolation (sandboxing)
    • Does not require a virtualized environment, runs on the host OS
    

    View Slide

17. W H AT I S D O C K E R ?
    

    View Slide

18. W H AT I S D O C K E R ?
    Server (Real or Virtual)
    

    View Slide

19. W H AT I S D O C K E R ?
    Host OS (Linux)
    Server (Real or Virtual)
    

    View Slide

20. W H AT I S D O C K E R ?
    Host OS (Linux)
    Server (Real or Virtual)
    Docker Daemon
    

    View Slide

21. W H AT I S D O C K E R ?
    Container
    Host OS (Linux)
    Server (Real or Virtual)
    Docker Daemon
    binaries/libs
    Container
    binaries/libs
    Container
    binaries/libs
    Container
    binaries/libs
    

    View Slide

22. D O C K E R O N M A C O S X / W I N D O W S
    • docker machine
    • Lightweight Linux distro for running Docker in a
    VM
    • 27MB
    

    View Slide

23. D O C K E R I M A G E S
    L I K E A N O N I O N : I T H A S L A Y E R S
    CC-BY-SA:
    

    View Slide

24. U N I O N FS : L AY E R E D I M A G E S
    references
    parent image
    {
    readonly
    writable
    

    View Slide

25. E XT E N D I N G I M A G E S
    • You can build an image from scratch: don’t
    • Extend from a base image
    • Ubuntu, Debian
    • CentOS, RHEL, Fedora
    • ArchLinux
    • OpenSUSE
    • Gentoo
    • CoreOS
    

    View Slide

26. CO R E O S
    CC-BY-ND 2.0:
    

    View Slide

27. CO R E O S
    • Minimal Distro (based on Gentoo)
    • Automatic Updates (Atomic + Rollbacks)
    • Container Support
    • Cluster Management (fleet)
    • Service Discovery (etcd)
    • Everything is a service, accessed via an API
    

    View Slide

28. F L E E T
    • Manages Container
    • Systemd for the cluster
    • Schedules tasks automatically
    • Resolving conflicts
    • Automatically handles machine failure
    

    View Slide

29. E TC D
    • Key-Value Store
    • Handles service discovery
    • Configuration Storage
    • Guaranteed Consistency
    • Useful for implementing things like distributed locking
    

    View Slide

30. OT H E R TO O LS
    • Flannel: Container Networking Layer
    • Rkt: CoreOS backed container format (alternative to Docker)
    • Locksmith: Reboot Manager, allows you to smartly reboot
    segments of a cluster and ensure zero interruptions
    • Many more…
    

    View Slide

31. B U I L D I N G A N I M A G E
    CC-BY 2.0:
    

    View Slide

32. B U I L D I N G A N I M A G E
    • Create a Dockerfile
    • Have Docker Hub build it for you by linking to a Github/Bitbucket
    repo
    • Build it locally
    • Build it on deploy with Deis
    

    View Slide

33. D O C K E R H U B
    CC-BY 2.0:
    

    View Slide

34. D O C K E R H U B
    • Github for Docker Images
    • Sign up with Github (or with bespoke credentials)
    • Supports organizations
    • Private images (one free)
    • Automatic builds on push to Github/Bitbucket
    • Images: /
    

    View Slide

35. D O C K E R F I L E E X A M P L E : M E M C A C H E D
    FROM ubuntu:wily
    MAINTAINER Davey Shafik
    RUN apt-get update -qq
    RUN apt-get install -q -y memcached
    CMD ["memcached", "-u", "daemon"]
    EXPOSE 11211
    

    View Slide

36. D O C K E R F I L E E X A M P L E : M E M C A C H E D
    • Must start with FROM (first non-comment), defines the base image
    • Creates images after each step as required
    • Caches and will re-use any step that it can
    • The container will continue running for as long as the CMD is
    running the foreground. Will only run the last CMD
    • With Deis, you may only EXPOSE one port
    

    View Slide

37. D E M O
    CC-BY 2.0:
    

    View Slide

38. View Slide

39. View Slide

40. B U I L D CO N T E XT
    CC-BY-SA 2.0:
    

    View Slide

41. CO N T E XT
    • The entire CWD is available to the Dockerfile: This is the build Context
    • Use .dockerignore file to ignore files in the CWD. Users Go’s
    filepath.Match pattern matching
    • Use WORKDIR to change CWD
    • Use ADD to add additional files, directories, or remote files
    • ADD
    • # Required for paths with whitespace

    ADD ["src", "dest"]
    • Supports wildcards
    

    View Slide

42. R U N N I N G CO M M A N D S
    CC-BY-SA 2.0:
    

    View Slide

43. R U N N I N G CO M M A N D S
    • RUN: Run commands to build the final container image
    • CMD: The default process, or arguments the container is going to
    run when run
    • ENTRYPOINT: A default command to which default arguments from CMD, or
    those passed in via docker run, are passed.
    • Relative to the WORKDIR
    • Runs as root unless changed with USER
    

    View Slide

44. R U N N I N G CO M M A N D S
    • All three take two forms (at least):
    • exec form:
    • shell form:
    • CMD also takes just arguments to pass to the ENTRYPOINT:
    •
    • exec and param form do not perform shell interpolation of params
    (e.g. $USER or `hostname`)
    ["executable", "param1", "param…"]
    executable param1 param…
    ["param1", "param…"]
    

    View Slide

45. R U N N I N G CO M M A N D S : R U N
    RUN apt-get install -y memcached
    RUN ["apt-get", "install", "-y", "memcached"]
    # This is NOT the same: 

    RUN ["apt-get", "install -y memcached"]
    

    View Slide

46. R U N N I N G CO M M A N D S : C M D
    CMD memcached -u daemon
    CMD ["memcached", "-u", "daemon"]
    

    View Slide

47. R U N N I N G CO M M A N D S : E N T R Y P O I N T
    ENTRYPOINT memcached
    CMD ["-u", "daemon"]
    ENTRYPOINT memcached
    $ docker run -u daemon
    $ docker exec -u daemon -p 11212
    

    View Slide

48. D E F E R R E D CO M M A N D S
    

    View Slide

49. D E F E R R E D CO M M A N D S
    • Commands to run when using the image as the base for another
    image
    • Allows you to call any other Dockerfile instruction (some may not
    make sense however)
    • For example: the base ubuntu image could ensure that apt-get
    update is always run whenever you build upon that base image.
    • ONBUILD RUN apt-get update -qq
    

    View Slide

50. CO PY I N G F I L E S
    CC-BY-SA:
    

    View Slide

51. CO PY I N G F I L E S
    • Similar to ADD but instead of adding files to the context, it copies it from
    the context into the resulting image
    • Two syntaxes:
    • COPY
    • COPY ["src", "dest"]
    • Supports wildcards
    • Relative to the WORKDIR
    

    View Slide

52. S H A R I N G F I L E S
    CC-BY 2.0:
    

    View Slide

53. S H A R I N G F I L E S
    • Volumes create a mount point within the container
    • Volumes are shared with the host, or other containers
    • Set at runtime
    • Files created within the VOLUME path prior to running are copied
    over to the mounted share at runtime
    

    View Slide

54. M E TA - DATA
    CC-BY:
    

    View Slide

55. M E TA - D ATA
    • Associate meta-data using LABEL
    • Each LABEL creates a new image!
    • LABEL version="1.0"
    • Read meta-data using docker inspect
    

    View Slide

56. R U N N I N G A CO N TA I N E R
    CC-BY 2.0:
    

    View Slide

57. D E M O
    CC-BY-SA 2.0:
    

    View Slide

58. D E M O
    • docker run -d -p 11211:11211 dshafik/memcached
    • -d: daemonizes the container
    • -p: bind container and host port
    • : the image to launch
    • docker ps: shows currently running containers
    • telnet 11211: telnet to the mecached daemon
    • docker stop : stop the container
    

    View Slide

59. View Slide

60. D E M O
    CC-BY-SA 2.0:
    

    View Slide

61. D E M O
    • docker run -d -P dshafik/memcached
    • -P: bind all container ports to random host ports
    • docker port
    

    View Slide

62. P O RTS
    CC-BY-SA 2.0:
    

    View Slide

63. P O RTS
    • EXPOSE: In the Dockerfile
    • --expose with docker run (useful for with custom run commands)
    • Bind to host:
    • -p: bind host port to container port: -p :
    • -P: bind all exposed ports to a random ports on the host
    • Find ports: docker port
    

    View Slide

64. L I N K I N G CO N TA I N E RS
    CC-BY 2.0:
    

    View Slide

65. L I N K I N G CO N TA I N E RS
    • Intra-Container Communication (TCP and/or UDP)
    • Linked by container name
    • Sets ENVironment variables and
    • Updates /etc/hosts file
    • Doesn’t require ports be exposed to the outside (e.g. using -p or -
    P)
    

    View Slide

66. L I N K I N G CO N TA I N E RS
    $ docker run -d -P --name
    $ docker run -d -P --link :
    

    View Slide

67. L I N K I N G CO N TA I N E RS
    • Exposes all ENV vars from source container
    • Creates ENV vars:
    - _PORT___ADDR =
    - _PORT___PORT=
    - _PORT___PROTO=
    - _PORT=
    - _ENV_ =
    • Add to hosts file: ping : IP>
    

    View Slide

68. S H A R I N G I M AG E S
    CC-BY-SA 2.0:
    

    View Slide

69. S H A R I N G I M A G E S
    • Using docker hub
    • docker push
    • docker pull
    • Without docker hub
    • docker save -o .tar
    • docker load -i .tar
    

    View Slide

70. D E I S
    

    View Slide

71. D E I S
    • Build Your Own Heroku
    • Supports Heroku Buildpacks
    • Docker-based PaaS management
    • 100% Open Source
    • Community First Project
    • Maintained by Engine Yard
    

    View Slide

72. h tt p : / /d s h a f i k . g i t h u b . i o /d e i s - d o c ke r- w o r k s h o p
    

    View Slide

73. T W E LV E FACTO R : A P P S FO R T H E C LO U D
    1 2
    CC-BY 2.0:
    

    View Slide

74. W H AT I S 1 2 FA CTO R ?
    CC-BY 2.0:
    

    View Slide

75. F R O M T H E F I N E FO L K S AT
    

    View Slide

76. 1 2 FA CTO R A P P S
    • Declarative Setup and Configuration
    • Clean Contracts Between Services
    • Scalability
    • Minimal Divergence Between Environments
    

    View Slide

77. P U T A N OT H E R W AY…
    • Easy to Deploy Anywhere
    • Easy to Scale
    • Easy to Maintain
    • Easy to Develop
    

    View Slide

78. S H A R E D N O T H I N G
    

    View Slide

79. S H A R E D N OT H I N G
    • Any time you share resources, scaling is hard
    • Shared Nothing tries to eliminate any shared resources
    • At worst, it puts shared resources in appropriate, easily scaled
    data stores
    • e.g. sessions in memcached, search in ElasticSearch
    

    View Slide

80. 1 . CO D E B A S E
    

    View Slide

81. Ƙ
    Codebase
    Dev
    Staging
    QA
    Production
    Deploys
    

    View Slide

82. 1 . CO D E B A S E
    • One codebase, many deployments
    • Dev, Staging, QA, Production
    • Multiple codebases = distributed system
    • Shared code should be in re-usable libraries
    • Micro-services architecture good
    • Each service must itself adhere to 12-factor!
    • Environments should be functionally identical
    

    View Slide

83. 2 . D E P E N D E N C I E S
    CC-BY 2.0:
    

    View Slide

84. 2 . D E P E N D E N C I E S
    • Never rely on system-wide packages (e.g. gems)
    • Always vendor dependencies, including system tools (e.g. imagick)
    • Lock files are your friend
    • Package Managers:
    • Ruby: bundler
    • PHP: composer
    • Node.js: npm
    • Python: pip
    

    View Slide

85. 3 . CO N F I G
    CC-BY 2.0:
    

    View Slide

86. 3 . CO N F I G
    • Configuration is variables that will change between deployments
    (dev/qa/staging/prod)
    • Database credentials
    • Service credentials
    • Service locations
    • Stored in Environment Variables
    • Language/OS agnostic
    • Easy to change
    • You won’t accidentally commit things like passwords into RCS
    

    View Slide

87. 4 . B A C K I N G S E R V I C E S
    CC-BY-ND 2.0:
    

    View Slide

88. Ȑ
    

    View Slide

89. 
    Database
    Ȑ
    

    View Slide

90. 
    
    Database
    Akamai NetStorage
    Ȑ
    

    View Slide

91. 
    
    
    Database
    Akamai NetStorage
    Twitter
    Ȑ
    

    View Slide

92. 4 . B A C K I N G S E R V I C E S
    • All resources are treated as services
    • Treat local and third-party services identically
    • All resources should be accessed via an API, making them an
    implementation detail that can be switched out
    • Service Locations and Credentials stored in the shared config
    

    View Slide

93. 5 . B U I L D , R E L E A S E , R U N
    CC-BY 2.0:
    

    View Slide

94. Ƙ
    Codebase
    Build
    Config
    + =
    

    View Slide

95. Ƙ
    Codebase
    Build
    Config
    + Release
    =
    

    View Slide

96. 5 . B U I L D , R E L E A S E , R U N
    • The Build performs tasks to create a distributable version of the code
    base
    • Bundle dependencies
    • Compiles binaries
    • Builds assets
    • The Release combines the Build with the Config
    • The Run is the actual execution of the release in it’s intended
    environment
    

    View Slide

97. 6 . P R O C E SS E S
    CC-BY 2.0:
    

    View Slide

98. 6 . P R O C E SS E S
    • Execute the app as one or more stateless processes
    • Any data that needs to persist must be stored in a stateful backing
    service
    • Memory and local FS may be used as a brief, single transaction cache
    (such as handling large files)
    • Dependencies are incorporated during the build, rather than on deploy
    • Sticky sessions are a violation of 12-factor. Use a shared session store.
    

    View Slide

99. 7 . P O RT B I N D I N G
    CC-BY 2.0:
    

    View Slide

100. 7 . P O RT B I N D I N G
     • All applications should expose themselves via userland HTTP
     • This allows for a completely self-contained application
     • I strongly disagree with this!
     • Great for dev, often terrible for production (particularly in PHP)
     • Doesn’t work with middle-ware type stacks, like WSGI and PSR-7
     

     View Slide

101. CC-BY: Alosh Bennett
     8 . CO N CU R R E N C Y
     

     View Slide

102. processor.1 translator.1
     web.1
     web.2 processor.2 translator.2
     translator.3
     translator.4
     Scale

     (running processes)
     Workload Diversity
     (process types)
     web.3
     

     View Slide

103. 8 . CO N CU R R E N C Y
     • Applications should scale by spawning more processes
     • This enables easy horizontal scaling
     • Shared-nothing makes this easy
     • By using micro services and different process types we can easily
     scale different workloads with different resource needs
     

     View Slide

104. 9 . D I S P O S A B I L I TY
     CC-BY 2.0:
     

     View Slide

105. 9 . D I S P O S A B I L I TY
     • All processes should be disposable — this helps with rapid deployment
     • All processes should gracefully handle SIGTERM (e.g. return job to the
     queue)
     • Handling of unexpected death should be handled when possible
     • All jobs should ideally be idempotent, and must be reentrant
     — running them again after death should be possible
     • Processes should minimize startup time
     

     View Slide

106. 1 0 . D E V/ P R O D U CT I O N PA R I TY
     CC-BY-SA 2.0:
     

     View Slide

107. 1 0 . D E V/ P R O D U CT I O N PA R I TY
     • Minimize differences between development and production
     environments
     • Time: Quick deploys ensures that the codebases diverge minimally
     • Personnel: Code authors should also be deploying it
     • Tools: Use the same tools in both systems, do not use lightweight alternatives
     (e.g. SQLite) or different OSes (use a VM if necessary)
     

     View Slide

108. 1 1 . LO G S
     CC-BY-SA 2.0:
     

     View Slide

109. 1 1 . LO G S
     • Logs should be treated as an event stream
     • All processes should emit logs to STDOUT
     • During development, a developer should have these visible in the
     foreground
     • In other deploys the environment is responsible for collating these all
     together into a single cohesive event stream
     • Routed to one or more final destinations (file, network storage, splunk)
     

     View Slide

110. 1 2 . A D M I N P R O C E SS E S
     CC-BY-SA 2.0:
     

     View Slide

111. 1 2 . A D M I N P R O C E SS E S
     • Admin and management processes (such as DB migrations)
     should be run as one-off processes.
     • Run in a duplicate environment with the same config (so that it
     points to the same data sources)
     • Run against a specific release
     • Admin processes must ship the release code alongside to avoid
     sync issues
     

     View Slide

112. F E E D B AC K & Q U E ST I O N S
     Feedback:
     Twitter:
     Email:
     Slides:
     https://joind.in/
     @dshafik
     dshafik@akamai.com
     http://daveyshafik.com/slides
     15739
     

     View Slide