Herding your containers and services

Herding your sites and services

Whether you are moving away from a monolithic web application to a microservices oriented architecture or just have loads of sites to manage; you can benefit from running your own private managed cloud. In this talk, I will be introducing you to the wondrous world of containers and container orchestration using Docker, Rancher and Cattle. Even if you don’t plan on a really large infrastructure, as soon as you have more than a couple of servers you too can benefit from this.

Hoe sites en diensten te hoeden

Of je aan het overstappen bent van een monolitisch web applicatie naar een architectuur met microservices of je hebt heel veel sites te beheren; je kan door het inzetten van je eigen privé cloud een hoop besparen. In deze presentatie geef ik een introductie in de wonderlijke wereld van containers en container orchestration middels Docker, Rancher en Cattle. Zelfs als je niet van plan bent om een grote infrastructuur te beheren, zodra je al meer dan een paar servers beheert kan je voordeel hebben van deze techniek.

Transcript

1. HERDING YOUR CONTAINERS
   AND SERVICES
   Mike van Riel - @mvriel
   

   View Slide

2. MIKE VAN RIEL
   ➤ Part of Ingewikkeld
   ➤ Architect, Engineer, Trainer,
   Coach and Game Developer
   ➤ Lead Developer for
   phpDocumentor
   ➤ Game Designer for Elrakis
   ➤ Contributor for many projects
   

   View Slide

3. YOU WANT A WEBAPP
   Hypothesis
   

   View Slide

4. SERVER
   FIRST YOU NEED A SERVER
   

   View Slide

5. SERVER
   … FOR YOUR SWEET LITTLE WEBAPP
   PHP
   

   View Slide

6. SERVER
   … WHO NEEDS A FRIEND TO LOOK AFTER HIS STUFF
   PHP
   MYSQL
   

   View Slide

7. YOU WANT MORE
   Hypothesis
   

   View Slide

8. SERVER
   ANOTHER SWEET LITTLE WEBAPP JOINS THE FAMILY
   PHP
   MYSQL
   PHP
   

   View Slide

9. SERVER
   … AND ANOTHER
   PHP
   MYSQL
   PHP
   PHP
   

   View Slide

10. SERVER
    … AND ANOTHER
    PHP
    MYSQL
    PHP
    PHP
    PHP
    

    View Slide

11. YOU NOW HAVE WEBAPPS
    Hypothesis
    

    View Slide

12. ONE OF THEM GROWS UP
    Hypothesis
    

    View Slide

13. SERVER
    BILLY GOT BIG
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    

    View Slide

14. SERVER
    BILLY GROWS EVEN MORE, SQUEEZING THE OTHERS
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    

    View Slide

15. TIME FOR A NEW SERVER
    Hypothesis
    

    View Slide

16. YOU BUY A NEW SERVER
    SERVER1
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    SERVER2
    

    View Slide

17. SERVER2
    SERVER1
    AND BILLY MOVES OUT OF THE HOUSE
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    OHHH. A MIGRATION!
    

    View Slide

18. SERVER2
    SERVER1
    BUT BILLY STILL WANTS A FRIEND TO KEEP HIS STUFF
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    MORE MAINTENANCE
    

    View Slide

19. BILLY BECOMES POPULAR
    Hypothesis
    

    View Slide

20. SERVER2
    SERVER1
    BILLY NEEDS TO CLONE HIMSELF
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    SERVER3
    BILLY
    MORE MAINTENANCE
    

    View Slide

21. SERVER2
    SERVER1
    PSST. YOU KNOW YOU PAY TOO MUCH?
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    SERVER3
    BILLY
    WASTE
    

    View Slide

22. SERVER2
    SERVER1
    BILLY GETS NEW FRIENDS!
    PHP
    MYSQL
    BILLY
    PHP
    PHP
    SERVER3
    BILLY REDIS
    ELASTIC
    

    View Slide

23. SERVER2
    SERVER1
    AND BILLY GROWS EVEN MORE
    PHP
    MYSQL BILLY
    PHP
    PHP
    SERVER3
    BILLY
    REDIS
    ELASTIC
    

    View Slide

24. A SHORT SUMMARY
    ➤ When a web app grows it gets more dependencies
    ➤ Apps become woven with the server
    ➤ As a result migration is painful
    ➤ Servers pollute during their lifecycle
    ➤ Server configuration goes out of sync after time
    ➤ As a result maintenance costs rise
    

    View Slide

25. BREAKING THINGS APART
    The story of containers
    

    View Slide

26. WHAT IS A CONTAINER?
    Sort of a Virtual Machine, but not quite
    

    View Slide

27. TRADITIONAL LAMP STACK
    SERVER PHP MYSQL
    O/S Apache
    

    View Slide

28. CONTAINERIZED LAMP STACK
    SERVER
    PHP MYSQL
    O/S
    Apache
    DOCKER
    Containers
    

    View Slide

29. CONTAINERIZED LNMP STACK
    SERVER
    PHP-FPM MYSQL
    O/S
    NGINX
    DOCKER
    Containers
    PHP-FPM
    PHP-FPM
    

    View Slide

30. CONTAINERS ARE SELF-CONTAINED SERVICES
    ➤The more isolated a container, the better
    it works:
    ➤Should be stateless
    ➤Should be independent
    ➤Should be disposable
    ➤Should not need scripted provisioning
    

    View Slide

31. “
    Containers, by their nature, do not
    know or care whether you run them
    locally or on production
    

    View Slide

32. WARNING:
    LIVE DEMOS AHEAD
    I hope my self-worth survives
    

    View Slide

33. PULLING A RABBIT OUT OF
    A CONTAINER
    docker run -d -p15672:15672
    rabbitmq:management
    

    View Slide

34. CREATING YOUR OWN
    CONTAINER
    Getting a PHP application to work.
    

    View Slide

35. DESCRIBE YOUR SERVICE USING A “DOCKERFILE”
    FROM php:apache
    MAINTAINER Mike van Riel
    ADD . /var/www/html
    Dockerfile
    

    View Slide

36. AN EXAMPLE PHP SCRIPT
    phpinfo();
    index.php
    

    View Slide

37. BUILD IT!
    docker build -t killing-billy .
    

    View Slide

38. RUN IT!
    docker run -d -p8042:80 killing-billy
    

    View Slide

39. DID IT WORK AND WHAT IS THE STATUS?
    CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
    43685e9df887 killing-billy "docker-php-entryp..." 4 seconds ago Up 2 seconds 0.0.0.0:8042->80/tcp heuristic_mcclintock
    $ docker ps
    

    View Slide

40. WHAT ABOUT PROVISIONING?
    ➤ Some applications require provisioning
    ➤ Often provisioning is more like priming
    ➤ Do this while building your container
    using the RUN command
    ➤ Or do this as part of starting the
    container using the CMD or
    ENTRYPOINT command
    

    View Slide

41. COMBINING
    SERVICES
    USING
    DOCKER
    COMPOSE
    Not Docker Composer!
    

    View Slide

42. DOCKER-COMPOSE.YML
    database:
    image: mysql:latest
    ports: [ “3306:3306" ]
    environment:
    MYSQL_ROOT_PASSWORD: "secret"
    website:
    env_file: .env
    build: .
    ports: [ "8042:80" ]
    volumes:
    - .:/var/www/html
    links:
    - database
    

    View Slide

43. BUT THIS IS ONLY
    LOCALLY, RIGHT?
    Let’s fix that.
    

    View Slide

44. REGISTRIES
    ➤ Containers can be stored in ‘registries’
    ➤ Public registries such as http://hub.docker.com
    ➤ Private registries
    ➤ Self-hosted
    ➤ Gitlab.com
    ➤ Quay.io
    ➤ Docker Hub
    ➤ And more…
    

    View Slide

45. HOSTING ON GITLAB AND USING THEIR REGISTRY
    

    View Slide

46. BUILD IT!
    docker build -t registry.gitlab.com/mvriel/
    herding-your-containers/killing-billy .
    

    View Slide

47. PUSH IT!
    docker push registry.gitlab.com/mvriel/herding-
    your-containers/killing-billy
    

    View Slide

48. TO THE BAT-SERVER!
    

    View Slide

49. RUN IT!
    docker run -d -p8042:80 registry.gitlab.com/
    mvriel/herding-your-containers/killing-billy
    

    View Slide

50. CONTINUOUS
    INTEGRATION
    Or: how to build your containers automatically
    

    View Slide

51. BUILDING A CONTAINER - .GITLAB-CI.YML
    build:
    stage: build
    image: "docker:git"
    services: [ "docker:dind" ]
    cache: { paths: [ vendor/ ] }
    # install dependencies
    before_script:
    - docker run -t -v $CI_PROJECT_DIR:/app --env-file
    $CI_PROJECT_DIR/.env composer/composer
    install -o --no-interaction --no-dev --ignore-platform-reqs --
    no-progress
    script:
    # build container
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN
    registry.gitlab.com
    - docker build -t registry.gitlab.com/ingewikkeld/
    

    View Slide

52. A SHORT SUMMARY
    ➤ Containers are isolated from the server
    ➤ They are cheap to create and destroy
    ➤ They can be combined together to form a more
    intricate system
    ➤ Building and provisioning is easy and automated
    ➤ You can run them anywhere (provided Docker is
    installed)
    

    View Slide

53. ORCHESTRATION
    The robots are taking over
    

    View Slide

54. A SERVER-AGNOSTIC LAYER
    Without orchestration, containers are
    bound to a specific server
    With orchestration, containers are
    free to move around
    

    View Slide

55. “
    This is why containers must be
    disposable; they should be able to be
    destroyed and created arbitrarily
    

    View Slide

56. “
    This is why containers must not be
    post-provisioned; they should be able
    to be destroyed and created quickly
    

    View Slide

57. REMEMBER THE DOCKER-COMPOSE.YML?
    database:
    image: mysql:latest
    ports: [ “3306:3306" ]
    environment:
    MYSQL_ROOT_PASSWORD: "secret"
    website:
    env_file: .env
    build: .
    ports: [ "8042:80" ]
    volumes:
    - .:/var/www
    links:
    - database
    

    View Slide

58. SEVERAL TECHNOLOGIES
    

    View Slide

59. ENVIRONMENTS, STACKS AND SERVICES
    Environment
    Stack
    Service
    Service
    Stack
    Service
    Environment
    Stack
    Service
    Service
    Stack
    Service
    Environment
    Stack
    Service
    Service
    Stack
    Service
    

    View Slide

60. ENVIRONMENTS, STACKS AND SERVICES
    Test
    Billy
    MySQL
    PHP
    Stack
    PHP
    Acceptance
    Billy
    MySQL
    PHP
    Stack
    PHP
    Production
    Billy
    MySQL
    PHP
    Stack
    PHP
    

    View Slide

61. Production
    HOW DOES THIS TRANSLATE?
    PHP
    MySQL
    PHP
    PHP
    MySQL
    

    View Slide

62. INSTALLING RANCHER
    

    View Slide

63. View Slide

64. STARTING A RANCHER SERVER
    database:
    image: mariadb:latest
    env_file: .env
    volumes:
    - "/mnt/mariadb:/var/lib/mysql"
    restart: unless-stopped
    rancher:
    image: rancher/server
    env_file: .env
    container_name: rancher-server
    ports:
    - "8080:8080"
    restart: unless-stopped
    links:
    - "database"
    docker-compose.yml
    

    View Slide

65. RANCHER SERVER - ENVIRONMENT VARIABLES
    MYSQL_ROOT_PASSWORD=[INSERT PASSWORD]
    MYSQL_DATABASE=cattle
    MYSQL_USER=cattle
    MYSQL_PASSWORD=[INSERT THE CATTLE USER PASSWORD]
    CATTLE_DB_CATTLE_DATABASE=mysql
    CATTLE_DB_CATTLE_MYSQL_HOST=database
    CATTLE_DB_CATTLE_MYSQL_NAME=cattle
    CATTLE_DB_CATTLE_USERNAME=cattle
    CATTLE_DB_CATTLE_PASSWORD=[INSERT THE CATTLE USER PASSWORD]
    .env
    

    View Slide

66. CONTINUOUS
    DELIVERY
    Or: how to deploy your containers automatically
    

    View Slide

67. BUILDING A CONTAINER - .GITLAB-CI.YML
    build:
    stage: build
    image: "docker:git"
    services: [ "docker:dind" ]
    cache: { paths: [ vendor/ ] }
    # install dependencies
    before_script:
    - docker run -t -v $CI_PROJECT_DIR:/app --env-file
    $CI_PROJECT_DIR/.env -e SYMFONY_ENV=prod composer/composer
    install -o --no-interaction --no-dev --ignore-platform-reqs --
    no-progress
    script:
    # build container
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN
    registry.gitlab.com
    - docker build -t registry.gitlab.com/ingewikkeld/
    

    View Slide

68. DEPLOYING A CONTAINER - .GITLAB-CI.YML
    stages:
    - build
    - deploy
    build
    [...]
    deploy:
    stage: deploy
    image: cdrx/rancher-gitlab-deploy
    variables: { GIT_STRATEGY: none }
    script:
    # deploy container to registry
    - upgrade --environment="ingewikkeld" --stack="wecamp"
    --service="website" --finish-upgrade
    only:
    - master
    

    View Slide

69. THINGS TO
    WATCH OUT
    FOR
    Because containers move around
    PORT CONFLICTS
    Use load balancers to direct
    traffic to the right service.
    DNS
    Use services that
    dynamically update your
    DNS records.
    PERSISTENT STORAGE
    Use cloud-based file storage,
    NFS or build your app for
    this using FlySystem.
    

    View Slide

70. DEMO
    Where we go, we don’t need slides.
    

    View Slide

71. THANK YOU
    ➤ Contact me on twitter: @mvriel
    ➤ Please leave a rating on joind.in
    ➤ If you have questions that don’t fit in 140 characters:
    me@mikevanriel.com
    BILLY
    

    View Slide