A smooth migration to Docker focusing on build pipelines

Transcript

1. Paul Bonaud @paulrb_r Maxime Visonneau @mvisonneau A SMOOTH MIGRATION TO DOCKER FOCUSING ON BUILD PIPELINES
2. ↓ ↓
3. Paul Bonaud ⌨ + ⚙ + ▶ Infra & Ops Engineer ⏸ So ware Developer Maxime Visonneau ⌨ + + Infrastructure Engineer
4. Meet Maurice
6. Meet Maurice aka build2
8. Manually administrated
10. Building all projects Android, JS (Ember), C++, Ruby
12. Deploying everything Nginx, apps, infrastructure
14. Single architecture (Debian 7)
16. New era
18. Docker
19. Open Source ❤
20. Isolated
21. Immutable
22. Stateless
23. Version controlled Dockerfile
24. Dockerfile FROM ruby:2.4-jessie RUN apt-get update RUN apt-get -y install libpq-dev ghostscript WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
25. Dockerfile FROM ruby:2.4-jessie RUN apt-get update RUN apt-get -y install libpq-dev ghostscript WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
26. Dockerfile FROM ruby:2.4-jessie RUN apt-get update && \ apt-get -y install libpq-dev ghostscript && \ rm -rf /var/lib/apt/lists/* WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
27. Dockerfile FROM ruby:2.4-jessie RUN apt-get update && \ apt-get -y install libpq-dev ghostscript && \ rm -rf /var/lib/apt/lists/* WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
28. Dockerfile FROM ruby:2.4-jessie RUN apt-get update && \ apt-get -y install libpq-dev ghostscript && \ rm -rf /var/lib/apt/lists/* WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
29. Dockerfile FROM ruby:2.4-jessie RUN apt-get update && \ apt-get -y install libpq-dev ghostscript && \ rm -rf /var/lib/apt/lists/* WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
30. GitLab
31. Open Source ❤
34. Integrated Docker registry! registry.gitlab.com
35. Gitlab-CI
36. Open Source ❤
37. Declarative
38. Integrated
39. Version controlled .gitlab-ci.yml
40. .gitlab-ci.yml stages: - build - test - package - deploy
41. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} LATEST_IMAGE: ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest build:docker: stage: build tags: [ privileged ] image: "docker:latest" before_script: - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} script: - docker pull ${LATEST_IMAGE} || true - docker build --cache-from ${LATEST_IMAGE} -t ${IMAGE} -t ${LATEST_IMAGE} . - docker push ${IMAGE} - docker push ${LATEST_IMAGE}
42. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} LATEST_IMAGE: ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest build:docker: stage: build tags: [ privileged ] image: "docker:latest" before_script: - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} script: - docker pull ${LATEST_IMAGE} || true - docker build --cache-from ${LATEST_IMAGE} -t ${IMAGE} -t ${LATEST_IMAGE} . - docker push ${IMAGE} - docker push ${LATEST_IMAGE}
43. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} LATEST_IMAGE: ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest build:docker: stage: build tags: [ privileged ] image: "docker:latest" before_script: - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} script: - docker pull ${LATEST_IMAGE} || true - docker build --cache-from ${LATEST_IMAGE} -t ${IMAGE} -t ${LATEST_IMAGE} . - docker push ${IMAGE} - docker push ${LATEST_IMAGE}
44. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} LATEST_IMAGE: ${CI_REGISTRY}/${CI_PROJECT_PATH}:latest build:docker: stage: build tags: [ privileged ] image: "docker:latest" before_script: - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} script: - docker pull ${LATEST_IMAGE} || true - docker build --cache-from ${LATEST_IMAGE} -t ${IMAGE} -t ${LATEST_IMAGE} . - docker push ${IMAGE} - docker push ${LATEST_IMAGE}
45. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} # (...) image: ${IMAGE}
46. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} # (...) image: ${IMAGE} test: stage: test script: - make test
47. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} # (...) image: ${IMAGE} test: stage: test script: - make test package: stage: package script: - make package > release.tar.gz artifacts: paths: [ release.tar.gz ]
48. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} # (...) deploy:integration: stage: deploy image: ${CI_REGISTRY}/infra/ansible:latest variables: ENV: integration script: - make deploy file=release.tar.gz env=${ENV}
49. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} # (...) deploy:integration: &deploy stage: deploy image: ${CI_REGISTRY}/infra/ansible:latest variables: ENV: integration script: - make deploy file=release.tar.gz env=${ENV} deploy:production: <<: *deploy when: manual variables: ENV: production
50. What about easy OS testing and upgrades?!
51. Dockerfile FROM ruby:2.4-jessie RUN apt-get update RUN apt-get -y install libpq-dev ghostscript WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
52. Dockerfile FROM ruby:2.4-stretch RUN apt-get update RUN apt-get -y install libpq-dev ghostscript WORKDIR /opt/app COPY Gemfile Gemfile.lock /opt/app RUN bundle install VOLUME /opt/app CMD [ "make", "run" ]
53. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHA} # (...) image: ${IMAGE}
54. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${OS}_${CI_COMMIT_SHA} # (...) image: ${IMAGE}
55. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${OS}_${CI_COMMIT_SHA} # (...) image: ${IMAGE} test:jessie: stage: test variables: OS: jessie script: - make test
56. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${OS}_${CI_COMMIT_SHA} # (...) image: ${IMAGE} test:jessie: &test stage: test variables: OS: jessie script: - make test test:stretch: <<: *test variables: OS: stretch
57. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${OS}_${CI_COMMIT_SHA} # (...) image: ${IMAGE} package:jessie: &package stage: package variables: OS: jessie script: - make package > release-${OS}.tar.gz artifacts: paths: [ release-${OS}.tar.gz ] package:stretch: <<: *package variables: OS: stretch
58. .gitlab-ci.yml variables: IMAGE: ${CI_REGISTRY_IMAGE}:${OS}_${CI_COMMIT_SHA} # (...) deploy:integration: &deploy stage: deploy image: ${CI_REGISTRY}/infra/ansible:latest variables: OS: stretch ENV: integration script: - make deploy file=release-${OS}.tar.gz env=${ENV} deploy:production: <<: *deploy when: manual variables: OS: jessie ENV: production
60. Runners
61. Di erent Executors / Multiple OSes Shell Docker VirtualBox Parallels SSH Kubernetes Linux Windows Mac OS X FreeBSD
62. Di erent Executors / Multiple OSes Shell Docker VirtualBox Parallels SSH Kubernetes Linux Windows Mac OS X FreeBSD
63. Portable
64. Light system dependencies
65. Easily configurable and maintainable
66. --- classes: - docker - gitlab::ci gitlab::ci::runners: "%{::fqdn}_docker": executor: docker docker-image: docker:latest
67. --- classes: - docker - gitlab::ci gitlab::ci::runners: "%{::fqdn}_docker": executor: docker docker-image: docker:latest docker-volumes: "/var/run/docker.sock:/var/run/docker.sock" docker-privileged: true
68. AWS
69. Terraform
70. Open Source ❤
71. terraform.tf resource "aws_instance" "gitlab_runner" { count = 10 ami = "ami-6b2cd712" instance_type = "m4.large" [..] }
74. MISSINGDEP Who had any clue this system library was required?
75. MISSINGDEP Update Dockerfiles with missing dependencies
76. ENOSPC No space le on device
77. ENOSPC Increase space / Automate cleanup of old containers and images
78. UNAUTH_DEPLOY Where are my SSH keys for Ansible?
79. UNAUTH_DEPLOY Enhance security by creating individual keys per project/env
80. SHM_OOM Less than 64MB of free space in temporary directory for shared memory files
81. SHM_OOM Configure max SHM size
82. IOWAIT On heavy load tests
83. IOWAIT Bigger disks Provisioned IOPS Leverage tmpfs
87. AWS
88. Docker
89. Terraform
90. terraform.tf resource "aws_autoscaling_group" "gitlab_runner" { desired_capacity = 10 min_size = 4 max_size = 24 launch_configuration = "${aws_launch_configuration.runner.name}" [..] }
91. We can do better and easier!
92. Ourselves with an interesting challenge
93. So ware engineers with a reliable CI implementation
94. Finance by making some savings
95. docker-machine integrated out-of-the-box with gitlab-runner scheduling jobs on spot instances
96. gitlab::ci::runners: 'aws_spot_docker_machine': executor: docker+machine limit: 24 machine-IdleCount: 4 machine-IdleTime: 600 machine-OffPeakPeriods: - "* * 0-8,18-23 * * mon-fri *" - "* * * * * sat,sun *" machine-OffPeakIdleCount: 0 machine-OffPeakIdleTime: 600 machine-MaxBuilds: 30 machine-MachineDriver: amazonec2 machine-MachineOptions: - amazonec2-instance-type=m4.large - amazonec2-request-spot-instance=true - amazonec2-spot-price=0.10 [..]
97. gitlab::ci::runners: 'aws_spot_docker_machine': executor: docker+machine limit: 24 machine-IdleCount: 4 machine-IdleTime: 600 machine-OffPeakPeriods: - "* * 0-8,18-23 * * mon-fri *" - "* * * * * sat,sun *" machine-OffPeakIdleCount: 0 machine-OffPeakIdleTime: 600 machine-MaxBuilds: 30 machine-MachineDriver: amazonec2 machine-MachineOptions: - amazonec2-instance-type=m4.large - amazonec2-request-spot-instance=true - amazonec2-spot-price=0.10 [..]
98. gitlab::ci::runners: 'aws_spot_docker_machine': executor: docker+machine limit: 24 machine-IdleCount: 4 machine-IdleTime: 600 machine-OffPeakPeriods: - "* * 0-8,18-23 * * mon-fri *" - "* * * * * sat,sun *" machine-OffPeakIdleCount: 0 machine-OffPeakIdleTime: 600 machine-MaxBuilds: 30 machine-MachineDriver: amazonec2 machine-MachineOptions: - amazonec2-instance-type=m4.large - amazonec2-request-spot-instance=true - amazonec2-spot-price=0.10 [..]
99. gitlab::ci::runners: 'aws_spot_docker_machine': executor: docker+machine limit: 24 machine-IdleCount: 4 machine-IdleTime: 600 machine-OffPeakPeriods: - "* * 0-8,18-23 * * mon-fri *" - "* * * * * sat,sun *" machine-OffPeakIdleCount: 0 machine-OffPeakIdleTime: 600 machine-MaxBuilds: 30 machine-MachineDriver: amazonec2 machine-MachineOptions: - amazonec2-instance-type=m4.large - amazonec2-request-spot-instance=true - amazonec2-spot-price=0.10 [..]
100. gitlab::ci::runners: 'aws_spot_docker_machine': executor: docker+machine limit: 24 machine-IdleCount: 4 machine-IdleTime: 600 machine-OffPeakPeriods: - "* * 0-8,18-23 * * mon-fri *" - "* * * * * sat,sun *" machine-OffPeakIdleCount: 0 machine-OffPeakIdleTime: 600 machine-MaxBuilds: 30 machine-MachineDriver: amazonec2 machine-MachineOptions: - amazonec2-instance-type=m4.large - amazonec2-request-spot-instance=true - amazonec2-spot-price=0.10 [..]
101. gitlab::ci::runners: 'aws_spot_docker_machine': executor: docker+machine limit: 24 machine-IdleCount: 4 machine-IdleTime: 600 machine-OffPeakPeriods: - "* * 0-8,18-23 * * mon-fri *" - "* * * * * sat,sun *" machine-OffPeakIdleCount: 0 machine-OffPeakIdleTime: 600 machine-MaxBuilds: 30 machine-MachineDriver: amazonec2 machine-MachineOptions: - amazonec2-instance-type=m4.large - amazonec2-request-spot-instance=true - amazonec2-spot-price=0.10 [..]
102. gitlab::ci::runners: 'aws_spot_docker_machine': executor: docker+machine limit: 24 machine-IdleCount: 4 machine-IdleTime: 600 machine-OffPeakPeriods: - "* * 0-8,18-23 * * mon-fri *" - "* * * * * sat,sun *" machine-OffPeakIdleCount: 0 machine-OffPeakIdleTime: 600 machine-MaxBuilds: 30 machine-MachineDriver: amazonec2 machine-MachineOptions: - amazonec2-instance-type=m4.large - amazonec2-request-spot-instance=true - amazonec2-spot-price=0.10 [..]
105. A small drawback..
106. Scheduling
107. Kubernetes
108. --- apiVersion: v1 kind: ConfigMap metadata: name: gitlab-runner namespace: gitlab-ci data: config.toml: | [[runners]] name = "k8s_runner" url = "https://gitlab.example.com/" executor = "kubernetes" [runners.kubernetes] cpu_limit = "1" memory_limit = "256Mi" [..]
109. ~$ kubectl -n gitlab-ci get po NAME READY STATUS RESTARTS AGE runner-315e4d80-0qj79g 2/2 Running 0 3m runner-315e4d80-9kdkee 2/2 Running 0 2m runner-315e4d80-kdfdfe 2/2 Running 0 1m runner-315e4d80-ldorpk 2/2 Running 0 1m runner-315e4d80-pleofi 2/2 Running 0 20s
110. Definitely promising
111. Still a work in progress for us
113. QUESTIONS?
114. REFERENCES · Dockerfile documentation · .gitlab-ci.yml documentation · Slides https://docs.docker.com/engine/reference/builder/ https://gitlab.com/help/ci/yaml/README.md http://polr.fr/tiaddocker