4 CI/CD systems in 40 minutes

4 CI/CD systems in 40 minutes MAD · NOV 24-25
· 2017 Alejandro Guirao @lekum github.com/lekum lekum.org https://speakerdeck.com/lekum /cd-systems-in-40-minutes

@lekum GITLAB CI

@lekum History and evolution

@lekum History of GitlabCI 2011- Birth of the Gitlab Project
2012 - Gitlab CI was created 2015 - Gitlab CI was integrated in Gitlab 8 2017 - Auto DevOps was added to Gitlab 10

@lekum How to use it

@lekum .gitlab-ci.yml image: python:3 services: - postgres stages: - test
job1: stage: test before_script: - pip install -r requirements.txt script: - python -m pytest . only: - master

@lekum Job orchestration stages: - build - test - deploy
build_amd64: stage: build script: [...] build_win32: stage: build test_amd64: stage: test test_win32: stage: test deploy_all_binaries: stage: deploy

@lekum Job orchestration (II) stages: - build - cleanup_build -
test - deploy - cleanup build_job: stage: build script: - make build cleanup_build_job: stage: cleanup_build script: - cleanup build when failed when: on_failure test_job: stage: test script: - make test deploy_job: stage: deploy script: - make deploy when: manual cleanup_job: stage: cleanup script: - cleanup after jobs when: always

@lekum only and except job: # use regexp only: -
/^issue-.*$/ # use special keyword except: - branches job: # use special keywords only: - tags - triggers - schedules job: only: - branches@gitlab-org/gitlab-ce except: - master@gitlab-org/gitlab-ce

@lekum Environments deploy_staging: stage: deploy script: - echo "Deploy to
staging server" environment: name: staging url: https://staging.example.com deploy_prod: stage: deploy script: - echo "Deploy to production server" environment: name: production url: https://example.com when: manual

@lekum Variables ▪ Trigger variables or scheduled pipeline variables ▪
Project-level secret variables or protected secret variables ▪ Group-level secret variables or protected secret variables ▪ YAML-defined job-level variables ▪ YAML-defined global variables ▪ Deployment variables ▪ Predefined variables job_name: script: - echo $CI_JOB_ID

@lekum Artifacts and dependencies build:osx: stage: build script: make build:osx
artifacts: paths: - binaries/ build:linux: stage: build script: make build:linux artifacts: paths: - binaries/ test:osx: stage: test script: make test:osx dependencies: - build:osx test:linux: stage: test script: make test:linux dependencies: - build:linux deploy: stage: deploy script: make deploy

@lekum Auto DevOps

@lekum Auto DevOps ▪ Auto Build ∘ Dockerfile, Herokuish and
Heroku Build Packs ▪ Auto Test ∘ Herokuish and Heroku Build Packs ▪ Auto Code Quality ∘ Codeclimate ▪ Auto Review Apps ▪ Auto Deploy ∘ Helm & k8s ▪ Auto Monitoring ∘ Prometheus

@lekum How to deploy it

@lekum Deploy Gitlab web: image: 'gitlab/gitlab-ce:latest' restart: always hostname: 'gitlab.example.com'
environment: GITLAB_OMNIBUS_CONFIG: | external_url 'https://gitlab.example.com' # Add any other gitlab.rb configuration here, each on its own line ports: - '80:80' - '443:443' - '22:22' volumes: - '/srv/gitlab/config:/etc/gitlab' - '/srv/gitlab/logs:/var/log/gitlab' - '/srv/gitlab/data:/var/opt/gitlab'

@lekum Deploy a GitlabCI runner docker run -d --name gitlab-runner
--restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest

@lekum Register the GitlabCI runner

@lekum Register the GitlabCI runner docker exec -it gitlab-runner gitlab-runner
register --name my-runner --url http://gitlab.example.com --registration-token my-registration-token

@lekum Types of executors

@lekum Docker-machine autoscaling concurrent = 50 # All registered Runners
can run up to 50 concurrent builds [[runners]] url = "https://gitlab.com" token = "RUNNER_TOKEN" name = "autoscale-runner" executor = "docker+machine" # This Runner is using the 'docker+machine' executor limit = 10 # This Runner can execute up to 10 builds (created machines)

@lekum CONCOURSE CI

@lekum History of Concourse CI 2014 - Version 0.1.0 2016
- Version 1.0 (Hacker News!) 2017 - Current version: 3.6

@lekum hello.yml resources: - name: 5m type: time source: {interval:
5m} jobs: - name: hello-world plan: - get: 5m trigger: true - task: say-hello config: platform: linux image_resource: type: docker-image source: {repository: ubuntu} run: path: echo args: ["Hello, world!"] Task Resource Job Pipeline

@lekum Deploy the pipeline $ fly -t hello-world login -c
http://192.168.100.4:8080 $ fly -t hello-world set-pipeline -p hello-world -c hello.yml $ fly -t hello-world unpause-pipeline -p hello-word

@lekum A task and a job plan platform: linux image_resource:
type: docker-image source: repository: golang tag: '1.9' inputs: - name: project-src ouputs: - name: built-project run: path: project-src/ci/build build.yml [...] plan: - get: project-src - task: build-bin file: project-src/ci/build.yml - put: project-bin params: file: built-project/my-project

@lekum Resources: git resources: - name: source-code type: git source:
uri: [email protected]:concourse/git-resource.git branch: master private_key: | -----BEGIN RSA PRIVATE KEY----- [...] -----END RSA PRIVATE KEY----- git_config: - name: core.bigFileThreshold value: 10m disable_ci_skip: true

@lekum Resources: s3 - name: release type: s3 source: bucket:
releases regexp: directory_on_s3/release-(.*).tgz access_key_id: ACCESS-KEY secret_access_key: SECRET

@lekum Resources: docker-image - name: my-docker-image type: docker-image source: email:
{{docker-hub-email}} username: {{docker-hub-username}} password: {{docker-hub-password}} repository: {{docker-hub-image-dummy-resource}}

@lekum Implementing a new resource /opt/resource/check -> List of versions
from one point in time /opt/resource/in -> Get a particular version /opt/resource/out -> Update the resource with a new version Docker image that implements these scripts:

@lekum Example: /opt/resource/in for Git { "source": { "uri": "git://some-uri",
"branch": "develop", "private_key": "..." }, "version": { "ref": "61cebf" } } { "version": { "ref": "61cebf" }, "metadata": [ { "name": "commit", "value": "61cebf" }, { "name": "author", "value": "Hulk Hogan" } ] } Input: Output:

@lekum Step orchestration plan: - aggregate: - get: component-a -
get: component-b - get: integration-suite - task: integration file: integration-suite/task.yml plan: - aggregate: - task: unit - do: - get: something-else - task: something-else-unit

@lekum Pipelines: passing resources between jobs - name: job-bump-date serial:
true plan: - get: resource-tutorial - get: resource-gist - task: bump-timestamp-file file: resource-tutorial/[...]/bump-timestamp-file.yml - put: resource-gist params: {repository: updated-gist} - name: job-show-date plan: - get: resource-tutorial - get: resource-gist passed: [job-bump-date] trigger: true - task: show-date config: platform: linux image_resource: type: docker-image source: {repository: busybox} inputs: - name: resource-gist run: path: cat args: [resource-gist/bumpme]

@lekum Pipelines

@lekum Via docker-compose.yml services: concourse-db: image: postgres:9.5 [...] concourse-web: image:
concourse/concourse [...] concourse-worker: image: concourse/concourse privileged: true [...] mkdir -p keys/web keys/worker ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N '' ssh-keygen -t rsa -f ./keys/web/session_signing_key -N '' ssh-keygen -t rsa -f ./keys/worker/worker_key -N '' cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys cp ./keys/web/tsa_host_key.pub ./keys/worker export CONCOURSE_EXTERNAL_URL=http://192.168.1.142:8080

@lekum GOCD

@lekum History of GoCD 2007- ThoughtWorks creates Cruise 2010 -
Cruise renamed to Go 2014 - Go released BSD and named GoCD 2017 - Current version: 17.11

@lekum Pipeline concept

@lekum Pipeline in GoCD

@lekum Materials and dependencies

@lekum … in the GoCD interface

@lekum Managing GoCD: Dashboard

@lekum Managing GoCD: Pipeline activity

@lekum Managing GoCD: Job details

@lekum Managing GoCD: adding a task

@lekum Managing GoCD: Agents

@lekum Configuring pipelines as code <config-repos> <config-repo pluginId="yaml.config.plugin" id="repo1"> <git
url="https://github.com/tomzo/gocd-yaml-config-example.git" /> </config-repo> </config-repos> Add <config-repo> sections to the Config XML

@lekum Configuring pipelines as code #ci.gocd.yaml format_version: 1 environments: testing:
environment_variables: DEPLOYMENT: testing secure_variables: ENV_PASSWORD: "s&Du#@$xsSa" pipelines: - example-deploy-testing - build-testing pipelines: mypipe1 group: mygroup param1: value1 materials: mygit git: http://my.example.org/mygit.git branch: ci myupstream: pipeline: pipe2 stage: test stages: - build: clean_workspace: true jobs: csharp: # name of the job resources: - net45 artifacts: - build: source: bin/ destination: build - test: source: tests/ destination: test-reports/ tasks: # ordered list of tasks to execute in job csharp - fetch: pipeline: pipe2 stage: build job: test source: test-bin/ destination: bin/ - exec: # indicates type of task command: make arguments: - "VERBOSE=true" # shorthand for script-executor plugin - script: ./build.sh ci

@lekum Run the GoCD server docker run -d -p8153:8153 -p8154:8154
gocd/gocd-server:v17.11.0 -v /path/to/godata:/godata -v /path/to/home-dir:/home/go

@lekum Run the agents docker run -d -e GO_SERVER_URL=... gocd/gocd-agent-ubuntu-16.04:v17.11.0
-v /path/to/godata:/godata -v /path/to/home-dir:/home/go Other base images: gocd-agent-centos-6, gocd-agent-alpine-3.5

@lekum SPINNAKER

@lekum History of Spinnaker 2016 - Public release by Netflix
2017 - Current version: 1.5 (November)

@lekum Concepts: cluster management

@lekum Concepts: deployment management

@lekum Example of workflow

@lekum Setting up a trigger

@lekum Build and package

@lekum Bake

@lekum Deploy

@lekum Architecture

@lekum Proper deployment 1. Install Halyard 2. Choose local or
distributed installation 3. Configure a persistent storage 4. Choose a cloud provider 5. Deploy with: hal deploy apply

@lekum Experimental https://github.com/spinnaker/spinnaker/tree/mas ter/experimental/docker-compose

@lekum CONCLUSIONS

@lekum GitlabCI Incredible feature set Very easy to use Very
good approach to pipelines-as-code Impossible to use without Gitlab

@lekum ConcourseCI Declarative, API and CLI-friendly Easy setup Very good
at artifact creation Works only with Docker images Not focused on deployment

@lekum GoCD Supports extremely complex pipelines Easy-to-medium setup complexity GUI-centric
Scalability problems Pipelines-as-code implementation

@lekum Spinnaker The most powerful tool for deployments Does not
do CI (relies on third-parties) GUI-dependency for pipeline creation (Foremast!)

@lekum Happy hacking! Alejandro Guirao @lekum lekum.org

4 CI/CD systems in 40 minutes

4 CI/CD systems in 40 minutes

More Decks by Alejandro Guirao Rodríguez

Featured

Transcript