Pro Yearly is on sale from $80 to $50! »

4 CI/CD systems in 40 minutes

4 CI/CD systems in 40 minutes

Talk at Codemotion2017


Alejandro Guirao Rodríguez

November 24, 2017


  1. 4 CI/CD systems in 40 minutes MAD · NOV 24-25

    · 2017 Alejandro Guirao @lekum /cd-systems-in-40-minutes
  2. @lekum GITLAB CI

  3. @lekum History and evolution

  4. @lekum History of GitlabCI 2011- Birth of the Gitlab Project

    2012 - Gitlab CI was created 2015 - Gitlab CI was integrated in Gitlab 8 2017 - Auto DevOps was added to Gitlab 10
  5. @lekum How to use it

  6. @lekum .gitlab-ci.yml image: python:3 services: - postgres stages: - test

    job1: stage: test before_script: - pip install -r requirements.txt script: - python -m pytest . only: - master
  7. @lekum Job orchestration stages: - build - test - deploy

    build_amd64: stage: build script: [...] build_win32: stage: build test_amd64: stage: test test_win32: stage: test deploy_all_binaries: stage: deploy
  8. @lekum Job orchestration (II) stages: - build - cleanup_build -

    test - deploy - cleanup build_job: stage: build script: - make build cleanup_build_job: stage: cleanup_build script: - cleanup build when failed when: on_failure test_job: stage: test script: - make test deploy_job: stage: deploy script: - make deploy when: manual cleanup_job: stage: cleanup script: - cleanup after jobs when: always
  9. @lekum only and except job: # use regexp only: -

    /^issue-.*$/ # use special keyword except: - branches job: # use special keywords only: - tags - triggers - schedules job: only: - branches@gitlab-org/gitlab-ce except: - master@gitlab-org/gitlab-ce
  10. @lekum Environments deploy_staging: stage: deploy script: - echo "Deploy to

    staging server" environment: name: staging url: deploy_prod: stage: deploy script: - echo "Deploy to production server" environment: name: production url: when: manual
  11. @lekum Variables ▪ Trigger variables or scheduled pipeline variables ▪

    Project-level secret variables or protected secret variables ▪ Group-level secret variables or protected secret variables ▪ YAML-defined job-level variables ▪ YAML-defined global variables ▪ Deployment variables ▪ Predefined variables job_name: script: - echo $CI_JOB_ID
  12. @lekum Artifacts and dependencies build:osx: stage: build script: make build:osx

    artifacts: paths: - binaries/ build:linux: stage: build script: make build:linux artifacts: paths: - binaries/ test:osx: stage: test script: make test:osx dependencies: - build:osx test:linux: stage: test script: make test:linux dependencies: - build:linux deploy: stage: deploy script: make deploy
  13. @lekum Auto DevOps

  14. @lekum Auto DevOps ▪ Auto Build ∘ Dockerfile, Herokuish and

    Heroku Build Packs ▪ Auto Test ∘ Herokuish and Heroku Build Packs ▪ Auto Code Quality ∘ Codeclimate ▪ Auto Review Apps ▪ Auto Deploy ∘ Helm & k8s ▪ Auto Monitoring ∘ Prometheus
  15. @lekum How to deploy it

  16. @lekum Deploy Gitlab web: image: 'gitlab/gitlab-ce:latest' restart: always hostname: ''

    environment: GITLAB_OMNIBUS_CONFIG: | external_url '' # Add any other gitlab.rb configuration here, each on its own line ports: - '80:80' - '443:443' - '22:22' volumes: - '/srv/gitlab/config:/etc/gitlab' - '/srv/gitlab/logs:/var/log/gitlab' - '/srv/gitlab/data:/var/opt/gitlab'
  17. @lekum Deploy a GitlabCI runner docker run -d --name gitlab-runner

    --restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest
  18. @lekum Register the GitlabCI runner

  19. @lekum Register the GitlabCI runner docker exec -it gitlab-runner gitlab-runner

    register --name my-runner --url --registration-token my-registration-token
  20. @lekum Types of executors

  21. @lekum Docker-machine autoscaling concurrent = 50 # All registered Runners

    can run up to 50 concurrent builds [[runners]] url = "" token = "RUNNER_TOKEN" name = "autoscale-runner" executor = "docker+machine" # This Runner is using the 'docker+machine' executor limit = 10 # This Runner can execute up to 10 builds (created machines)
  22. @lekum CONCOURSE CI

  23. @lekum History and evolution

  24. @lekum History of Concourse CI 2014 - Version 0.1.0 2016

    - Version 1.0 (Hacker News!) 2017 - Current version: 3.6
  25. @lekum How to use it

  26. @lekum hello.yml resources: - name: 5m type: time source: {interval:

    5m} jobs: - name: hello-world plan: - get: 5m trigger: true - task: say-hello config: platform: linux image_resource: type: docker-image source: {repository: ubuntu} run: path: echo args: ["Hello, world!"] Task Resource Job Pipeline
  27. @lekum Deploy the pipeline $ fly -t hello-world login -c $ fly -t hello-world set-pipeline -p hello-world -c hello.yml $ fly -t hello-world unpause-pipeline -p hello-word
  28. @lekum A task and a job plan platform: linux image_resource:

    type: docker-image source: repository: golang tag: '1.9' inputs: - name: project-src ouputs: - name: built-project run: path: project-src/ci/build build.yml [...] plan: - get: project-src - task: build-bin file: project-src/ci/build.yml - put: project-bin params: file: built-project/my-project
  29. @lekum Resources: git resources: - name: source-code type: git source:

    uri: branch: master private_key: | -----BEGIN RSA PRIVATE KEY----- [...] -----END RSA PRIVATE KEY----- git_config: - name: core.bigFileThreshold value: 10m disable_ci_skip: true
  30. @lekum Resources: s3 - name: release type: s3 source: bucket:

    releases regexp: directory_on_s3/release-(.*).tgz access_key_id: ACCESS-KEY secret_access_key: SECRET
  31. @lekum Resources: docker-image - name: my-docker-image type: docker-image source: email:

    {{docker-hub-email}} username: {{docker-hub-username}} password: {{docker-hub-password}} repository: {{docker-hub-image-dummy-resource}}
  32. @lekum Implementing a new resource /opt/resource/check -> List of versions

    from one point in time /opt/resource/in -> Get a particular version /opt/resource/out -> Update the resource with a new version Docker image that implements these scripts:
  33. @lekum Example: /opt/resource/in for Git { "source": { "uri": "git://some-uri",

    "branch": "develop", "private_key": "..." }, "version": { "ref": "61cebf" } } { "version": { "ref": "61cebf" }, "metadata": [ { "name": "commit", "value": "61cebf" }, { "name": "author", "value": "Hulk Hogan" } ] } Input: Output:
  34. @lekum Step orchestration plan: - aggregate: - get: component-a -

    get: component-b - get: integration-suite - task: integration file: integration-suite/task.yml plan: - aggregate: - task: unit - do: - get: something-else - task: something-else-unit
  35. @lekum Pipelines: passing resources between jobs - name: job-bump-date serial:

    true plan: - get: resource-tutorial - get: resource-gist - task: bump-timestamp-file file: resource-tutorial/[...]/bump-timestamp-file.yml - put: resource-gist params: {repository: updated-gist} - name: job-show-date plan: - get: resource-tutorial - get: resource-gist passed: [job-bump-date] trigger: true - task: show-date config: platform: linux image_resource: type: docker-image source: {repository: busybox} inputs: - name: resource-gist run: path: cat args: [resource-gist/bumpme]
  36. @lekum Pipelines

  37. @lekum How to deploy it

  38. @lekum Via docker-compose.yml services: concourse-db: image: postgres:9.5 [...] concourse-web: image:

    concourse/concourse [...] concourse-worker: image: concourse/concourse privileged: true [...] mkdir -p keys/web keys/worker ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N '' ssh-keygen -t rsa -f ./keys/web/session_signing_key -N '' ssh-keygen -t rsa -f ./keys/worker/worker_key -N '' cp ./keys/worker/ ./keys/web/authorized_worker_keys cp ./keys/web/ ./keys/worker export CONCOURSE_EXTERNAL_URL=
  39. @lekum GOCD

  40. @lekum History and evolution

  41. @lekum History of GoCD 2007- ThoughtWorks creates Cruise 2010 -

    Cruise renamed to Go 2014 - Go released BSD and named GoCD 2017 - Current version: 17.11
  42. @lekum How to use it

  43. @lekum Pipeline concept

  44. @lekum Pipeline in GoCD

  45. @lekum Materials and dependencies

  46. @lekum … in the GoCD interface

  47. @lekum Managing GoCD: Dashboard

  48. @lekum Managing GoCD: Pipeline activity

  49. @lekum Managing GoCD: Job details

  50. @lekum Managing GoCD: adding a task

  51. @lekum Managing GoCD: Agents

  52. @lekum Configuring pipelines as code <config-repos> <config-repo pluginId="yaml.config.plugin" id="repo1"> <git

    url="" /> </config-repo> </config-repos> Add <config-repo> sections to the Config XML
  53. @lekum Configuring pipelines as code #ci.gocd.yaml format_version: 1 environments: testing:

    environment_variables: DEPLOYMENT: testing secure_variables: ENV_PASSWORD: "s&Du#@$xsSa" pipelines: - example-deploy-testing - build-testing pipelines: mypipe1 group: mygroup param1: value1 materials: mygit git: branch: ci myupstream: pipeline: pipe2 stage: test stages: - build: clean_workspace: true jobs: csharp: # name of the job resources: - net45 artifacts: - build: source: bin/ destination: build - test: source: tests/ destination: test-reports/ tasks: # ordered list of tasks to execute in job csharp - fetch: pipeline: pipe2 stage: build job: test source: test-bin/ destination: bin/ - exec: # indicates type of task command: make arguments: - "VERBOSE=true" # shorthand for script-executor plugin - script: ./ ci
  54. @lekum How to deploy it

  55. @lekum Run the GoCD server docker run -d -p8153:8153 -p8154:8154

    gocd/gocd-server:v17.11.0 -v /path/to/godata:/godata -v /path/to/home-dir:/home/go
  56. @lekum Run the agents docker run -d -e GO_SERVER_URL=... gocd/gocd-agent-ubuntu-16.04:v17.11.0

    -v /path/to/godata:/godata -v /path/to/home-dir:/home/go Other base images: gocd-agent-centos-6, gocd-agent-alpine-3.5
  57. @lekum SPINNAKER

  58. @lekum History and evolution

  59. @lekum History of Spinnaker 2016 - Public release by Netflix

    2017 - Current version: 1.5 (November)
  60. @lekum How to use it

  61. @lekum Concepts: cluster management

  62. @lekum Concepts: deployment management

  63. @lekum Concepts: deployment management

  64. @lekum Example of workflow

  65. @lekum Setting up a trigger

  66. @lekum Build and package

  67. @lekum Bake

  68. @lekum Deploy

  69. @lekum Deploy

  70. @lekum How to deploy it

  71. @lekum Architecture

  72. @lekum Proper deployment 1. Install Halyard 2. Choose local or

    distributed installation 3. Configure a persistent storage 4. Choose a cloud provider 5. Deploy with: hal deploy apply
  73. @lekum Experimental ter/experimental/docker-compose

  74. @lekum CONCLUSIONS

  75. @lekum GitlabCI Incredible feature set Very easy to use Very

    good approach to pipelines-as-code Impossible to use without Gitlab
  76. @lekum ConcourseCI Declarative, API and CLI-friendly Easy setup Very good

    at artifact creation Works only with Docker images Not focused on deployment
  77. @lekum GoCD Supports extremely complex pipelines Easy-to-medium setup complexity GUI-centric

    Scalability problems Pipelines-as-code implementation
  78. @lekum Spinnaker The most powerful tool for deployments Does not

    do CI (relies on third-parties) GUI-dependency for pipeline creation (Foremast!)
  79. @lekum Happy hacking! Alejandro Guirao @lekum