4 CI/CD systems in 40 minutes

4 CI/CD systems in 40 minutes

Talk at Codemotion2017

57da667eb23d6e814a844a41e2e9de08?s=128

Alejandro Guirao Rodríguez

November 24, 2017
Tweet

Transcript

  1. 1.

    4 CI/CD systems in 40 minutes MAD · NOV 24-25

    · 2017 Alejandro Guirao @lekum github.com/lekum lekum.org https://speakerdeck.com/lekum /cd-systems-in-40-minutes
  2. 4.

    @lekum History of GitlabCI 2011- Birth of the Gitlab Project

    2012 - Gitlab CI was created 2015 - Gitlab CI was integrated in Gitlab 8 2017 - Auto DevOps was added to Gitlab 10
  3. 6.

    @lekum .gitlab-ci.yml image: python:3 services: - postgres stages: - test

    job1: stage: test before_script: - pip install -r requirements.txt script: - python -m pytest . only: - master
  4. 7.

    @lekum Job orchestration stages: - build - test - deploy

    build_amd64: stage: build script: [...] build_win32: stage: build test_amd64: stage: test test_win32: stage: test deploy_all_binaries: stage: deploy
  5. 8.

    @lekum Job orchestration (II) stages: - build - cleanup_build -

    test - deploy - cleanup build_job: stage: build script: - make build cleanup_build_job: stage: cleanup_build script: - cleanup build when failed when: on_failure test_job: stage: test script: - make test deploy_job: stage: deploy script: - make deploy when: manual cleanup_job: stage: cleanup script: - cleanup after jobs when: always
  6. 9.

    @lekum only and except job: # use regexp only: -

    /^issue-.*$/ # use special keyword except: - branches job: # use special keywords only: - tags - triggers - schedules job: only: - branches@gitlab-org/gitlab-ce except: - master@gitlab-org/gitlab-ce
  7. 10.

    @lekum Environments deploy_staging: stage: deploy script: - echo "Deploy to

    staging server" environment: name: staging url: https://staging.example.com deploy_prod: stage: deploy script: - echo "Deploy to production server" environment: name: production url: https://example.com when: manual
  8. 11.

    @lekum Variables ▪ Trigger variables or scheduled pipeline variables ▪

    Project-level secret variables or protected secret variables ▪ Group-level secret variables or protected secret variables ▪ YAML-defined job-level variables ▪ YAML-defined global variables ▪ Deployment variables ▪ Predefined variables job_name: script: - echo $CI_JOB_ID
  9. 12.

    @lekum Artifacts and dependencies build:osx: stage: build script: make build:osx

    artifacts: paths: - binaries/ build:linux: stage: build script: make build:linux artifacts: paths: - binaries/ test:osx: stage: test script: make test:osx dependencies: - build:osx test:linux: stage: test script: make test:linux dependencies: - build:linux deploy: stage: deploy script: make deploy
  10. 14.

    @lekum Auto DevOps ▪ Auto Build ∘ Dockerfile, Herokuish and

    Heroku Build Packs ▪ Auto Test ∘ Herokuish and Heroku Build Packs ▪ Auto Code Quality ∘ Codeclimate ▪ Auto Review Apps ▪ Auto Deploy ∘ Helm & k8s ▪ Auto Monitoring ∘ Prometheus
  11. 16.

    @lekum Deploy Gitlab web: image: 'gitlab/gitlab-ce:latest' restart: always hostname: 'gitlab.example.com'

    environment: GITLAB_OMNIBUS_CONFIG: | external_url 'https://gitlab.example.com' # Add any other gitlab.rb configuration here, each on its own line ports: - '80:80' - '443:443' - '22:22' volumes: - '/srv/gitlab/config:/etc/gitlab' - '/srv/gitlab/logs:/var/log/gitlab' - '/srv/gitlab/data:/var/opt/gitlab'
  12. 17.

    @lekum Deploy a GitlabCI runner docker run -d --name gitlab-runner

    --restart always \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /srv/gitlab-runner/config:/etc/gitlab-runner \ gitlab/gitlab-runner:latest
  13. 19.

    @lekum Register the GitlabCI runner docker exec -it gitlab-runner gitlab-runner

    register --name my-runner --url http://gitlab.example.com --registration-token my-registration-token
  14. 21.

    @lekum Docker-machine autoscaling concurrent = 50 # All registered Runners

    can run up to 50 concurrent builds [[runners]] url = "https://gitlab.com" token = "RUNNER_TOKEN" name = "autoscale-runner" executor = "docker+machine" # This Runner is using the 'docker+machine' executor limit = 10 # This Runner can execute up to 10 builds (created machines)
  15. 24.

    @lekum History of Concourse CI 2014 - Version 0.1.0 2016

    - Version 1.0 (Hacker News!) 2017 - Current version: 3.6
  16. 26.

    @lekum hello.yml resources: - name: 5m type: time source: {interval:

    5m} jobs: - name: hello-world plan: - get: 5m trigger: true - task: say-hello config: platform: linux image_resource: type: docker-image source: {repository: ubuntu} run: path: echo args: ["Hello, world!"] Task Resource Job Pipeline
  17. 27.

    @lekum Deploy the pipeline $ fly -t hello-world login -c

    http://192.168.100.4:8080 $ fly -t hello-world set-pipeline -p hello-world -c hello.yml $ fly -t hello-world unpause-pipeline -p hello-word
  18. 28.

    @lekum A task and a job plan platform: linux image_resource:

    type: docker-image source: repository: golang tag: '1.9' inputs: - name: project-src ouputs: - name: built-project run: path: project-src/ci/build build.yml [...] plan: - get: project-src - task: build-bin file: project-src/ci/build.yml - put: project-bin params: file: built-project/my-project
  19. 29.

    @lekum Resources: git resources: - name: source-code type: git source:

    uri: git@github.com:concourse/git-resource.git branch: master private_key: | -----BEGIN RSA PRIVATE KEY----- [...] -----END RSA PRIVATE KEY----- git_config: - name: core.bigFileThreshold value: 10m disable_ci_skip: true
  20. 30.

    @lekum Resources: s3 - name: release type: s3 source: bucket:

    releases regexp: directory_on_s3/release-(.*).tgz access_key_id: ACCESS-KEY secret_access_key: SECRET
  21. 31.

    @lekum Resources: docker-image - name: my-docker-image type: docker-image source: email:

    {{docker-hub-email}} username: {{docker-hub-username}} password: {{docker-hub-password}} repository: {{docker-hub-image-dummy-resource}}
  22. 32.

    @lekum Implementing a new resource /opt/resource/check -> List of versions

    from one point in time /opt/resource/in -> Get a particular version /opt/resource/out -> Update the resource with a new version Docker image that implements these scripts:
  23. 33.

    @lekum Example: /opt/resource/in for Git { "source": { "uri": "git://some-uri",

    "branch": "develop", "private_key": "..." }, "version": { "ref": "61cebf" } } { "version": { "ref": "61cebf" }, "metadata": [ { "name": "commit", "value": "61cebf" }, { "name": "author", "value": "Hulk Hogan" } ] } Input: Output:
  24. 34.

    @lekum Step orchestration plan: - aggregate: - get: component-a -

    get: component-b - get: integration-suite - task: integration file: integration-suite/task.yml plan: - aggregate: - task: unit - do: - get: something-else - task: something-else-unit
  25. 35.

    @lekum Pipelines: passing resources between jobs - name: job-bump-date serial:

    true plan: - get: resource-tutorial - get: resource-gist - task: bump-timestamp-file file: resource-tutorial/[...]/bump-timestamp-file.yml - put: resource-gist params: {repository: updated-gist} - name: job-show-date plan: - get: resource-tutorial - get: resource-gist passed: [job-bump-date] trigger: true - task: show-date config: platform: linux image_resource: type: docker-image source: {repository: busybox} inputs: - name: resource-gist run: path: cat args: [resource-gist/bumpme]
  26. 38.

    @lekum Via docker-compose.yml services: concourse-db: image: postgres:9.5 [...] concourse-web: image:

    concourse/concourse [...] concourse-worker: image: concourse/concourse privileged: true [...] mkdir -p keys/web keys/worker ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N '' ssh-keygen -t rsa -f ./keys/web/session_signing_key -N '' ssh-keygen -t rsa -f ./keys/worker/worker_key -N '' cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys cp ./keys/web/tsa_host_key.pub ./keys/worker export CONCOURSE_EXTERNAL_URL=http://192.168.1.142:8080
  27. 41.

    @lekum History of GoCD 2007- ThoughtWorks creates Cruise 2010 -

    Cruise renamed to Go 2014 - Go released BSD and named GoCD 2017 - Current version: 17.11
  28. 52.

    @lekum Configuring pipelines as code <config-repos> <config-repo pluginId="yaml.config.plugin" id="repo1"> <git

    url="https://github.com/tomzo/gocd-yaml-config-example.git" /> </config-repo> </config-repos> Add <config-repo> sections to the Config XML
  29. 53.

    @lekum Configuring pipelines as code #ci.gocd.yaml format_version: 1 environments: testing:

    environment_variables: DEPLOYMENT: testing secure_variables: ENV_PASSWORD: "s&Du#@$xsSa" pipelines: - example-deploy-testing - build-testing pipelines: mypipe1 group: mygroup param1: value1 materials: mygit git: http://my.example.org/mygit.git branch: ci myupstream: pipeline: pipe2 stage: test stages: - build: clean_workspace: true jobs: csharp: # name of the job resources: - net45 artifacts: - build: source: bin/ destination: build - test: source: tests/ destination: test-reports/ tasks: # ordered list of tasks to execute in job csharp - fetch: pipeline: pipe2 stage: build job: test source: test-bin/ destination: bin/ - exec: # indicates type of task command: make arguments: - "VERBOSE=true" # shorthand for script-executor plugin - script: ./build.sh ci
  30. 55.

    @lekum Run the GoCD server docker run -d -p8153:8153 -p8154:8154

    gocd/gocd-server:v17.11.0 -v /path/to/godata:/godata -v /path/to/home-dir:/home/go
  31. 56.

    @lekum Run the agents docker run -d -e GO_SERVER_URL=... gocd/gocd-agent-ubuntu-16.04:v17.11.0

    -v /path/to/godata:/godata -v /path/to/home-dir:/home/go Other base images: gocd-agent-centos-6, gocd-agent-alpine-3.5
  32. 59.

    @lekum History of Spinnaker 2016 - Public release by Netflix

    2017 - Current version: 1.5 (November)
  33. 72.

    @lekum Proper deployment 1. Install Halyard 2. Choose local or

    distributed installation 3. Configure a persistent storage 4. Choose a cloud provider 5. Deploy with: hal deploy apply
  34. 75.

    @lekum GitlabCI Incredible feature set Very easy to use Very

    good approach to pipelines-as-code Impossible to use without Gitlab
  35. 76.

    @lekum ConcourseCI Declarative, API and CLI-friendly Easy setup Very good

    at artifact creation Works only with Docker images Not focused on deployment
  36. 78.

    @lekum Spinnaker The most powerful tool for deployments Does not

    do CI (relies on third-parties) GUI-dependency for pipeline creation (Foremast!)