$30 off During Our Annual Pro Sale. View Details »

4 CI/CD systems in 40 minutes

4 CI/CD systems in 40 minutes

Talk at Codemotion2017

Alejandro Guirao Rodríguez

November 24, 2017
Tweet

Transcript

  1. 4 CI/CD systems in 40 minutes
    MAD · NOV 24-25 · 2017
    Alejandro Guirao
    @lekum
    github.com/lekum
    lekum.org
    https://speakerdeck.com/lekum
    /cd-systems-in-40-minutes

    View Slide

  2. @lekum
    GITLAB CI

    View Slide

  3. @lekum
    History
    and
    evolution

    View Slide

  4. @lekum
    History of GitlabCI
    2011- Birth of the Gitlab Project
    2012 - Gitlab CI was created
    2015 - Gitlab CI was integrated in Gitlab 8
    2017 - Auto DevOps was added to Gitlab 10

    View Slide

  5. @lekum
    How to
    use it

    View Slide

  6. @lekum
    .gitlab-ci.yml
    image: python:3
    services:
    - postgres
    stages:
    - test
    job1:
    stage: test
    before_script:
    - pip install -r requirements.txt
    script:
    - python -m pytest .
    only:
    - master

    View Slide

  7. @lekum
    Job orchestration
    stages:
    - build
    - test
    - deploy
    build_amd64:
    stage: build
    script: [...]
    build_win32:
    stage: build
    test_amd64:
    stage: test
    test_win32:
    stage: test
    deploy_all_binaries:
    stage: deploy

    View Slide

  8. @lekum
    Job orchestration (II)
    stages:
    - build
    - cleanup_build
    - test
    - deploy
    - cleanup
    build_job:
    stage: build
    script:
    - make build
    cleanup_build_job:
    stage: cleanup_build
    script:
    - cleanup build when failed
    when: on_failure
    test_job:
    stage: test
    script:
    - make test
    deploy_job:
    stage: deploy
    script:
    - make deploy
    when: manual
    cleanup_job:
    stage: cleanup
    script:
    - cleanup after jobs
    when: always

    View Slide

  9. @lekum
    only and except
    job:
    # use regexp
    only:
    - /^issue-.*$/
    # use special keyword
    except:
    - branches
    job:
    # use special keywords
    only:
    - tags
    - triggers
    - schedules
    job:
    only:
    - branches@gitlab-org/gitlab-ce
    except:
    - master@gitlab-org/gitlab-ce

    View Slide

  10. @lekum
    Environments
    deploy_staging:
    stage: deploy
    script:
    - echo "Deploy to staging server"
    environment:
    name: staging
    url: https://staging.example.com
    deploy_prod:
    stage: deploy
    script:
    - echo "Deploy to production server"
    environment:
    name: production
    url: https://example.com
    when: manual

    View Slide

  11. @lekum
    Variables
    ■ Trigger variables or scheduled pipeline variables
    ■ Project-level secret variables or protected secret
    variables
    ■ Group-level secret variables or protected secret
    variables
    ■ YAML-defined job-level variables
    ■ YAML-defined global variables
    ■ Deployment variables
    ■ Predefined variables
    job_name:
    script:
    - echo $CI_JOB_ID

    View Slide

  12. @lekum
    Artifacts and dependencies
    build:osx:
    stage: build
    script: make build:osx
    artifacts:
    paths:
    - binaries/
    build:linux:
    stage: build
    script: make build:linux
    artifacts:
    paths:
    - binaries/
    test:osx:
    stage: test
    script: make test:osx
    dependencies:
    - build:osx
    test:linux:
    stage: test
    script: make test:linux
    dependencies:
    - build:linux
    deploy:
    stage: deploy
    script: make deploy

    View Slide

  13. @lekum
    Auto DevOps

    View Slide

  14. @lekum
    Auto DevOps
    ■ Auto Build
    ∘ Dockerfile, Herokuish and Heroku Build Packs
    ■ Auto Test
    ∘ Herokuish and Heroku Build Packs
    ■ Auto Code Quality
    ∘ Codeclimate
    ■ Auto Review Apps
    ■ Auto Deploy
    ∘ Helm & k8s
    ■ Auto Monitoring
    ∘ Prometheus

    View Slide

  15. @lekum
    How to
    deploy it

    View Slide

  16. @lekum
    Deploy Gitlab
    web:
    image: 'gitlab/gitlab-ce:latest'
    restart: always
    hostname: 'gitlab.example.com'
    environment:
    GITLAB_OMNIBUS_CONFIG: |
    external_url 'https://gitlab.example.com'
    # Add any other gitlab.rb configuration here, each on its own line
    ports:
    - '80:80'
    - '443:443'
    - '22:22'
    volumes:
    - '/srv/gitlab/config:/etc/gitlab'
    - '/srv/gitlab/logs:/var/log/gitlab'
    - '/srv/gitlab/data:/var/opt/gitlab'

    View Slide

  17. @lekum
    Deploy a GitlabCI runner
    docker run -d --name gitlab-runner --restart always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /srv/gitlab-runner/config:/etc/gitlab-runner \
    gitlab/gitlab-runner:latest

    View Slide

  18. @lekum
    Register the GitlabCI runner

    View Slide

  19. @lekum
    Register the GitlabCI runner
    docker exec -it gitlab-runner gitlab-runner
    register --name my-runner --url
    http://gitlab.example.com --registration-token
    my-registration-token

    View Slide

  20. @lekum
    Types of executors

    View Slide

  21. @lekum
    Docker-machine autoscaling
    concurrent = 50 # All registered Runners can
    run up to 50 concurrent builds
    [[runners]]
    url = "https://gitlab.com"
    token = "RUNNER_TOKEN"
    name = "autoscale-runner"
    executor = "docker+machine" # This
    Runner is using the 'docker+machine' executor
    limit = 10 # This
    Runner can execute up to 10 builds (created
    machines)

    View Slide

  22. @lekum
    CONCOURSE
    CI

    View Slide

  23. @lekum
    History
    and
    evolution

    View Slide

  24. @lekum
    History of Concourse CI
    2014 - Version 0.1.0
    2016 - Version 1.0 (Hacker News!)
    2017 - Current version: 3.6

    View Slide

  25. @lekum
    How to
    use it

    View Slide

  26. @lekum
    hello.yml
    resources:
    - name: 5m
    type: time
    source: {interval: 5m}
    jobs:
    - name: hello-world
    plan:
    - get: 5m
    trigger: true
    - task: say-hello
    config:
    platform: linux
    image_resource:
    type: docker-image
    source: {repository: ubuntu}
    run:
    path: echo
    args: ["Hello, world!"]
    Task
    Resource
    Job
    Pipeline

    View Slide

  27. @lekum
    Deploy the pipeline
    $ fly -t hello-world login -c http://192.168.100.4:8080
    $ fly -t hello-world set-pipeline -p hello-world -c hello.yml
    $ fly -t hello-world unpause-pipeline -p hello-word

    View Slide

  28. @lekum
    A task and a job plan
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: golang
    tag: '1.9'
    inputs:
    - name: project-src
    ouputs:
    - name: built-project
    run:
    path: project-src/ci/build
    build.yml
    [...]
    plan:
    - get: project-src
    - task: build-bin
    file: project-src/ci/build.yml
    - put: project-bin
    params: file: built-project/my-project

    View Slide

  29. @lekum
    Resources: git
    resources:
    - name: source-code
    type: git
    source:
    uri: [email protected]:concourse/git-resource.git
    branch: master
    private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    [...]
    -----END RSA PRIVATE KEY-----
    git_config:
    - name: core.bigFileThreshold
    value: 10m
    disable_ci_skip: true

    View Slide

  30. @lekum
    Resources: s3
    - name: release
    type: s3
    source:
    bucket: releases
    regexp: directory_on_s3/release-(.*).tgz
    access_key_id: ACCESS-KEY
    secret_access_key: SECRET

    View Slide

  31. @lekum
    Resources: docker-image
    - name: my-docker-image
    type: docker-image
    source:
    email: {{docker-hub-email}}
    username: {{docker-hub-username}}
    password: {{docker-hub-password}}
    repository: {{docker-hub-image-dummy-resource}}

    View Slide

  32. @lekum
    Implementing a new resource
    /opt/resource/check -> List of versions from one point in time
    /opt/resource/in -> Get a particular version
    /opt/resource/out -> Update the resource with a new version
    Docker image that implements these scripts:

    View Slide

  33. @lekum
    Example:
    /opt/resource/in for Git
    {
    "source": {
    "uri": "git://some-uri",
    "branch": "develop",
    "private_key": "..."
    },
    "version": { "ref": "61cebf" }
    }
    {
    "version": { "ref": "61cebf" },
    "metadata": [
    { "name": "commit", "value": "61cebf" },
    { "name": "author", "value": "Hulk Hogan" }
    ]
    }
    Input: Output:

    View Slide

  34. @lekum
    Step orchestration
    plan:
    - aggregate:
    - get: component-a
    - get: component-b
    - get: integration-suite
    - task: integration
    file: integration-suite/task.yml
    plan:
    - aggregate:
    - task: unit
    - do:
    - get: something-else
    - task: something-else-unit

    View Slide

  35. @lekum
    Pipelines: passing resources
    between jobs
    - name: job-bump-date
    serial: true
    plan:
    - get: resource-tutorial
    - get: resource-gist
    - task: bump-timestamp-file
    file:
    resource-tutorial/[...]/bump-timestamp-file.yml
    - put: resource-gist
    params: {repository: updated-gist}
    - name: job-show-date
    plan:
    - get: resource-tutorial
    - get: resource-gist
    passed: [job-bump-date]
    trigger: true
    - task: show-date
    config:
    platform: linux
    image_resource:
    type: docker-image
    source: {repository: busybox}
    inputs:
    - name: resource-gist
    run:
    path: cat
    args: [resource-gist/bumpme]

    View Slide

  36. @lekum
    Pipelines

    View Slide

  37. @lekum
    How to
    deploy it

    View Slide

  38. @lekum
    Via docker-compose.yml
    services:
    concourse-db:
    image: postgres:9.5
    [...]
    concourse-web:
    image: concourse/concourse
    [...]
    concourse-worker:
    image: concourse/concourse
    privileged: true
    [...]
    mkdir -p keys/web keys/worker
    ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N ''
    ssh-keygen -t rsa -f ./keys/web/session_signing_key -N ''
    ssh-keygen -t rsa -f ./keys/worker/worker_key -N ''
    cp ./keys/worker/worker_key.pub
    ./keys/web/authorized_worker_keys
    cp ./keys/web/tsa_host_key.pub ./keys/worker
    export CONCOURSE_EXTERNAL_URL=http://192.168.1.142:8080

    View Slide

  39. @lekum
    GOCD

    View Slide

  40. @lekum
    History
    and
    evolution

    View Slide

  41. @lekum
    History of GoCD
    2007- ThoughtWorks creates Cruise
    2010 - Cruise renamed to Go
    2014 - Go released BSD and named GoCD
    2017 - Current version: 17.11

    View Slide

  42. @lekum
    How to use it

    View Slide

  43. @lekum
    Pipeline concept

    View Slide

  44. @lekum
    Pipeline in GoCD

    View Slide

  45. @lekum
    Materials and dependencies

    View Slide

  46. @lekum
    … in the GoCD interface

    View Slide

  47. @lekum
    Managing GoCD: Dashboard

    View Slide

  48. @lekum
    Managing GoCD: Pipeline activity

    View Slide

  49. @lekum
    Managing GoCD: Job details

    View Slide

  50. @lekum
    Managing GoCD: adding a task

    View Slide

  51. @lekum
    Managing GoCD: Agents

    View Slide

  52. @lekum
    Configuring pipelines as code





    Add sections to the Config XML

    View Slide

  53. @lekum
    Configuring pipelines as code
    #ci.gocd.yaml
    format_version: 1
    environments:
    testing:
    environment_variables:
    DEPLOYMENT: testing
    secure_variables:
    ENV_PASSWORD: "s&Du#@$xsSa"
    pipelines:
    - example-deploy-testing
    - build-testing
    pipelines:
    mypipe1
    group: mygroup
    param1: value1
    materials:
    mygit
    git: http://my.example.org/mygit.git
    branch: ci
    myupstream:
    pipeline: pipe2
    stage: test
    stages:
    - build:
    clean_workspace: true
    jobs:
    csharp: # name of the job
    resources:
    - net45
    artifacts:
    - build:
    source: bin/
    destination: build
    - test:
    source: tests/
    destination: test-reports/
    tasks: # ordered list of tasks to execute in job csharp
    - fetch:
    pipeline: pipe2
    stage: build
    job: test
    source: test-bin/
    destination: bin/
    - exec: # indicates type of task
    command: make
    arguments:
    - "VERBOSE=true"
    # shorthand for script-executor plugin
    - script: ./build.sh ci

    View Slide

  54. @lekum
    How to
    deploy it

    View Slide

  55. @lekum
    Run the GoCD server
    docker run -d -p8153:8153 -p8154:8154
    gocd/gocd-server:v17.11.0
    -v /path/to/godata:/godata
    -v /path/to/home-dir:/home/go

    View Slide

  56. @lekum
    Run the agents
    docker run -d -e GO_SERVER_URL=...
    gocd/gocd-agent-ubuntu-16.04:v17.11.0
    -v /path/to/godata:/godata
    -v /path/to/home-dir:/home/go
    Other base images:
    gocd-agent-centos-6, gocd-agent-alpine-3.5

    View Slide

  57. @lekum
    SPINNAKER

    View Slide

  58. @lekum
    History
    and
    evolution

    View Slide

  59. @lekum
    History of Spinnaker
    2016 - Public release by Netflix
    2017 - Current version: 1.5 (November)

    View Slide

  60. @lekum
    How to use it

    View Slide

  61. @lekum
    Concepts: cluster management

    View Slide

  62. @lekum
    Concepts: deployment management

    View Slide

  63. @lekum
    Concepts: deployment management

    View Slide

  64. @lekum
    Example of workflow

    View Slide

  65. @lekum
    Setting
    up a
    trigger

    View Slide

  66. @lekum
    Build and
    package

    View Slide

  67. @lekum
    Bake

    View Slide

  68. @lekum
    Deploy

    View Slide

  69. @lekum
    Deploy

    View Slide

  70. @lekum
    How to deploy it

    View Slide

  71. @lekum
    Architecture

    View Slide

  72. @lekum
    Proper deployment
    1. Install Halyard
    2. Choose local or distributed installation
    3. Configure a persistent storage
    4. Choose a cloud provider
    5. Deploy with: hal deploy apply

    View Slide

  73. @lekum
    Experimental
    https://github.com/spinnaker/spinnaker/tree/mas
    ter/experimental/docker-compose

    View Slide

  74. @lekum
    CONCLUSIONS

    View Slide

  75. @lekum
    GitlabCI
    Incredible feature set
    Very easy to use
    Very good approach to pipelines-as-code
    Impossible to use without Gitlab

    View Slide

  76. @lekum
    ConcourseCI
    Declarative, API and CLI-friendly
    Easy setup
    Very good at artifact creation
    Works only with Docker images
    Not focused on deployment

    View Slide

  77. @lekum
    GoCD
    Supports extremely complex pipelines
    Easy-to-medium setup complexity
    GUI-centric
    Scalability problems
    Pipelines-as-code implementation

    View Slide

  78. @lekum
    Spinnaker
    The most powerful tool for deployments
    Does not do CI (relies on third-parties)
    GUI-dependency for pipeline creation
    (Foremast!)

    View Slide

  79. @lekum
    Happy hacking!
    Alejandro Guirao
    @lekum
    lekum.org

    View Slide