Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to ConcourseCI

Introduction to ConcourseCI

Presentation from Devclub.eu (30.11.2016, Tallinn). Video is here (rus): https://www.youtube.com/watch?v=JtbR2bTmjkI

Links:
- ConcourseCI website: http://concourse.ci/
- concourse/docker-image-resource: https://github.com/concourse/docker-image-resource
- (old) mumoshu/dcind: https://github.com/mumoshu/dcind
- (updated) I use this image to run Docker in Docker: https://hub.docker.com/r/w32blaster/dcind/
- Atom Plugin: https://atom.io/packages/concourse-vis
- ConcourseCI Radiator: https://github.com/moodev/concourseci-radiator

The full working script to run Functional Tests is here:
https://gist.github.com/w32blaster/965bad80a6613fe924eb2f82aa159e4f

Ilja Hämäläinen

November 30, 2016
Tweet

More Decks by Ilja Hämäläinen

Other Decks in Programming

Transcript

  1. Concourse CI
    Clear Prop!
    Ilja Hämäläinen
    2016

    View full-size slide

  2. Concourse CI
    Clear Prop!
    Ilja Hämäläinen
    2016

    View full-size slide

  3. Need a CI server?

    View full-size slide

  4. Why
    ConcourseCI?
    1. Pipelines are first-class citizens
    2. Containers
    3. Everything is a code (automation)

    View full-size slide

  5. Let’s start learning

    View full-size slide

  6. 3 concepts

    Resources
    Gives a sources (code,
    binaries, files…)
    No actions

    Tasks
    Instructions, how
    something should be run.

    Jobs
    Combines tasks and
    resources to get some
    result.

    View full-size slide

  7. Resources
    Job
    Running
    Jobs
    Pipeline

    View full-size slide

  8. How to talk with the Concourse?

    View full-size slide

  9. fly - Command Line Interface

    View full-size slide

  10. fly -h
    Available commands:
    abort-build Abort a build (aliases: ab)
    builds List builds data (aliases: bs)
    check-resource Check a resource (aliases: cr)
    checklist Print a Checkfile of the given pipeline (aliases: cl)
    containers Print the active containers (aliases: cs)
    destroy-pipeline Destroy a pipeline (aliases: dp)
    execute Execute a one-off build using local bits (aliases: e)
    get-pipeline Get a pipeline's current configuration (aliases: gp)
    help Print this help message
    hijack Execute a command in a container (aliases: intercept, i)
    login Authenticate with the target (aliases: l)
    pause-job Pause a job (aliases: pj)
    pause-pipeline Pause a pipeline (aliases: pp)
    pipelines List the configured pipelines (aliases: ps)
    rename-pipeline Rename a pipeline (aliases: rp)
    set-pipeline Create or update a pipeline's configuration (aliases: sp)
    sync Download and replace the current fly from the target (aliases: s)
    targets List saved targets (aliases: ts)
    trigger-job Start a job in a pipeline (aliases: tj)
    unpause-job Unpause a job (aliases: uj)
    unpause-pipeline Un-pause a pipeline (aliases: up)
    volumes List the active volumes (aliases: vs)
    watch Stream a build's output (aliases: w)
    workers List the registered workers (aliases: ws)

    View full-size slide

  11. fly: login to server (only the first time)

    View full-size slide

  12. Fly: login to a server (1)
    fly --target office ...
    Custom server
    name

    View full-size slide

  13. Fly: login to a server (1)
    fly --target office login ...
    Command to
    login

    View full-size slide

  14. Fly: login to a server (1)
    fly --target office login --concourse-url http://concourse.myhost.com
    URL of our internal
    ConcoueseCI

    View full-size slide

  15. Fly: login to a server (2)
    $ cat ~/.flyrc
    targets:
    office:
    api: http://concourse.my-host.com
    token:
    type: Bearer
    value:
    tutorial:
    api: http://localhost:8080
    token:
    type: ""
    value: ""
    Server on the
    my-host.com
    Server on my
    localhost

    View full-size slide

  16. fly: create a new pipeline

    View full-size slide

  17. Fly: create new pipeline (1)
    fly sp ...
    alias for
    “set-pipiline”

    View full-size slide

  18. Fly: create new pipeline (1)
    fly sp -t office ...
    Execute this
    command on
    the “office”
    server

    View full-size slide

  19. Fly: create new pipeline (1)
    fly sp -t office -c pipeline.yml ...
    The pipeline
    itself in YAML
    format (config
    file)

    View full-size slide

  20. Fly: create new pipeline (1)
    fly sp -t office -c pipeline.yml -p my-pipeline
    The custom
    name for the
    pipeline. Will
    appear in GUI

    View full-size slide

  21. Fly: create new pipeline (2)
    ....
    job job-findbug has been added:
    name: job-findbug
    public: true
    plan:
    - get: git-resource
    passed:
    - job-build
    trigger: true
    - get: common-images
    passed:
    - job-build
    - task: task-findbug
    file: common-images/java-findbugs/task-findbugs.yml
    params:
    APP_TO_BE_CHECKED: /git-resource/webapp/target/service-registry-api.war
    apply configuration? [yN]:

    View full-size slide

  22. Fly: create new pipeline (3)
    Concourse CI server
    DELL
    Dev local computer
    fly sp -t office -c pipeline.yml -p my-pipeline
    pipeline.yml

    View full-size slide

  23. Available
    pipelines
    Pipeline

    View full-size slide

  24. How to make a pipeline.yml?

    View full-size slide

  25. 1) Specify resource
    resources:
    - name: git-resource
    type: git
    source:
    uri: http://git.myhost.com/<...>.git
    The custom name
    The resource type
    Git repo URL

    View full-size slide

  26. 2) Task
    ---
    platform: linux
    image_resource:
    type: docker-image
    source: {repository: busybox}
    run:
    path: echo
    args: [hello world]
    Set the Docker
    image
    The command to be
    executed inside the
    container

    View full-size slide

  27. 3) Job
    jobs:
    - name: job-build
    plan:
    - get: git-resource
    - task: task-1
    ......
    - put: slack-resource

    View full-size slide

  28. Job name
    Previous
    builds
    Button “run new build
    manually”
    Resource(s)
    Task(s)
    Resource(s)

    View full-size slide

  29. Example pipelines

    View full-size slide

  30. Scenario #1:
    1. Get the code from Git
    2. Build it (using Maven)

    View full-size slide

  31. ---
    resources:
    - name: git-resource
    type: git
    source:
    uri: http://git.myhost.com/myproject.git
    jobs:
    - name: job-build
    plan:
    - get: git-resource
    trigger: true
    - task: task-build
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: my.docker.registry/java-maven
    insecure_registries: ["my.docker.registry:5080"]
    tag: latest
    inputs:
    - name: git-resource
    run:
    path: mvn
    args: [ deploy ]
    1. Declare
    resources

    View full-size slide

  32. ---
    resources:
    - name: git-resource
    type: git
    source:
    uri: http://git.myhost.com/myproject.git
    jobs:
    - name: job-build
    plan:
    - get: git-resource
    trigger: true
    - task: task-build
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: my.docker.registry/java-maven
    insecure_registries: ["my.docker.registry:5080"]
    tag: latest
    inputs:
    - name: git-resource
    run:
    path: mvn
    args: [ deploy ]
    2. Declare one job
    “job-build”
    3. Set plan, which...

    View full-size slide

  33. ---
    resources:
    - name: git-resource
    type: git
    source:
    uri: http://git.myhost.com/myproject.git
    jobs:
    - name: job-build
    plan:
    - get: git-resource
    trigger: true
    - task: task-build
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: my.docker.registry/java-maven
    insecure_registries: ["my.docker.registry:5080"]
    tag: latest
    inputs:
    - name: git-resource
    run:
    path: mvn
    args: [ deploy ]
    a) gets the code from the declared resource
    (triggering on every new commit)

    View full-size slide

  34. ---
    resources:
    - name: git-resource
    type: git
    source:
    uri: http://git.myhost.com/myproject.git
    jobs:
    - name: job-build
    plan:
    - get: git-resource
    trigger: true
    - task: task-build
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: my.docker.registry/java-maven
    inputs:
    - name: git-resource
    run:
    path: mvn
    args: [ deploy ]
    b) executes one task, which...
    ...calls the command “mvn deploy”
    ...inside the container

    View full-size slide

  35. Your script should return
    standard UNIX status
    code (0 = success)
    You can run any executable in your task
    ...
    run:
    path: any-executable-you-like.sh

    View full-size slide

  36. Input, output for a task

    View full-size slide

  37. ---
    platform: linux
    #
    # General task to build a Maven based project.
    # It requires the pom.xml file at the root
    #
    image_resource:
    type: docker-image
    source:
    repository: alpine
    tag: latest
    inputs:
    - name: git-resource
    outputs:
    - name: result
    run:
    path: mvn
    args: [ deploy ]
    Inputs are used to pass some files inside a task.
    There will be /git-resource directory created
    in the container
    Outputs to send something to other tasks.
    There will be /result directory created in the
    container.

    View full-size slide

  38. Scenario #2: Inputs and Outputs
    1. Generate some content in Task1
    2. Access this content from Task2

    View full-size slide

  39. ---
    jobs:
    - name: job-build
    plan:
    - task: step-1
    ...
    outputs:
    - name: shared-dir
    run:
    path: sh
    args:
    - -exc
    - |
    echo “test” > shared-dir/hello.txt
    - task: step-2
    ...
    inputs:
    - name: shared-dir
    run:
    path: sh
    args:
    - -exc
    - |
    ls shared-dir && shared-dir/hello.txt
    1.
    There is the directory shared-dir in the
    container
    2. Save a file in this directory

    View full-size slide

  40. ---
    jobs:
    - name: job-build
    plan:
    - task: step-1
    ...
    outputs:
    - name: shared-dir
    run:
    path: sh
    args:
    - -exc
    - |
    echo “test” > shared-dir/hello.txt
    - task: step-2
    ...
    inputs:
    - name: shared-dir
    run:
    path: sh
    args:
    - -exc
    - |
    ls shared-dir && cat shared-dir/hello.txt
    3. There is the directory
    shared-dir in the container,
    with inputs data (file hello.txt)
    4. Demonstrate that we
    have access to a file,
    created in previous task (in
    another container!)

    View full-size slide

  41. 1. Task “Step-1” pulls
    the Docker image
    2. Creates file
    “hello.txt”
    3. Task “Step-2” prints
    Content of hello.txt

    View full-size slide

  42. What is happening inside a container?

    View full-size slide

  43. To view a task output you have three choices:
    1. From the GUI
    2. fly watch
    3. fly hijack

    View full-size slide

  44. Extracted tasks

    View full-size slide

  45. Firstly, let’s extract a task to a separate file...

    View full-size slide

  46. 1. Task could be stored in separate file
    pipeline.yml:
    jobs:
    - name: job-build
    plan:
    - get: git-resource
    - task: step-1
    platform: linux
    image_reource:
    <...>
    run:
    <...>
    task.yml:
    ---
    platform: linux
    image_resource:
    <...>
    run:
    < some commands >

    View full-size slide

  47. 1. Task could be stored in separate file
    pipeline.yml:
    jobs:
    - name: job-build
    plan:
    - get: git-resource
    - task: step-1
    file: path/to/task.yml
    task.yml:
    ---
    platform: linux
    image_resource:
    <...>
    run:
    < some commands >

    View full-size slide

  48. ...and we can reuse common tasks and images!
    Feature #1

    View full-size slide

  49. ---
    platform: linux
    #
    # General task to build a Maven based project.
    # It requires the pom.xml file at the root
    #
    image_resource:
    type: docker-image
    source:
    repository: my.docker.registry/java-maven
    inputs:
    - name: git-resource
    run:
    path: mvn
    args: [ deploy ]
    dir: git-resource
    Example of a common task for a Java project

    View full-size slide

  50. For example....

    View full-size slide

  51. Scenario #3: Common tasks
    1. Get resource from common-tasks repo
    2. Execute common task from resource
    3. Execute embedded task

    View full-size slide

  52. ---
    resources:
    - name: common-images
    type: git
    source:
    uri: http://git.myhost.com/common-ci-images.git
    jobs:
    - name: job-build
    plan:
    - get: common-images
    - task: task-build
    file: common-images/task-build.yml
    - task: task-deploy
    ...
    run:
    path: sh
    args:
    - -exc
    - |
    echo “Hello Devs”
    Resource 1: some
    project repository

    View full-size slide

  53. ---
    resources:
    - name: common-images
    type: git
    source:
    uri: http://git.myhost.com/common-ci-images.git
    jobs:
    - name: job-build
    plan:
    - get: common-images
    - task: task-build
    file: common-images/task-build.yml
    - task: task-deploy
    ...
    run:
    path: sh
    args:
    - -exc
    - |
    echo “Hello Devs”
    Get resource
    Build a project, using
    common task
    Perform some
    project-specific task

    View full-size slide

  54. What if I don’t want to commit changes?
    Feature #2

    View full-size slide

  55. ---
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: my.docker.registry:5080/java-maven
    tag: latest
    inputs:
    - name: source-code
    run:
    path: mvn
    args: [ deploy ]
    dir: source-code
    ~/workspace/task-build.yml

    View full-size slide

  56. fly -t office e -c ~/workspace/task-build.yml -i source-code=.
    Command “execute”
    (or, “e” as alias)

    View full-size slide

  57. fly -t office e -c ~/workspace/task-build.yml -i source-code=.
    Which local task to
    execute

    View full-size slide

  58. fly -t office e -c ~/workspace/task-build.yml -i source-code=.
    Pass path as
    mandatory input
    parameter (“.” in this
    case)

    View full-size slide

  59. Fly: execute one task
    Concourse CI server
    DELL
    Dev local computer
    fly -t office e -c ~/workspace/task-build.yml -i source-code=.
    Source code
    task-build.yml

    View full-size slide

  60. Show all the builds

    View full-size slide

  61. Functional & integration tests?

    View full-size slide

  62. What is the problem?
    1. Set up whole system
    2. Wait when it is ready
    3. Run tests
    4. Close system

    View full-size slide

  63. Service
    Database ...
    ...
    ...
    Nginx
    Web Tests

    View full-size slide

  64. One big uber container?

    View full-size slide

  65. Service
    Database ...
    ...
    ...
    Nginx

    View full-size slide

  66. Service
    Database ...
    ...
    ...
    Nginx

    View full-size slide

  67. Docker in Docker?

    View full-size slide

  68. dind - Docker in Docker
    Docker image that allows to run docker inside. For example,
    concourse/docker-image-resource:

    GitHub: https://github.com/concourse/docker-image-resource

    common.sh:
    https://github.com/concourse/docker-image-resource/blob/master/assets/common.sh
    mumoshu/dcind:

    GitHub: https://github.com/mumoshu/dcind
    Discussions are in the GitHub issue: https://github.com/concourse/concourse/issues/324

    View full-size slide

  69. Task to run functional tests:

    View full-size slide

  70. task.yml
    ---
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: mumoshu/dcind
    tag: latest
    inputs:
    - name: common-ci-images
    - name: source-code
    run:
    path: common-ci-images/run-functional-tests.sh

    View full-size slide

  71. task.yml
    ---
    platform: linux
    image_resource:
    type: docker-image
    source:
    repository: mumoshu/dcind
    tag: latest
    inputs:
    - name: common-ci-images
    - name: source-code
    run:
    path: common-ci-images/run-functional-tests.sh

    View full-size slide

  72. Run-functional-tests.sh (1: start docker)
    #!/bin/sh
    # Start docker service (script comes from image)
    source /docker-lib.sh
    start_docker

    View full-size slide

  73. Run-functional-tests.sh (2: run production system)
    cd source-code
    # run production system in detached mode
    docker-compose up -d
    # check that services are running
    IS_CORE_RUNNING=$(docker inspect core |
    jq -r '.[] | .State.Running')

    if [ "$IS_CORE_RUNNING" = false ] || … ;then
    exit 1;
    fi

    View full-size slide

  74. Run-functional-tests.sh (3: run functional tests)
    cd ../source-code
    ABS_SOURCE=$(pwd .)
    docker run -it -w /tests -v "$ABS_SOURCE":/tests \
    w32blaster/selenium-gradle \
    xvfb-daemon-run gradle -q runAllTests
    NGINX_VIRTUAL_IP=$(docker inspect nginx |
    jq -r '.[] | .NetworkSettings.Networks.bridge.IPAddress')
    -e "NGINX_VIRTUAL_IP=$NGINX_VIRTUAL_IP" \
    --add-host www.devclub.eu:"$NGINX_VIRTUAL_IP" \

    View full-size slide

  75. Run-functional-tests.sh (4: print the results)
    TEST_RESULTS=$?
    if [ $TEST_RESULTS -eq 0 ];then
    printf "The Functional tests are successful!"
    else
    printf "The Functional Tests are failed!"
    fi

    View full-size slide

  76. Run-functional-tests.sh (5: save logs and reports)
    # save logs and reports
    docker logs core > build/reports/core.log
    docker logs mongodb > build/reports/mongodb.log

    tar cfz test-results.tar.gz build/reports
    # save test-results.tar.gz somewhere
    # using scp, ftp, rsync or any other way you prefer

    View full-size slide

  77. Run-functional-tests.sh (6: stop services, exit)
    # stop the running containers, clean up etc...
    docker-compose stop
    # stop the docker service
    cd ..
    stop_docker
    # exit
    exit $TEST_RESULTS

    View full-size slide

  78. Atom plugin
    https://atom.io/packages/concourse-vis

    View full-size slide

  79. ConcourseCI Radiator
    https://github.com/moodev/concourseci-radiator

    View full-size slide

  80. Disadvantages

    The project is young, not so many information in the network

    You have to learn new YAML configuration (although, it’s easy)

    Can’t pass results between jobs (we need to deploy/push it somewhere)

    Few plugins so far

    Takes time to start from scratch

    Requires a lot of disk space:
    - /var/lib/docker - Docker Registry storage
    - /data/worker-state - inside a worker container. Needs to be mapped to
    large partition

    View full-size slide

  81. Advantages

    Forces to use local mirrors for a repository/registry

    Forces to use tiny containers (alpine, busybox etc…)

    Easy to scale horizontally

    Ideal for microservices (different languages/platforms)

    Very active project

    Requires Linux and Docker skills

    View full-size slide

  82. The End
    Drink a coffee, make a Java!

    View full-size slide