Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction to ConcourseCI

Introduction to ConcourseCI

Presentation from Devclub.eu (30.11.2016, Tallinn). Video is here (rus): https://www.youtube.com/watch?v=JtbR2bTmjkI

Links:
- ConcourseCI website: http://concourse.ci/
- concourse/docker-image-resource: https://github.com/concourse/docker-image-resource
- (old) mumoshu/dcind: https://github.com/mumoshu/dcind
- (updated) I use this image to run Docker in Docker: https://hub.docker.com/r/w32blaster/dcind/
- Atom Plugin: https://atom.io/packages/concourse-vis
- ConcourseCI Radiator: https://github.com/moodev/concourseci-radiator

The full working script to run Functional Tests is here:
https://gist.github.com/w32blaster/965bad80a6613fe924eb2f82aa159e4f

460df13d9de7c4d2d7c39598ad3e0c88?s=128

Ilja Hämäläinen

November 30, 2016
Tweet

Transcript

  1. Concourse CI Clear Prop! Ilja Hämäläinen 2016

  2. None
  3. Concourse CI Clear Prop! Ilja Hämäläinen 2016

  4. HOW?

  5. WHY?

  6. Need a CI server?

  7. None
  8. Why ConcourseCI? 1. Pipelines are first-class citizens 2. Containers 3.

    Everything is a code (automation)
  9. Let’s start learning

  10. 3 concepts • Resources Gives a sources (code, binaries, files…)

    No actions • Tasks Instructions, how something should be run. • Jobs Combines tasks and resources to get some result.
  11. Resources Job Running Jobs Pipeline

  12. How to talk with the Concourse?

  13. fly - Command Line Interface

  14. fly -h Available commands: abort-build Abort a build (aliases: ab)

    builds List builds data (aliases: bs) check-resource Check a resource (aliases: cr) checklist Print a Checkfile of the given pipeline (aliases: cl) containers Print the active containers (aliases: cs) destroy-pipeline Destroy a pipeline (aliases: dp) execute Execute a one-off build using local bits (aliases: e) get-pipeline Get a pipeline's current configuration (aliases: gp) help Print this help message hijack Execute a command in a container (aliases: intercept, i) login Authenticate with the target (aliases: l) pause-job Pause a job (aliases: pj) pause-pipeline Pause a pipeline (aliases: pp) pipelines List the configured pipelines (aliases: ps) rename-pipeline Rename a pipeline (aliases: rp) set-pipeline Create or update a pipeline's configuration (aliases: sp) sync Download and replace the current fly from the target (aliases: s) targets List saved targets (aliases: ts) trigger-job Start a job in a pipeline (aliases: tj) unpause-job Unpause a job (aliases: uj) unpause-pipeline Un-pause a pipeline (aliases: up) volumes List the active volumes (aliases: vs) watch Stream a build's output (aliases: w) workers List the registered workers (aliases: ws)
  15. fly: login to server (only the first time)

  16. Fly: login to a server (1) fly --target office ...

    Custom server name
  17. Fly: login to a server (1) fly --target office login

    ... Command to login
  18. Fly: login to a server (1) fly --target office login

    --concourse-url http://concourse.myhost.com URL of our internal ConcoueseCI
  19. Fly: login to a server (2) $ cat ~/.flyrc targets:

    office: api: http://concourse.my-host.com token: type: Bearer value: <long token, skipping for simplicity> tutorial: api: http://localhost:8080 token: type: "" value: "" Server on the my-host.com Server on my localhost
  20. fly: create a new pipeline

  21. Fly: create new pipeline (1) fly sp ... alias for

    “set-pipiline”
  22. Fly: create new pipeline (1) fly sp -t office ...

    Execute this command on the “office” server
  23. Fly: create new pipeline (1) fly sp -t office -c

    pipeline.yml ... The pipeline itself in YAML format (config file)
  24. Fly: create new pipeline (1) fly sp -t office -c

    pipeline.yml -p my-pipeline The custom name for the pipeline. Will appear in GUI
  25. Fly: create new pipeline (2) .... job job-findbug has been

    added: name: job-findbug public: true plan: - get: git-resource passed: - job-build trigger: true - get: common-images passed: - job-build - task: task-findbug file: common-images/java-findbugs/task-findbugs.yml params: APP_TO_BE_CHECKED: /git-resource/webapp/target/service-registry-api.war apply configuration? [yN]:
  26. Fly: create new pipeline (3) Concourse CI server DELL Dev

    local computer fly sp -t office -c pipeline.yml -p my-pipeline pipeline.yml
  27. Available pipelines Pipeline

  28. How to make a pipeline.yml?

  29. 1) Specify resource resources: - name: git-resource type: git source:

    uri: http://git.myhost.com/<...>.git The custom name The resource type Git repo URL
  30. 2) Task --- platform: linux image_resource: type: docker-image source: {repository:

    busybox} run: path: echo args: [hello world] Set the Docker image The command to be executed inside the container
  31. 3) Job jobs: - name: job-build plan: - get: git-resource

    - task: task-1 ...<task body from previous slide>... - put: slack-resource
  32. Job name Previous builds Button “run new build manually” Resource(s)

    Task(s) Resource(s)
  33. Example pipelines

  34. Scenario #1: 1. Get the code from Git 2. Build

    it (using Maven)
  35. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git

    jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven insecure_registries: ["my.docker.registry:5080"] tag: latest inputs: - name: git-resource run: path: mvn args: [ deploy ] 1. Declare resources
  36. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git

    jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven insecure_registries: ["my.docker.registry:5080"] tag: latest inputs: - name: git-resource run: path: mvn args: [ deploy ] 2. Declare one job “job-build” 3. Set plan, which...
  37. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git

    jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven insecure_registries: ["my.docker.registry:5080"] tag: latest inputs: - name: git-resource run: path: mvn args: [ deploy ] a) gets the code from the declared resource (triggering on every new commit)
  38. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git

    jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven inputs: - name: git-resource run: path: mvn args: [ deploy ] b) executes one task, which... ...calls the command “mvn deploy” ...inside the container
  39. Your script should return standard UNIX status code (0 =

    success) You can run any executable in your task ... run: path: any-executable-you-like.sh
  40. Input, output for a task

  41. --- platform: linux # # General task to build a

    Maven based project. # It requires the pom.xml file at the root # image_resource: type: docker-image source: repository: alpine tag: latest inputs: - name: git-resource outputs: - name: result run: path: mvn args: [ deploy ] Inputs are used to pass some files inside a task. There will be /git-resource directory created in the container Outputs to send something to other tasks. There will be /result directory created in the container.
  42. Scenario #2: Inputs and Outputs 1. Generate some content in

    Task1 2. Access this content from Task2
  43. --- jobs: - name: job-build plan: - task: step-1 ...

    outputs: - name: shared-dir run: path: sh args: - -exc - | echo “test” > shared-dir/hello.txt - task: step-2 ... inputs: - name: shared-dir run: path: sh args: - -exc - | ls shared-dir && shared-dir/hello.txt 1. There is the directory shared-dir in the container 2. Save a file in this directory
  44. --- jobs: - name: job-build plan: - task: step-1 ...

    outputs: - name: shared-dir run: path: sh args: - -exc - | echo “test” > shared-dir/hello.txt - task: step-2 ... inputs: - name: shared-dir run: path: sh args: - -exc - | ls shared-dir && cat shared-dir/hello.txt 3. There is the directory shared-dir in the container, with inputs data (file hello.txt) 4. Demonstrate that we have access to a file, created in previous task (in another container!)
  45. 1. Task “Step-1” pulls the Docker image 2. Creates file

    “hello.txt” 3. Task “Step-2” prints Content of hello.txt
  46. What is happening inside a container?

  47. To view a task output you have three choices: 1.

    From the GUI 2. fly watch 3. fly hijack
  48. Extracted tasks

  49. Firstly, let’s extract a task to a separate file...

  50. 1. Task could be stored in separate file pipeline.yml: jobs:

    - name: job-build plan: - get: git-resource - task: step-1 platform: linux image_reource: <...> run: <...> task.yml: --- platform: linux image_resource: <...> run: < some commands >
  51. 1. Task could be stored in separate file pipeline.yml: jobs:

    - name: job-build plan: - get: git-resource - task: step-1 file: path/to/task.yml task.yml: --- platform: linux image_resource: <...> run: < some commands >
  52. ...and we can reuse common tasks and images! Feature #1

  53. --- platform: linux # # General task to build a

    Maven based project. # It requires the pom.xml file at the root # image_resource: type: docker-image source: repository: my.docker.registry/java-maven inputs: - name: git-resource run: path: mvn args: [ deploy ] dir: git-resource Example of a common task for a Java project
  54. For example....

  55. Scenario #3: Common tasks 1. Get resource from common-tasks repo

    2. Execute common task from resource 3. Execute embedded task
  56. --- resources: - name: common-images type: git source: uri: http://git.myhost.com/common-ci-images.git

    jobs: - name: job-build plan: - get: common-images - task: task-build file: common-images/task-build.yml - task: task-deploy ... run: path: sh args: - -exc - | echo “Hello Devs” Resource 1: some project repository
  57. --- resources: - name: common-images type: git source: uri: http://git.myhost.com/common-ci-images.git

    jobs: - name: job-build plan: - get: common-images - task: task-build file: common-images/task-build.yml - task: task-deploy ... run: path: sh args: - -exc - | echo “Hello Devs” Get resource Build a project, using common task Perform some project-specific task
  58. What if I don’t want to commit changes? Feature #2

  59. fly execute

  60. --- platform: linux image_resource: type: docker-image source: repository: my.docker.registry:5080/java-maven tag:

    latest inputs: - name: source-code run: path: mvn args: [ deploy ] dir: source-code ~/workspace/task-build.yml
  61. then...

  62. fly -t office e -c ~/workspace/task-build.yml -i source-code=. Command “execute”

    (or, “e” as alias)
  63. fly -t office e -c ~/workspace/task-build.yml -i source-code=. Which local

    task to execute
  64. fly -t office e -c ~/workspace/task-build.yml -i source-code=. Pass path

    as mandatory input parameter (“.” in this case)
  65. Fly: execute one task Concourse CI server DELL Dev local

    computer fly -t office e -c ~/workspace/task-build.yml -i source-code=. Source code task-build.yml
  66. Show all the builds

  67. Functional & integration tests?

  68. What is the problem? 1. Set up whole system 2.

    Wait when it is ready 3. Run tests 4. Close system
  69. Service Database ... ... ... Nginx Web Tests

  70. One big uber container?

  71. None
  72. Service Database ... ... ... Nginx

  73. Service Database ... ... ... Nginx

  74. Docker in Docker?

  75. dind - Docker in Docker Docker image that allows to

    run docker inside. For example, concourse/docker-image-resource: • GitHub: https://github.com/concourse/docker-image-resource • common.sh: https://github.com/concourse/docker-image-resource/blob/master/assets/common.sh mumoshu/dcind: • GitHub: https://github.com/mumoshu/dcind Discussions are in the GitHub issue: https://github.com/concourse/concourse/issues/324
  76. Task to run functional tests:

  77. task.yml --- platform: linux image_resource: type: docker-image source: repository: mumoshu/dcind

    tag: latest inputs: - name: common-ci-images - name: source-code run: path: common-ci-images/run-functional-tests.sh
  78. None
  79. task.yml --- platform: linux image_resource: type: docker-image source: repository: mumoshu/dcind

    tag: latest inputs: - name: common-ci-images - name: source-code run: path: common-ci-images/run-functional-tests.sh
  80. Run-functional-tests.sh (1: start docker) #!/bin/sh # Start docker service (script

    comes from image) source /docker-lib.sh start_docker
  81. Run-functional-tests.sh (2: run production system) cd source-code # run production

    system in detached mode docker-compose up -d # check that services are running IS_CORE_RUNNING=$(docker inspect core | jq -r '.[] | .State.Running') … if [ "$IS_CORE_RUNNING" = false ] || … ;then exit 1; fi
  82. Run-functional-tests.sh (3: run functional tests) cd ../source-code ABS_SOURCE=$(pwd .) docker

    run -it -w /tests -v "$ABS_SOURCE":/tests \ w32blaster/selenium-gradle \ xvfb-daemon-run gradle -q runAllTests NGINX_VIRTUAL_IP=$(docker inspect nginx | jq -r '.[] | .NetworkSettings.Networks.bridge.IPAddress') -e "NGINX_VIRTUAL_IP=$NGINX_VIRTUAL_IP" \ --add-host www.devclub.eu:"$NGINX_VIRTUAL_IP" \
  83. Run-functional-tests.sh (4: print the results) TEST_RESULTS=$? if [ $TEST_RESULTS -eq

    0 ];then printf "The Functional tests are successful!" else printf "The Functional Tests are failed!" fi
  84. Run-functional-tests.sh (5: save logs and reports) # save logs and

    reports docker logs core > build/reports/core.log docker logs mongodb > build/reports/mongodb.log … tar cfz test-results.tar.gz build/reports # save test-results.tar.gz somewhere # using scp, ftp, rsync or any other way you prefer
  85. Run-functional-tests.sh (6: stop services, exit) # stop the running containers,

    clean up etc... docker-compose stop # stop the docker service cd .. stop_docker # exit exit $TEST_RESULTS
  86. None
  87. Atom plugin https://atom.io/packages/concourse-vis

  88. ConcourseCI Radiator https://github.com/moodev/concourseci-radiator

  89. None
  90. Disadvantages • The project is young, not so many information

    in the network • You have to learn new YAML configuration (although, it’s easy) • Can’t pass results between jobs (we need to deploy/push it somewhere) • Few plugins so far • Takes time to start from scratch • Requires a lot of disk space: - /var/lib/docker - Docker Registry storage - /data/worker-state - inside a worker container. Needs to be mapped to large partition
  91. Advantages • Forces to use local mirrors for a repository/registry

    • Forces to use tiny containers (alpine, busybox etc…) • Easy to scale horizontally • Ideal for microservices (different languages/platforms) • Very active project • Requires Linux and Docker skills
  92. The End Drink a coffee, make a Java!