Introduction to ConcourseCI

Presentation from Devclub.eu (30.11.2016, Tallinn). Video is here (rus): https://www.youtube.com/watch?v=JtbR2bTmjkI

Links:
- ConcourseCI website: http://concourse.ci/
- concourse/docker-image-resource: https://github.com/concourse/docker-image-resource
- (old) mumoshu/dcind: https://github.com/mumoshu/dcind
- (updated) I use this image to run Docker in Docker: https://hub.docker.com/r/w32blaster/dcind/
- Atom Plugin: https://atom.io/packages/concourse-vis
- ConcourseCI Radiator: https://github.com/moodev/concourseci-radiator

The full working script to run Functional Tests is here:
https://gist.github.com/w32blaster/965bad80a6613fe924eb2f82aa159e4f

Transcript

1. Concourse CI Clear Prop! Ilja Hämäläinen 2016
3. Concourse CI Clear Prop! Ilja Hämäläinen 2016
4. HOW?
5. WHY?
6. Need a CI server?
8. Why ConcourseCI? 1. Pipelines are first-class citizens 2. Containers 3. Everything is a code (automation)
9. Let’s start learning
10. 3 concepts • Resources Gives a sources (code, binaries, files…) No actions • Tasks Instructions, how something should be run. • Jobs Combines tasks and resources to get some result.
11. Resources Job Running Jobs Pipeline
12. How to talk with the Concourse?
13. fly - Command Line Interface
14. fly -h Available commands: abort-build Abort a build (aliases: ab) builds List builds data (aliases: bs) check-resource Check a resource (aliases: cr) checklist Print a Checkfile of the given pipeline (aliases: cl) containers Print the active containers (aliases: cs) destroy-pipeline Destroy a pipeline (aliases: dp) execute Execute a one-off build using local bits (aliases: e) get-pipeline Get a pipeline's current configuration (aliases: gp) help Print this help message hijack Execute a command in a container (aliases: intercept, i) login Authenticate with the target (aliases: l) pause-job Pause a job (aliases: pj) pause-pipeline Pause a pipeline (aliases: pp) pipelines List the configured pipelines (aliases: ps) rename-pipeline Rename a pipeline (aliases: rp) set-pipeline Create or update a pipeline's configuration (aliases: sp) sync Download and replace the current fly from the target (aliases: s) targets List saved targets (aliases: ts) trigger-job Start a job in a pipeline (aliases: tj) unpause-job Unpause a job (aliases: uj) unpause-pipeline Un-pause a pipeline (aliases: up) volumes List the active volumes (aliases: vs) watch Stream a build's output (aliases: w) workers List the registered workers (aliases: ws)
15. fly: login to server (only the first time)
16. Fly: login to a server (1) fly --target office ... Custom server name
17. Fly: login to a server (1) fly --target office login ... Command to login
18. Fly: login to a server (1) fly --target office login --concourse-url http://concourse.myhost.com URL of our internal ConcoueseCI
19. Fly: login to a server (2) $ cat ~/.flyrc targets: office: api: http://concourse.my-host.com token: type: Bearer value: <long token, skipping for simplicity> tutorial: api: http://localhost:8080 token: type: "" value: "" Server on the my-host.com Server on my localhost
20. fly: create a new pipeline
21. Fly: create new pipeline (1) fly sp ... alias for “set-pipiline”
22. Fly: create new pipeline (1) fly sp -t office ... Execute this command on the “office” server
23. Fly: create new pipeline (1) fly sp -t office -c pipeline.yml ... The pipeline itself in YAML format (config file)
24. Fly: create new pipeline (1) fly sp -t office -c pipeline.yml -p my-pipeline The custom name for the pipeline. Will appear in GUI
25. Fly: create new pipeline (2) .... job job-findbug has been added: name: job-findbug public: true plan: - get: git-resource passed: - job-build trigger: true - get: common-images passed: - job-build - task: task-findbug file: common-images/java-findbugs/task-findbugs.yml params: APP_TO_BE_CHECKED: /git-resource/webapp/target/service-registry-api.war apply configuration? [yN]:
26. Fly: create new pipeline (3) Concourse CI server DELL Dev local computer fly sp -t office -c pipeline.yml -p my-pipeline pipeline.yml
27. Available pipelines Pipeline
28. How to make a pipeline.yml?
29. 1) Specify resource resources: - name: git-resource type: git source: uri: http://git.myhost.com/<...>.git The custom name The resource type Git repo URL
30. 2) Task --- platform: linux image_resource: type: docker-image source: {repository: busybox} run: path: echo args: [hello world] Set the Docker image The command to be executed inside the container
31. 3) Job jobs: - name: job-build plan: - get: git-resource - task: task-1 ...<task body from previous slide>... - put: slack-resource
32. Job name Previous builds Button “run new build manually” Resource(s) Task(s) Resource(s)
33. Example pipelines
34. Scenario #1: 1. Get the code from Git 2. Build it (using Maven)
35. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven insecure_registries: ["my.docker.registry:5080"] tag: latest inputs: - name: git-resource run: path: mvn args: [ deploy ] 1. Declare resources
36. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven insecure_registries: ["my.docker.registry:5080"] tag: latest inputs: - name: git-resource run: path: mvn args: [ deploy ] 2. Declare one job “job-build” 3. Set plan, which...
37. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven insecure_registries: ["my.docker.registry:5080"] tag: latest inputs: - name: git-resource run: path: mvn args: [ deploy ] a) gets the code from the declared resource (triggering on every new commit)
38. --- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven inputs: - name: git-resource run: path: mvn args: [ deploy ] b) executes one task, which... ...calls the command “mvn deploy” ...inside the container
39. Your script should return standard UNIX status code (0 = success) You can run any executable in your task ... run: path: any-executable-you-like.sh
40. Input, output for a task
41. --- platform: linux # # General task to build a Maven based project. # It requires the pom.xml file at the root # image_resource: type: docker-image source: repository: alpine tag: latest inputs: - name: git-resource outputs: - name: result run: path: mvn args: [ deploy ] Inputs are used to pass some files inside a task. There will be /git-resource directory created in the container Outputs to send something to other tasks. There will be /result directory created in the container.
42. Scenario #2: Inputs and Outputs 1. Generate some content in Task1 2. Access this content from Task2
43. --- jobs: - name: job-build plan: - task: step-1 ... outputs: - name: shared-dir run: path: sh args: - -exc - | echo “test” > shared-dir/hello.txt - task: step-2 ... inputs: - name: shared-dir run: path: sh args: - -exc - | ls shared-dir && shared-dir/hello.txt 1. There is the directory shared-dir in the container 2. Save a file in this directory
44. --- jobs: - name: job-build plan: - task: step-1 ... outputs: - name: shared-dir run: path: sh args: - -exc - | echo “test” > shared-dir/hello.txt - task: step-2 ... inputs: - name: shared-dir run: path: sh args: - -exc - | ls shared-dir && cat shared-dir/hello.txt 3. There is the directory shared-dir in the container, with inputs data (file hello.txt) 4. Demonstrate that we have access to a file, created in previous task (in another container!)
45. 1. Task “Step-1” pulls the Docker image 2. Creates file “hello.txt” 3. Task “Step-2” prints Content of hello.txt
46. What is happening inside a container?
47. To view a task output you have three choices: 1. From the GUI 2. fly watch 3. fly hijack
48. Extracted tasks
49. Firstly, let’s extract a task to a separate file...
50. 1. Task could be stored in separate file pipeline.yml: jobs: - name: job-build plan: - get: git-resource - task: step-1 platform: linux image_reource: <...> run: <...> task.yml: --- platform: linux image_resource: <...> run: < some commands >
51. 1. Task could be stored in separate file pipeline.yml: jobs: - name: job-build plan: - get: git-resource - task: step-1 file: path/to/task.yml task.yml: --- platform: linux image_resource: <...> run: < some commands >
52. ...and we can reuse common tasks and images! Feature #1
53. --- platform: linux # # General task to build a Maven based project. # It requires the pom.xml file at the root # image_resource: type: docker-image source: repository: my.docker.registry/java-maven inputs: - name: git-resource run: path: mvn args: [ deploy ] dir: git-resource Example of a common task for a Java project
54. For example....
55. Scenario #3: Common tasks 1. Get resource from common-tasks repo 2. Execute common task from resource 3. Execute embedded task
56. --- resources: - name: common-images type: git source: uri: http://git.myhost.com/common-ci-images.git jobs: - name: job-build plan: - get: common-images - task: task-build file: common-images/task-build.yml - task: task-deploy ... run: path: sh args: - -exc - | echo “Hello Devs” Resource 1: some project repository
57. --- resources: - name: common-images type: git source: uri: http://git.myhost.com/common-ci-images.git jobs: - name: job-build plan: - get: common-images - task: task-build file: common-images/task-build.yml - task: task-deploy ... run: path: sh args: - -exc - | echo “Hello Devs” Get resource Build a project, using common task Perform some project-specific task
58. What if I don’t want to commit changes? Feature #2
59. fly execute
60. --- platform: linux image_resource: type: docker-image source: repository: my.docker.registry:5080/java-maven tag: latest inputs: - name: source-code run: path: mvn args: [ deploy ] dir: source-code ~/workspace/task-build.yml
61. then...
62. fly -t office e -c ~/workspace/task-build.yml -i source-code=. Command “execute” (or, “e” as alias)
63. fly -t office e -c ~/workspace/task-build.yml -i source-code=. Which local task to execute
64. fly -t office e -c ~/workspace/task-build.yml -i source-code=. Pass path as mandatory input parameter (“.” in this case)
65. Fly: execute one task Concourse CI server DELL Dev local computer fly -t office e -c ~/workspace/task-build.yml -i source-code=. Source code task-build.yml
66. Show all the builds
67. Functional & integration tests?
68. What is the problem? 1. Set up whole system 2. Wait when it is ready 3. Run tests 4. Close system
69. Service Database ... ... ... Nginx Web Tests
70. One big uber container?
72. Service Database ... ... ... Nginx
73. Service Database ... ... ... Nginx
74. Docker in Docker?
75. dind - Docker in Docker Docker image that allows to run docker inside. For example, concourse/docker-image-resource: • GitHub: https://github.com/concourse/docker-image-resource • common.sh: https://github.com/concourse/docker-image-resource/blob/master/assets/common.sh mumoshu/dcind: • GitHub: https://github.com/mumoshu/dcind Discussions are in the GitHub issue: https://github.com/concourse/concourse/issues/324
76. Task to run functional tests:
77. task.yml --- platform: linux image_resource: type: docker-image source: repository: mumoshu/dcind tag: latest inputs: - name: common-ci-images - name: source-code run: path: common-ci-images/run-functional-tests.sh
79. task.yml --- platform: linux image_resource: type: docker-image source: repository: mumoshu/dcind tag: latest inputs: - name: common-ci-images - name: source-code run: path: common-ci-images/run-functional-tests.sh
80. Run-functional-tests.sh (1: start docker) #!/bin/sh # Start docker service (script comes from image) source /docker-lib.sh start_docker
81. Run-functional-tests.sh (2: run production system) cd source-code # run production system in detached mode docker-compose up -d # check that services are running IS_CORE_RUNNING=$(docker inspect core | jq -r '.[] | .State.Running') … if [ "$IS_CORE_RUNNING" = false ] || … ;then exit 1; fi
82. Run-functional-tests.sh (3: run functional tests) cd ../source-code ABS_SOURCE=$(pwd .) docker run -it -w /tests -v "$ABS_SOURCE":/tests \ w32blaster/selenium-gradle \ xvfb-daemon-run gradle -q runAllTests NGINX_VIRTUAL_IP=$(docker inspect nginx | jq -r '.[] | .NetworkSettings.Networks.bridge.IPAddress') -e "NGINX_VIRTUAL_IP=$NGINX_VIRTUAL_IP" \ --add-host www.devclub.eu:"$NGINX_VIRTUAL_IP" \
83. Run-functional-tests.sh (4: print the results) TEST_RESULTS=$? if [ $TEST_RESULTS -eq 0 ];then printf "The Functional tests are successful!" else printf "The Functional Tests are failed!" fi
84. Run-functional-tests.sh (5: save logs and reports) # save logs and reports docker logs core > build/reports/core.log docker logs mongodb > build/reports/mongodb.log … tar cfz test-results.tar.gz build/reports # save test-results.tar.gz somewhere # using scp, ftp, rsync or any other way you prefer
85. Run-functional-tests.sh (6: stop services, exit) # stop the running containers, clean up etc... docker-compose stop # stop the docker service cd .. stop_docker # exit exit $TEST_RESULTS
87. Atom plugin https://atom.io/packages/concourse-vis
88. ConcourseCI Radiator https://github.com/moodev/concourseci-radiator
90. Disadvantages • The project is young, not so many information in the network • You have to learn new YAML configuration (although, it’s easy) • Can’t pass results between jobs (we need to deploy/push it somewhere) • Few plugins so far • Takes time to start from scratch • Requires a lot of disk space: - /var/lib/docker - Docker Registry storage - /data/worker-state - inside a worker container. Needs to be mapped to large partition
91. Advantages • Forces to use local mirrors for a repository/registry • Forces to use tiny containers (alpine, busybox etc…) • Easy to scale horizontally • Ideal for microservices (different languages/platforms) • Very active project • Requires Linux and Docker skills
92. The End Drink a coffee, make a Java!