Introduction to ConcourseCI

Slide 1

Slide 1 text

Concourse CI Clear Prop! Ilja Hämäläinen 2016

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Concourse CI Clear Prop! Ilja Hämäläinen 2016

Slide 4

Slide 4 text

HOW?

Slide 5

Slide 5 text

WHY?

Slide 6

Slide 6 text

Need a CI server?

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Why ConcourseCI? 1. Pipelines are first-class citizens 2. Containers 3. Everything is a code (automation)

Slide 9

Slide 9 text

Let’s start learning

Slide 10

Slide 10 text

3 concepts ● Resources Gives a sources (code, binaries, files…) No actions ● Tasks Instructions, how something should be run. ● Jobs Combines tasks and resources to get some result.

Slide 11

Slide 11 text

Resources Job Running Jobs Pipeline

Slide 12

Slide 12 text

How to talk with the Concourse?

Slide 13

Slide 13 text

fly - Command Line Interface

Slide 14

Slide 14 text

fly -h Available commands: abort-build Abort a build (aliases: ab) builds List builds data (aliases: bs) check-resource Check a resource (aliases: cr) checklist Print a Checkfile of the given pipeline (aliases: cl) containers Print the active containers (aliases: cs) destroy-pipeline Destroy a pipeline (aliases: dp) execute Execute a one-off build using local bits (aliases: e) get-pipeline Get a pipeline's current configuration (aliases: gp) help Print this help message hijack Execute a command in a container (aliases: intercept, i) login Authenticate with the target (aliases: l) pause-job Pause a job (aliases: pj) pause-pipeline Pause a pipeline (aliases: pp) pipelines List the configured pipelines (aliases: ps) rename-pipeline Rename a pipeline (aliases: rp) set-pipeline Create or update a pipeline's configuration (aliases: sp) sync Download and replace the current fly from the target (aliases: s) targets List saved targets (aliases: ts) trigger-job Start a job in a pipeline (aliases: tj) unpause-job Unpause a job (aliases: uj) unpause-pipeline Un-pause a pipeline (aliases: up) volumes List the active volumes (aliases: vs) watch Stream a build's output (aliases: w) workers List the registered workers (aliases: ws)

Slide 15

Slide 15 text

fly: login to server (only the first time)

Slide 16

Slide 16 text

Fly: login to a server (1) fly --target office ... Custom server name

Slide 17

Slide 17 text

Fly: login to a server (1) fly --target office login ... Command to login

Slide 18

Slide 18 text

Fly: login to a server (1) fly --target office login --concourse-url http://concourse.myhost.com URL of our internal ConcoueseCI

Slide 19

Slide 19 text

Fly: login to a server (2) $ cat ~/.flyrc targets: office: api: http://concourse.my-host.com token: type: Bearer value: tutorial: api: http://localhost:8080 token: type: "" value: "" Server on the my-host.com Server on my localhost

Slide 20

Slide 20 text

fly: create a new pipeline

Slide 21

Slide 21 text

Fly: create new pipeline (1) fly sp ... alias for “set-pipiline”

Slide 22

Slide 22 text

Fly: create new pipeline (1) fly sp -t office ... Execute this command on the “office” server

Slide 23

Slide 23 text

Fly: create new pipeline (1) fly sp -t office -c pipeline.yml ... The pipeline itself in YAML format (config file)

Slide 24

Slide 24 text

Fly: create new pipeline (1) fly sp -t office -c pipeline.yml -p my-pipeline The custom name for the pipeline. Will appear in GUI

Slide 25

Slide 25 text

Fly: create new pipeline (2) .... job job-findbug has been added: name: job-findbug public: true plan: - get: git-resource passed: - job-build trigger: true - get: common-images passed: - job-build - task: task-findbug file: common-images/java-findbugs/task-findbugs.yml params: APP_TO_BE_CHECKED: /git-resource/webapp/target/service-registry-api.war apply configuration? [yN]:

Slide 26

Slide 26 text

Fly: create new pipeline (3) Concourse CI server DELL Dev local computer fly sp -t office -c pipeline.yml -p my-pipeline pipeline.yml

Slide 27

Slide 27 text

Available pipelines Pipeline

Slide 28

Slide 28 text

How to make a pipeline.yml?

Slide 29

Slide 29 text

1) Specify resource resources: - name: git-resource type: git source: uri: http://git.myhost.com/<...>.git The custom name The resource type Git repo URL

Slide 30

Slide 30 text

2) Task --- platform: linux image_resource: type: docker-image source: {repository: busybox} run: path: echo args: [hello world] Set the Docker image The command to be executed inside the container

Slide 31

Slide 31 text

3) Job jobs: - name: job-build plan: - get: git-resource - task: task-1 ...... - put: slack-resource

Slide 32

Slide 32 text

Job name Previous builds Button “run new build manually” Resource(s) Task(s) Resource(s)

Slide 33

Slide 33 text

Example pipelines

Slide 34

Slide 34 text

Scenario #1: 1. Get the code from Git 2. Build it (using Maven)

Slide 35

Slide 35 text

--- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven insecure_registries: ["my.docker.registry:5080"] tag: latest inputs: - name: git-resource run: path: mvn args: [ deploy ] 1. Declare resources

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

--- resources: - name: git-resource type: git source: uri: http://git.myhost.com/myproject.git jobs: - name: job-build plan: - get: git-resource trigger: true - task: task-build platform: linux image_resource: type: docker-image source: repository: my.docker.registry/java-maven inputs: - name: git-resource run: path: mvn args: [ deploy ] b) executes one task, which... ...calls the command “mvn deploy” ...inside the container

Slide 39

Slide 39 text

Your script should return standard UNIX status code (0 = success) You can run any executable in your task ... run: path: any-executable-you-like.sh

Slide 40

Slide 40 text

Input, output for a task

Slide 41

Slide 41 text

--- platform: linux # # General task to build a Maven based project. # It requires the pom.xml file at the root # image_resource: type: docker-image source: repository: alpine tag: latest inputs: - name: git-resource outputs: - name: result run: path: mvn args: [ deploy ] Inputs are used to pass some files inside a task. There will be /git-resource directory created in the container Outputs to send something to other tasks. There will be /result directory created in the container.

Slide 42

Slide 42 text

Scenario #2: Inputs and Outputs 1. Generate some content in Task1 2. Access this content from Task2

Slide 43

Slide 43 text

--- jobs: - name: job-build plan: - task: step-1 ... outputs: - name: shared-dir run: path: sh args: - -exc - | echo “test” > shared-dir/hello.txt - task: step-2 ... inputs: - name: shared-dir run: path: sh args: - -exc - | ls shared-dir && shared-dir/hello.txt 1. There is the directory shared-dir in the container 2. Save a file in this directory

Slide 44

Slide 44 text

--- jobs: - name: job-build plan: - task: step-1 ... outputs: - name: shared-dir run: path: sh args: - -exc - | echo “test” > shared-dir/hello.txt - task: step-2 ... inputs: - name: shared-dir run: path: sh args: - -exc - | ls shared-dir && cat shared-dir/hello.txt 3. There is the directory shared-dir in the container, with inputs data (file hello.txt) 4. Demonstrate that we have access to a file, created in previous task (in another container!)

Slide 45

Slide 45 text

1. Task “Step-1” pulls the Docker image 2. Creates file “hello.txt” 3. Task “Step-2” prints Content of hello.txt

Slide 46

Slide 46 text

What is happening inside a container?

Slide 47

Slide 47 text

To view a task output you have three choices: 1. From the GUI 2. fly watch 3. fly hijack

Slide 48

Slide 48 text

Extracted tasks

Slide 49

Slide 49 text

Firstly, let’s extract a task to a separate file...

Slide 50

Slide 50 text

1. Task could be stored in separate file pipeline.yml: jobs: - name: job-build plan: - get: git-resource - task: step-1 platform: linux image_reource: <...> run: <...> task.yml: --- platform: linux image_resource: <...> run: < some commands >

Slide 51

Slide 51 text

1. Task could be stored in separate file pipeline.yml: jobs: - name: job-build plan: - get: git-resource - task: step-1 file: path/to/task.yml task.yml: --- platform: linux image_resource: <...> run: < some commands >

Slide 52

Slide 52 text

...and we can reuse common tasks and images! Feature #1

Slide 53

Slide 53 text

--- platform: linux # # General task to build a Maven based project. # It requires the pom.xml file at the root # image_resource: type: docker-image source: repository: my.docker.registry/java-maven inputs: - name: git-resource run: path: mvn args: [ deploy ] dir: git-resource Example of a common task for a Java project

Slide 54

Slide 54 text

For example....

Slide 55

Slide 55 text

Scenario #3: Common tasks 1. Get resource from common-tasks repo 2. Execute common task from resource 3. Execute embedded task

Slide 56

Slide 56 text

--- resources: - name: common-images type: git source: uri: http://git.myhost.com/common-ci-images.git jobs: - name: job-build plan: - get: common-images - task: task-build file: common-images/task-build.yml - task: task-deploy ... run: path: sh args: - -exc - | echo “Hello Devs” Resource 1: some project repository

Slide 57

Slide 57 text

Slide 58

Slide 58 text

What if I don’t want to commit changes? Feature #2

Slide 59

Slide 59 text

fly execute

Slide 60

Slide 60 text

--- platform: linux image_resource: type: docker-image source: repository: my.docker.registry:5080/java-maven tag: latest inputs: - name: source-code run: path: mvn args: [ deploy ] dir: source-code ~/workspace/task-build.yml

Slide 61

Slide 61 text

then...

Slide 62

Slide 62 text

fly -t office e -c ~/workspace/task-build.yml -i source-code=. Command “execute” (or, “e” as alias)

Slide 63

Slide 63 text

fly -t office e -c ~/workspace/task-build.yml -i source-code=. Which local task to execute

Slide 64

Slide 64 text

fly -t office e -c ~/workspace/task-build.yml -i source-code=. Pass path as mandatory input parameter (“.” in this case)

Slide 65

Slide 65 text

Fly: execute one task Concourse CI server DELL Dev local computer fly -t office e -c ~/workspace/task-build.yml -i source-code=. Source code task-build.yml

Slide 66

Slide 66 text

Show all the builds

Slide 67

Slide 67 text

Functional & integration tests?

Slide 68

Slide 68 text

What is the problem? 1. Set up whole system 2. Wait when it is ready 3. Run tests 4. Close system

Slide 69

Slide 69 text

Service Database ... ... ... Nginx Web Tests

Slide 70

Slide 70 text

One big uber container?

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

Service Database ... ... ... Nginx

Slide 73

Slide 73 text

Service Database ... ... ... Nginx

Slide 74

Slide 74 text

Docker in Docker?

Slide 75

Slide 75 text

dind - Docker in Docker Docker image that allows to run docker inside. For example, concourse/docker-image-resource: ● GitHub: https://github.com/concourse/docker-image-resource ● common.sh: https://github.com/concourse/docker-image-resource/blob/master/assets/common.sh mumoshu/dcind: ● GitHub: https://github.com/mumoshu/dcind Discussions are in the GitHub issue: https://github.com/concourse/concourse/issues/324

Slide 76

Slide 76 text

Task to run functional tests:

Slide 77

Slide 77 text

task.yml --- platform: linux image_resource: type: docker-image source: repository: mumoshu/dcind tag: latest inputs: - name: common-ci-images - name: source-code run: path: common-ci-images/run-functional-tests.sh

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

Slide 80

Slide 80 text

Run-functional-tests.sh (1: start docker) #!/bin/sh # Start docker service (script comes from image) source /docker-lib.sh start_docker

Slide 81

Slide 81 text

Run-functional-tests.sh (2: run production system) cd source-code # run production system in detached mode docker-compose up -d # check that services are running IS_CORE_RUNNING=$(docker inspect core | jq -r '.[] | .State.Running') … if [ "$IS_CORE_RUNNING" = false ] || … ;then exit 1; fi

Slide 82

Slide 82 text

Run-functional-tests.sh (3: run functional tests) cd ../source-code ABS_SOURCE=$(pwd .) docker run -it -w /tests -v "$ABS_SOURCE":/tests \ w32blaster/selenium-gradle \ xvfb-daemon-run gradle -q runAllTests NGINX_VIRTUAL_IP=$(docker inspect nginx | jq -r '.[] | .NetworkSettings.Networks.bridge.IPAddress') -e "NGINX_VIRTUAL_IP=$NGINX_VIRTUAL_IP" \ --add-host www.devclub.eu:"$NGINX_VIRTUAL_IP" \

Slide 83

Slide 83 text

Run-functional-tests.sh (4: print the results) TEST_RESULTS=$? if [ $TEST_RESULTS -eq 0 ];then printf "The Functional tests are successful!" else printf "The Functional Tests are failed!" fi

Slide 84

Slide 84 text

Run-functional-tests.sh (5: save logs and reports) # save logs and reports docker logs core > build/reports/core.log docker logs mongodb > build/reports/mongodb.log … tar cfz test-results.tar.gz build/reports # save test-results.tar.gz somewhere # using scp, ftp, rsync or any other way you prefer

Slide 85

Slide 85 text

Run-functional-tests.sh (6: stop services, exit) # stop the running containers, clean up etc... docker-compose stop # stop the docker service cd .. stop_docker # exit exit $TEST_RESULTS

Slide 86

Slide 86 text

No content

Slide 87

Slide 87 text

Atom plugin https://atom.io/packages/concourse-vis

Slide 88

Slide 88 text

ConcourseCI Radiator https://github.com/moodev/concourseci-radiator

Slide 89

Slide 89 text

No content

Slide 90

Slide 90 text

Disadvantages ● The project is young, not so many information in the network ● You have to learn new YAML configuration (although, it’s easy) ● Can’t pass results between jobs (we need to deploy/push it somewhere) ● Few plugins so far ● Takes time to start from scratch ● Requires a lot of disk space: - /var/lib/docker - Docker Registry storage - /data/worker-state - inside a worker container. Needs to be mapped to large partition

Slide 91

Slide 91 text

Advantages ● Forces to use local mirrors for a repository/registry ● Forces to use tiny containers (alpine, busybox etc…) ● Easy to scale horizontally ● Ideal for microservices (different languages/platforms) ● Very active project ● Requires Linux and Docker skills

Slide 92

Slide 92 text

The End Drink a coffee, make a Java!