Jenkins Automation

When you want to guarantee code quality, do continuous deployments or any other automation you will need a good continuous integration server.

While there are many great (hosted) solutions like TravisCi, CircleCi, etc … when it comes to **full control** or the need for private hosting, **Jenkins** is the right candidate. However, Jenkins could be known to be hard to learn and to configure. But what if you could automate all that complexity via some **easy scripting**? We are developers after all ;).

This talk will give you an introduction on how to automate the setup and the configuration of your CI whether you are still using the old fashion way (freestyle jobs) or the newer recommended way (Pipelines).

Some examples that will be covered:

– Auto setup jenkins with plugins, credentials, …
– Generate jobs per branch fetched directly for your VCS (Bitbucket, Github, …) (the old fashion way)
– Arrange your jobs in folders
– Make reusable DSL functions
– Using conditionals (e.g. master => should deploy if build successful)
– generate (multi branch) pipeline jobs with, optionally, fully integration with Bitbucket or Github

Transcript

1. JENKINS AUTOMATION TONI VAN DE VOORDE
2. ABOUT ME Toni Van de Voorde tonivdv toni.vdv@gmail.com @tonivdv ▸ CTO @ Adsdaq ▸ +10 years experience building web applications ▸ Love clean stuff and automations ▸ “Relax” with family
3. BACKGROUND ▸ Written in Java ▸ First released in 2005 ▸ Formerly known as the “Hudson Project” ▸ The name “Jenkins” was born in 2011
4. WHY ▸ Free ▸ OSS ▸ Customisation ▸ Plugin System ▸ Full Control
5. BUT “ JENKINS & JOBS CAN BE TIME- CONSUMING”
6. USE CASES
7. USE CASES ▸ Traditional: Freestyle Job

8. USE CASES ▸ Traditional: Freestyle Job
 ▸ Jenkins Pipeline

9. USE CASES ▸ Traditional: Freestyle Job
 ▸ Jenkins Pipeline
 ▸ Intro to JCasC

10. USE CASES ▸ Traditional: Freestyle Job
 ▸ Jenkins Pipeline
 ▸ Intro to JCasC
 ▸ Extra: Demo
11. USE CASES Freestyle Job
12. FREESTYLE JOB
13. FREESTYLE JOB Job DSL Job DSL plugin
14. Groovy-based Domain Specific Language FREESTYLE JOB Job DSL
15. FREESTYLE JOB Job DSL job(‘jobs-name’) {
 
 scm {
 git('git@github.com:adlogix/zf-symfony-container.git')
 } triggers {
 scm('H/15 * * * *')
 } steps {
 shell('composer install && vendor/bin/phpunit')
 }
 
 }
16. FREESTYLE JOB Job DSL job(‘jobs-name’) {
 
 scm {
 git('git@github.com:adlogix/zf-symfony-container.git')
 } triggers {
 scm('H/15 * * * *')
 } steps {
 shell('composer install && vendor/bin/phpunit')
 }
 
 }
17. FREESTYLE JOB Job DSL Installation Dockerfile FROM jenkins/jenkins:lts-alpine
 RUN /usr/local/bin/install-plugins.sh job-dsl
18. FREESTYLE JOB Job DSL Seed Job
19. FREESTYLE JOB Job DSL Seed Job
20. FREESTYLE JOB Job DSL Seed Job
21. FREESTYLE JOB Job DSL Seed Job
22. FREESTYLE JOB Job DSL Seed Job
23. FREESTYLE JOB Job DSL Oops
24. FREESTYLE JOB Job DSL Security 1. Script Approval
 2. Sandboxing
 3. Disable
25. FREESTYLE JOB Job DSL Example: Folders folder(‘components’)
 
 job(‘components/component1’) {
 ...
 } Plugin ID: cloudbees-folder
26. FREESTYLE JOB Job DSL def gitUrl = 'git://github.com/jenkinsci/job-dsl-plugin.git’ def branches = ['master','dev','hotfix'] branches.each { branch -> 
 job(“demo-${branch}") { 
 scm { 
 git(gitUrl, branch) 
 } 
 }
 } Example: Static Branches
27. FREESTYLE JOB Job DSL def project = 'Netflix/asgard’
 def branchApi = new URL("https://api.github.com/repos/${project}/branches")
 def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader()) branches.each { def branchName = it.name 
 def jobName = "${project}-${branchName}".replaceAll('/','-') job(jobName) { 
 scm { 
 git("https://github.com/${project}.git", branchName) 
 } 
 }
 } Example: Dynamic Branches
28. FREESTYLE JOB Job DSL def project = 'automate-jenkins-job-creation’
 def repo = "https://tonivdv@bitbucket.org/tonivdv/${project}.git" Process proc1 = "git ls-remote -h ${repo}".execute()
 Process proc2 = 'sed s/.*refs\\/heads\\///g'.execute()
 Process all = proc1 | proc2
 all.waitFor() def branches = "${all.in.text}".split('\\n') branches.each {
 …
 } Example: Dynamic Branches
29. FREESTYLE JOB Job DSL # ./scripts/utilities/GitUtils.groovy
 
 package utilities
 
 public class GitUtils { 
 public static String[] getRemoteBranches(def repoUrl, def out) { 
 
 Process proc1 = "git ls-remote -h ${repoUrl}".execute() 
 Process proc2 = 'sed s/.*refs\\/heads\\///g'.execute() 
 Process all = proc1 | proc2 all.waitFor() 
 
 def branches = "${all.in.text}".split('\\n') 
 return branches }
 } Example: Functions
30. FREESTYLE JOB Job DSL # ./scripts/demo.groovy import utilities.GitUtils
 
 def project = 'automate-jenkins-job-creation’
 def repo = https://tonivdv@bitbucket.org/tonivdv/${project}.git
 
 def branches = GitUtils.getRemoteBranches(repo, out) Example: Functions
31. FREESTYLE JOB Job DSL ‣ Don’t store passwords in your DSL scripts! DON’T ! 
 ‣ Use the Credentials 
 Plugin Example: Credentials Plugin ID: credentials
32. FREESTYLE JOB Job DSL job(‘jobs-name') {
 scm {
 git {
 remote {
 github('account/repo', 'ssh')
 credentials('github-ci-key')
 }
 }
 }
 } Example: Credentials
33. FREESTYLE JOB Job DSL // assign the my-userpassword credentials to the PASSWORD build variable
 
 job(‘jobs-name') {
 wrappers {
 credentialsBinding { 
 usernamePassword('PASSWORD', ‘my-userpassword’)
 }
 }
 } Example: Credentials
34. FREESTYLE JOB Job DSL Api Viewer https://jenkinsci.github.io/job-dsl-plugin/
35. Help !!! 
 My Plugin is not Supported FREESTYLE JOB Job DSL
36. FREESTYLE JOB Job DSL 1. Dynamic DSL
 2. Configure Block
37. FREESTYLE JOB Job DSL ‣ Powerful & Easy ‣ Looks like a generic DSL ‣ Available in API Viewer ‣ But plugin must use following annotations: ‣ @DataBoundConstructor ‣ @DataBoundSetter
 Dynamic DSL
38. FREESTYLE JOB Job DSL Dynamic DSL https://{base_url}/plugin/job-dsl/api-viewer/index.html
39. FREESTYLE JOB Job DSL Dynamic DSL job(‘jobs-name’) {
 
 scm {
 git(‘git@bitbucket.org:...’)
 bitbucketWeb {
 repoUrl(‘https://...’)
 }
 }
 
 }
40. FREESTYLE JOB Job DSL ‣ Gives access to underlying XML configuration ‣ Harder & Ugly ‣ Not fun ‣ BUT it works Configure Block
41. FREESTYLE JOB Job DSL Configure Block job('example-1') {
 
 configure { node -> // node represents <project> }
 
 }
42. FREESTYLE JOB Job DSL Configure Block https://{base_url}/job/demo2/config.xml
43. USE CASES Jenkins Pipeline
44. PIPELINE Jenkins Pipeline (or simply "Pipeline" with a capital "P") is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. © https://jenkins.io/doc/book/pipeline/
45. Why? PIPELINE ‣ “as-code” paradigm (DSL) ‣ Durable ‣ Pausable ‣ Parallel executions ‣ …
46. PIPELINE © https://jenkins.io/doc/book/pipeline/
47. How? PIPELINE Jenkinsfile
48. Example PIPELINE Jenkinsfile pipeline {
 agent any 
 stages {
 stage('Checkout') { 
 steps {
 Checkout scm
 }
 }
 stage('Build') { 
 steps {
 sh 'make it-build’
 }
 }
 stage('Test') { 
 steps {
 sh 'make test’
 }
 }
 
 }
 }
49. 2 Flavours … PIPELINE Jenkinsfile ‣ Declarative Pipeline ‣ Easier to use ‣ Limited ‣ Targets easy CD ‣ Scripted Pipeline ‣ Power Users ‣ Groovy is the limit
50. Declarative Pipeline PIPELINE Jenkinsfile pipeline {
 agent any 
 stages {
 ...
 stage('Deploy') { 
 when {
 branch 'master'
 }
 steps {
 make deploy
 }
 }
 }
 }
51. PIPELINE Jenkinsfile node { 
 stage('Deploy') { 
 if (env.BRANCH_NAME == 'master') { 
 make deploy 
 }
 }
 } Scripted Pipeline
52. PIPELINE Jenkinsfile pipeline {
 agent none
 stages {
 stage('php 7.3') {
 agent {
 docker { image 'php:7.3-alpine' }
 }
 steps {
 sh 'php --version'
 }
 } stage('php 7.2') {
 agent {
 docker { image 'php:7.2-alpine' }
 }
 steps {
 sh 'php --version'
 }
 }
 }
 } Using Docker Blue Ocean Plugin
53. PIPELINE Jenkinsfile pipeline {
 agent none
 stages {
 stage(‘php') {
 parallel {
 stage('php 7.3') {
 agent {
 docker { image 'php:7.3-alpine' }
 }
 steps {
 sh 'php --version'
 }
 } stage('php 7.2') {
 agent {
 docker { image 'php:7.2-alpine' }
 }
 steps {
 sh 'php --version'
 }
 }
 }
 }
 }
 } Parallel Blue Ocean Plugin
54. PIPELINE Jenkinsfile Discovering ‣ Pipeline Job ‣ Multi-Branch Pipeline ‣ Organisation Folder ‣ Github ‣ Bitbucket
55. PIPELINE Jenkinsfile Discovering
56. PIPELINE Jenkinsfile Multi-Branch Pipeline
57. PIPELINE Jenkinsfile Bitbucket Organisation
58. PIPELINE Job DSL plugin
59. PIPELINE multibranchPipelineJob('mypipelinejob') {
 branchSources {
 branchSource {
 source {
 bitbucket {
 id('bitbucket')
 serverUrl('https://bitbucket.org')
 repoOwner(‘my-user')
 repository('my-repository')
 credentialsId(‘bitbucket-ssh’)
 }
 }
 }
 }
 
 ...
 }
 
 Job DSL Multi-Branch Pipeline
60. PIPELINE multibranchPipelineJob('mypipelinejob') {
 ...
 
 // discover Branches (workaround due to JENKINS-46202)
 configure { def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits // Exclude branches that are also filed as PRs 
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait' {
 strategyId(1)
 }
 
 // Discover each pull request once ... without merging
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait' {
 strategyId(2)
 }
 }
 }
 
 Job DSL Multi-Branch Pipeline
61. PIPELINE organizationFolder('example') {
 
 description(‘Some description')
 displayName(‘Other name')
 organizations {
 bitbucket { 
 serverUrl('https://bitbucket.org') 
 repoOwner('adqphpbenl19')
 credentialsId('bitbucket-tonivdv') 
 }
 }
 
 ...
 } Job DSL Organisation Folder
62. PIPELINE organizationFolder('example') {
 
 ...
 
 configure {
 def traits = it / navigators / 'com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator' / traits
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait' {
 strategyId(1)
 }
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait' {
 strategyId(2)
 }
 }
 
 } Job DSL Organisation Folder
63. USE CASES Intro to JCasC
64. JCASC JCasC or Jenkins Configuration as Code allows to define the “configuration” of plugins and parameters as a simple, human friendly, plain text Yaml syntax.
 
 © https://jenkins.io/projects/jcasc/
65. JCASC Installation
66. JCASC Installation ‣ Define location of the casc configurations ‣ Path to a folder containing a set of configs
 e.g. /var/jenkins_home/casc_configs ‣ Full path to single file
 e.g. /var/jenkins_home/casc_configs/jenkins.yaml ‣ A URL pointing to a file served on the web
 e.g. https://acme.org/jenkins.yaml ‣ Default: $JENKINS_ROOT/jenkins.yaml
67. JCASC # casc_configs/core.yaml
 
 jenkins: systemMessage: | Hello JCasC,
 
 This is exciting! numExecutors: 1 Examples
68. JCASC # casc_configs/realm.yaml
 
 jenkins:
 securityRealm:
 bitbucket:
 clientID: ${b_id}
 clientSecret: ${b_secret} Examples # casc_configs/authorization.yaml
 
 jenkins:
 authorizationStrategy:
 globalMatrix:
 grantedPermissions:
 - "Overall/Read:anonymous"
 - "Overall/Administer:authenticated"
69. JCASC Api Viewer
70. JCASC Api Viewer
71. JCASC Docker
72. DEMO
73. QUESTIONS
74. THANK YOU joind.in: 
 https://joind.in/talk/446a8
75. REFERENCES ‣ Doc ‣ https://jenkins.io/ ‣ https://github.com/jenkinsci/job-dsl-plugin/wiki ‣ https://jenkins.io/doc/book/pipeline/ ‣ https://www.phparch.com/article/jenkins-automation/ ‣ Code Examples ‣ https://github.com/tonivdv/rnd-jenkins-as-code ‣ https://github.com/tonivdv/automate-jenkins-job-creation ‣ Various ‣ https://jenkins.io/artwork/ ‣ https://www.cloudbees.com/