Jenkins Automation

Jenkins Automation

When you want to guarantee code quality, do continuous deployments or any other automation you will need a good continuous integration server.

While there are many great (hosted) solutions like TravisCi, CircleCi, etc … when it comes to **full control** or the need for private hosting, **Jenkins** is the right candidate. However, Jenkins could be known to be hard to learn and to configure. But what if you could automate all that complexity via some **easy scripting**? We are developers after all ;).

This talk will give you an introduction on how to automate the setup and the configuration of your CI whether you are still using the old fashion way (freestyle jobs) or the newer recommended way (Pipelines).

Some examples that will be covered:

– Auto setup jenkins with plugins, credentials, …
– Generate jobs per branch fetched directly for your VCS (Bitbucket, Github, …) (the old fashion way)
– Arrange your jobs in folders
– Make reusable DSL functions
– Using conditionals (e.g. master => should deploy if build successful)
– generate (multi branch) pipeline jobs with, optionally, fully integration with Bitbucket or Github

8b3f69dcdbfc477777ab524c316168c1?s=128

Toni Van de Voorde

January 26, 2019
Tweet

Transcript

  1. JENKINS AUTOMATION TONI VAN DE VOORDE

  2. ABOUT ME Toni Van de Voorde tonivdv toni.vdv@gmail.com @tonivdv ▸

    CTO @ Adsdaq ▸ +10 years experience building web applications ▸ Love clean stuff and automations ▸ “Relax” with family
  3. BACKGROUND ▸ Written in Java ▸ First released in 2005

    ▸ Formerly known as the “Hudson Project” ▸ The name “Jenkins” was born in 2011
  4. WHY ▸ Free ▸ OSS ▸ Customisation ▸ Plugin System

    ▸ Full Control
  5. BUT “ JENKINS & JOBS CAN BE TIME- CONSUMING”

  6. USE CASES

  7. USE CASES ▸ Traditional: Freestyle Job


  8. USE CASES ▸ Traditional: Freestyle Job
 ▸ Jenkins Pipeline


  9. USE CASES ▸ Traditional: Freestyle Job
 ▸ Jenkins Pipeline
 ▸

    Intro to JCasC

  10. USE CASES ▸ Traditional: Freestyle Job
 ▸ Jenkins Pipeline
 ▸

    Intro to JCasC
 ▸ Extra: Demo
  11. USE CASES Freestyle Job

  12. FREESTYLE JOB

  13. FREESTYLE JOB Job DSL Job DSL plugin

  14. Groovy-based Domain Specific Language FREESTYLE JOB Job DSL

  15. FREESTYLE JOB Job DSL job(‘jobs-name’) {
 
 scm {
 git('git@github.com:adlogix/zf-symfony-container.git')


    } triggers {
 scm('H/15 * * * *')
 } steps {
 shell('composer install && vendor/bin/phpunit')
 }
 
 }
  16. FREESTYLE JOB Job DSL job(‘jobs-name’) {
 
 scm {
 git('git@github.com:adlogix/zf-symfony-container.git')


    } triggers {
 scm('H/15 * * * *')
 } steps {
 shell('composer install && vendor/bin/phpunit')
 }
 
 }
  17. FREESTYLE JOB Job DSL Installation Dockerfile FROM jenkins/jenkins:lts-alpine
 RUN /usr/local/bin/install-plugins.sh

    job-dsl
  18. FREESTYLE JOB Job DSL Seed Job

  19. FREESTYLE JOB Job DSL Seed Job

  20. FREESTYLE JOB Job DSL Seed Job

  21. FREESTYLE JOB Job DSL Seed Job

  22. FREESTYLE JOB Job DSL Seed Job

  23. FREESTYLE JOB Job DSL Oops

  24. FREESTYLE JOB Job DSL Security 1. Script Approval
 2. Sandboxing


    3. Disable
  25. FREESTYLE JOB Job DSL Example: Folders folder(‘components’)
 
 job(‘components/component1’) {


    ...
 } Plugin ID: cloudbees-folder
  26. FREESTYLE JOB Job DSL def gitUrl = 'git://github.com/jenkinsci/job-dsl-plugin.git’ def branches

    = ['master','dev','hotfix'] branches.each { branch -> 
 job(“demo-${branch}") { 
 scm { 
 git(gitUrl, branch) 
 } 
 }
 } Example: Static Branches
  27. FREESTYLE JOB Job DSL def project = 'Netflix/asgard’
 def branchApi

    = new URL("https://api.github.com/repos/${project}/branches")
 def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader()) branches.each { def branchName = it.name 
 def jobName = "${project}-${branchName}".replaceAll('/','-') job(jobName) { 
 scm { 
 git("https://github.com/${project}.git", branchName) 
 } 
 }
 } Example: Dynamic Branches
  28. FREESTYLE JOB Job DSL def project = 'automate-jenkins-job-creation’
 def repo

    = "https://tonivdv@bitbucket.org/tonivdv/${project}.git" Process proc1 = "git ls-remote -h ${repo}".execute()
 Process proc2 = 'sed s/.*refs\\/heads\\///g'.execute()
 Process all = proc1 | proc2
 all.waitFor() def branches = "${all.in.text}".split('\\n') branches.each {
 …
 } Example: Dynamic Branches
  29. FREESTYLE JOB Job DSL # ./scripts/utilities/GitUtils.groovy
 
 package utilities
 


    public class GitUtils { 
 public static String[] getRemoteBranches(def repoUrl, def out) { 
 
 Process proc1 = "git ls-remote -h ${repoUrl}".execute() 
 Process proc2 = 'sed s/.*refs\\/heads\\///g'.execute() 
 Process all = proc1 | proc2 all.waitFor() 
 
 def branches = "${all.in.text}".split('\\n') 
 return branches }
 } Example: Functions
  30. FREESTYLE JOB Job DSL # ./scripts/demo.groovy import utilities.GitUtils
 
 def

    project = 'automate-jenkins-job-creation’
 def repo = https://tonivdv@bitbucket.org/tonivdv/${project}.git
 
 def branches = GitUtils.getRemoteBranches(repo, out) Example: Functions
  31. FREESTYLE JOB Job DSL ‣ Don’t store passwords in your

    DSL scripts! DON’T ! 
 ‣ Use the Credentials 
 Plugin Example: Credentials Plugin ID: credentials
  32. FREESTYLE JOB Job DSL job(‘jobs-name') {
 scm {
 git {


    remote {
 github('account/repo', 'ssh')
 credentials('github-ci-key')
 }
 }
 }
 } Example: Credentials
  33. FREESTYLE JOB Job DSL // assign the my-userpassword credentials to

    the PASSWORD build variable
 
 job(‘jobs-name') {
 wrappers {
 credentialsBinding { 
 usernamePassword('PASSWORD', ‘my-userpassword’)
 }
 }
 } Example: Credentials
  34. FREESTYLE JOB Job DSL Api Viewer https://jenkinsci.github.io/job-dsl-plugin/

  35. Help !!! 
 My Plugin is not Supported FREESTYLE JOB

    Job DSL
  36. FREESTYLE JOB Job DSL 1. Dynamic DSL
 2. Configure Block

  37. FREESTYLE JOB Job DSL ‣ Powerful & Easy ‣ Looks

    like a generic DSL ‣ Available in API Viewer ‣ But plugin must use following annotations: ‣ @DataBoundConstructor ‣ @DataBoundSetter
 Dynamic DSL
  38. FREESTYLE JOB Job DSL Dynamic DSL https://{base_url}/plugin/job-dsl/api-viewer/index.html

  39. FREESTYLE JOB Job DSL Dynamic DSL job(‘jobs-name’) {
 
 scm

    {
 git(‘git@bitbucket.org:...’)
 bitbucketWeb {
 repoUrl(‘https://...’)
 }
 }
 
 }
  40. FREESTYLE JOB Job DSL ‣ Gives access to underlying XML

    configuration ‣ Harder & Ugly ‣ Not fun ‣ BUT it works Configure Block
  41. FREESTYLE JOB Job DSL Configure Block job('example-1') {
 
 configure

    { node -> // node represents <project> }
 
 }
  42. FREESTYLE JOB Job DSL Configure Block https://{base_url}/job/demo2/config.xml

  43. USE CASES Jenkins Pipeline

  44. PIPELINE Jenkins Pipeline (or simply "Pipeline" with a capital "P")

    is a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins. © https://jenkins.io/doc/book/pipeline/
  45. Why? PIPELINE ‣ “as-code” paradigm (DSL) ‣ Durable ‣ Pausable

    ‣ Parallel executions ‣ …
  46. PIPELINE © https://jenkins.io/doc/book/pipeline/

  47. How? PIPELINE Jenkinsfile

  48. Example PIPELINE Jenkinsfile pipeline {
 agent any 
 stages {


    stage('Checkout') { 
 steps {
 Checkout scm
 }
 }
 stage('Build') { 
 steps {
 sh 'make it-build’
 }
 }
 stage('Test') { 
 steps {
 sh 'make test’
 }
 }
 
 }
 }
  49. 2 Flavours … PIPELINE Jenkinsfile ‣ Declarative Pipeline ‣ Easier

    to use ‣ Limited ‣ Targets easy CD ‣ Scripted Pipeline ‣ Power Users ‣ Groovy is the limit
  50. Declarative Pipeline PIPELINE Jenkinsfile pipeline {
 agent any 
 stages

    {
 ...
 stage('Deploy') { 
 when {
 branch 'master'
 }
 steps {
 make deploy
 }
 }
 }
 }
  51. PIPELINE Jenkinsfile node { 
 stage('Deploy') { 
 if (env.BRANCH_NAME

    == 'master') { 
 make deploy 
 }
 }
 } Scripted Pipeline
  52. PIPELINE Jenkinsfile pipeline {
 agent none
 stages {
 stage('php 7.3')

    {
 agent {
 docker { image 'php:7.3-alpine' }
 }
 steps {
 sh 'php --version'
 }
 } stage('php 7.2') {
 agent {
 docker { image 'php:7.2-alpine' }
 }
 steps {
 sh 'php --version'
 }
 }
 }
 } Using Docker Blue Ocean Plugin
  53. PIPELINE Jenkinsfile pipeline {
 agent none
 stages {
 stage(‘php') {


    parallel {
 stage('php 7.3') {
 agent {
 docker { image 'php:7.3-alpine' }
 }
 steps {
 sh 'php --version'
 }
 } stage('php 7.2') {
 agent {
 docker { image 'php:7.2-alpine' }
 }
 steps {
 sh 'php --version'
 }
 }
 }
 }
 }
 } Parallel Blue Ocean Plugin
  54. PIPELINE Jenkinsfile Discovering ‣ Pipeline Job ‣ Multi-Branch Pipeline ‣

    Organisation Folder ‣ Github ‣ Bitbucket
  55. PIPELINE Jenkinsfile Discovering

  56. PIPELINE Jenkinsfile Multi-Branch Pipeline

  57. PIPELINE Jenkinsfile Bitbucket Organisation

  58. PIPELINE Job DSL plugin

  59. PIPELINE multibranchPipelineJob('mypipelinejob') {
 branchSources {
 branchSource {
 source {
 bitbucket

    {
 id('bitbucket')
 serverUrl('https://bitbucket.org')
 repoOwner(‘my-user')
 repository('my-repository')
 credentialsId(‘bitbucket-ssh’)
 }
 }
 }
 }
 
 ...
 }
 
 Job DSL Multi-Branch Pipeline
  60. PIPELINE multibranchPipelineJob('mypipelinejob') {
 ...
 
 // discover Branches (workaround due

    to JENKINS-46202)
 configure { def traits = it / sources / data / 'jenkins.branch.BranchSource' / source / traits // Exclude branches that are also filed as PRs 
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait' {
 strategyId(1)
 }
 
 // Discover each pull request once ... without merging
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait' {
 strategyId(2)
 }
 }
 }
 
 Job DSL Multi-Branch Pipeline
  61. PIPELINE organizationFolder('example') {
 
 description(‘Some description')
 displayName(‘Other name')
 organizations {


    bitbucket { 
 serverUrl('https://bitbucket.org') 
 repoOwner('adqphpbenl19')
 credentialsId('bitbucket-tonivdv') 
 }
 }
 
 ...
 } Job DSL Organisation Folder
  62. PIPELINE organizationFolder('example') {
 
 ...
 
 configure {
 def traits

    = it / navigators / 'com.cloudbees.jenkins.plugins.bitbucket.BitbucketSCMNavigator' / traits
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.BranchDiscoveryTrait' {
 strategyId(1)
 }
 traits << 'com.cloudbees.jenkins.plugins.bitbucket.OriginPullRequestDiscoveryTrait' {
 strategyId(2)
 }
 }
 
 } Job DSL Organisation Folder
  63. USE CASES Intro to JCasC

  64. JCASC JCasC or Jenkins Configuration as Code allows to define

    the “configuration” of plugins and parameters as a simple, human friendly, plain text Yaml syntax.
 
 © https://jenkins.io/projects/jcasc/
  65. JCASC Installation

  66. JCASC Installation ‣ Define location of the casc configurations ‣

    Path to a folder containing a set of configs
 e.g. /var/jenkins_home/casc_configs ‣ Full path to single file
 e.g. /var/jenkins_home/casc_configs/jenkins.yaml ‣ A URL pointing to a file served on the web
 e.g. https://acme.org/jenkins.yaml ‣ Default: $JENKINS_ROOT/jenkins.yaml
  67. JCASC # casc_configs/core.yaml
 
 jenkins: systemMessage: | Hello JCasC,
 


    This is exciting! numExecutors: 1 Examples
  68. JCASC # casc_configs/realm.yaml
 
 jenkins:
 securityRealm:
 bitbucket:
 clientID: ${b_id}
 clientSecret:

    ${b_secret} Examples # casc_configs/authorization.yaml
 
 jenkins:
 authorizationStrategy:
 globalMatrix:
 grantedPermissions:
 - "Overall/Read:anonymous"
 - "Overall/Administer:authenticated"
  69. JCASC Api Viewer

  70. JCASC Api Viewer

  71. JCASC Docker

  72. DEMO

  73. QUESTIONS

  74. THANK YOU joind.in: 
 https://joind.in/talk/446a8

  75. REFERENCES ‣ Doc ‣ https://jenkins.io/ ‣ https://github.com/jenkinsci/job-dsl-plugin/wiki ‣ https://jenkins.io/doc/book/pipeline/ ‣

    https://www.phparch.com/article/jenkins-automation/ ‣ Code Examples ‣ https://github.com/tonivdv/rnd-jenkins-as-code ‣ https://github.com/tonivdv/automate-jenkins-job-creation ‣ Various ‣ https://jenkins.io/artwork/ ‣ https://www.cloudbees.com/