Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Andreas Caternberg on Jenkins Pipelines

Andreas Caternberg on Jenkins Pipelines

More Decks by Enterprise Java User Group Austria

Other Decks in Technology


  1. CloudBees Jenkins Enterprise © 2017 CloudBees, Inc. All Rights Reserved

    • Community Innovation • Jenkins for the Enterprise
  2. What we are going to cover • What is a

    pipeline • Compare scripted pipeline to declarative pipeline • Create a declarative pipeline • Learn about Blue Ocean • Share pipeline resources • Extra scripted pipeline information © 2017 CloudBees, Inc. All Rights Reserved
  3. What is a Jenkins pipeline • Formerly known as Jenkins

    Workflow, Jenkins Pipeline was introduced in 2014 • New job type – a single Groovy script using an embedded DSL - no need to jump between multiple job configuration screens to see what is going on • Durable - keeps running even if Jenkins master is not • Distributable – a Pipeline job may be run across an unlimited number of nodes, including in parallel • Pausable – allows waiting for input from users before continuing to the next step • Visualized - Pipeline Stage View provides status at-a-glance dashboards to include trending © 2017 CloudBees, Inc. All Rights Reserved
  4. What is a declarative pipeline? • A syntax for Jenkins

    Pipelines designed to… –Have smarter default behavior for most use cases –Make Pipelines more structured –Provide better error reporting and handling © 2017 CloudBees, Inc. All Rights Reserved
  5. Have a smarter behavior • Default SCM checkout • Built

    with Docker and containers in mind from day one • No more need for try/catch for notifications, etc. • More intuitive ways to specify job properties, triggers and parameters • Streamlined syntax for conditional execution of stages © 2017 CloudBees, Inc. All Rights Reserved
  6. More structure in Pipelines • Separate configuration from actual steps

    to be executed • Everything happens in a stage • Easier to look at any Declarative Pipeline and understand what it’s doing - more consistency! • Enables round-tripping with the visual editor (more on this later!) • Built with and for Blue Ocean visualization © 2017 CloudBees, Inc. All Rights Reserved
  7. Better error reporting • Syntax and validation errors reported at

    the very beginning of the build • No more waiting until a build has gone an hour to discover that you misspelled “timeout”! • Errors report as compilation errors, pointing to the specific code or text that’s causing problems • Eliminates many potentially confusing stack traces (though not all!) • Validates things like syntax, required configuration, step parameter types, and more © 2017 CloudBees, Inc. All Rights Reserved
  8. What happens to Scripted Pipeline? • In short? Nothing. •

    We’re now calling “traditional” Pipeline “Scripted Pipeline” to distinguish it from Declarative Pipeline • All one execution engine behind the scenes ◦Declarative is built on top of Scripted Pipeline, not replacing it ◦Blue Ocean and Stage View don’t distinguish between a Scripted Pipeline or a Declarative Pipeline • Scripted Pipeline still used *inside* Declarative Pipeline in steps blocks, post blocks ◦You can copy the contents of a Declarative steps block into a Scripted Pipeline (inside a node block, of course!) and it’ll Just Work. © 2017 CloudBees, Inc. All Rights Reserved
  9. Benefits of Declarative for new users • Lower barrier of

    entry than Scripted Pipelines • Human readable • Does not require Groovy-specific expertise • Create and edit Declarative Pipelines via the upcoming UI Editor © 2017 CloudBees, Inc. All Rights Reserved
  10. Benefits of Declarative for existing users • Uses same engine

    as Scripted, so existing investments still pay off • Expand usage and usability ◦Less burden on “experts” ◦Easier empowerment of other users • Easier collaboration and code review • Lintable! • Separation of Jenkins infrastructure-related configuration from steps to execute © 2017 CloudBees, Inc. All Rights Reserved
  11. Pipeline Comparison © 2017 CloudBees, Inc. All Rights Reserved Scripted

    Pipeline node { try { stage('Build') { checkout scm docker.image('ubuntu').inside { sh 'mvn clean install' } } } catch(exc) { mail to:'[email protected]', subject:'FAILURE:' } finally { deleteDir() } } Declarative Pipeline pipeline { agent { docker 'ubuntu' } stages { stage('Build') { steps { sh 'mvn clean install' } } } post { always { deleteDir() } failure { mail to:'[email protected]', subject:'FAILURE:' } } }
  12. Get Ready • Log into our CloudBees Jenkins Operations Center

    http://dpa.beedemo.net/pipeline- workshop/ • Create an account © 2017 CloudBees, Inc. All Rights Reserved
  13. Get Ready • Log in • Go to the LBAJP

    server © 2017 CloudBees, Inc. All Rights Reserved
  14. Create a Pipeline Job © 2017 CloudBees, Inc. All Rights

    Reserved • Select ”New Item” • Name your job
  15. A basic pipeline pipeline { agent any stages { stage('Build')

    { steps { echo 'Hello from Build' } } stage('Test') { steps { echo 'Testing Testing 123' } } stage('Deploy') { steps { echo 'Deploy some things' } } } } © 2017 CloudBees, Inc. All Rights Reserved
  16. Add an agent © 2017 CloudBees, Inc. All Rights Reserved

    pipeline { agent any stages { pipeline { agent { label ‘docker’ } stages {
  17. Stash Files © 2017 CloudBees, Inc. All Rights Reserved stages

    { stage('Build') { steps { echo 'Hello from Build’ writeFile(file: 'config', text: 'version=1', encoding: 'UTF-8') stash(name: 'pom-config', includes: 'config') } } stage('Test') { environment { configValue = readFile(encoding: 'UTF-8', file: 'config') } steps { echo 'Testing Testing 123’ unstash 'pom-config' sh 'echo "$configValue"' } }
  18. Create an input step stage('Deploy') { steps { echo 'Deploy

    some things' input 'Do you want to deploy?' echo 'deployed' } } © 2017 CloudBees, Inc. All Rights Reserved
  19. Blue Ocean Visualization • Some special smarts for optimized visualization

    • Blue Ocean keeps built-in stages for things like checkout, Docker image prep, post-build actions out of your way unless there’s a problem • Special marking of stages that have been skipped due to either an unsatisfied when condition or an earlier failed stage © 2017 CloudBees, Inc. All Rights Reserved
  20. Resources on Declarative • Documentation https://jenkins.io/doc/ • Examples https://github.com/jenkinsci/pipeline-examples •

    Plugin source https://github.com/jenkinsci/pipeline-model-definition-plugin © 2017 CloudBees, Inc. All Rights Reserved
  21. Key Benefits ✓ Long-running ✓ Durable ✓ Scriptable ✓ One-place

    for Everything Pipeline: a new job type ✓ Makes building Pipelines Simple
  22. Domain Specific Language Simple, Groovy-based DSL (“Domain Specific Language”) to

    manage build step orchestration Can be defined as DSL in Jenkins or as Jenkinsfile in SCM Groovy is widely used in Jenkins ecosystem. You can leverage the large Jenkins Groovy experience If you don’t like/care about Groovy, just consider the DSL as a dedicated simple syntax
  23. Create a Pipeline - core steps 50 stage - group

    your steps into its component parts and control concurrency node - schedules the steps to run by adding them to Jenkins' build queue and creates a workspace specific to this pipeline sh (or bat) - execute shell (*nix) or batch scripts (Windows) just like freestyle jobs, calling any tools in the agent
  24. Create a Pipeline stage('build') { echo 'hello from jenkins master'

    node { sh 'echo hello from jenkins agent' } } stage('test') { echo 'test some things' } stage('deploy') { echo 'deploy some things' } 51
  25. Files stash - store some files for later use unstash

    - retrieve previously stashed files (even across nodes) writeFile - write a file to the workspace readFile - read a file from the workspace 53
  26. Checkout from SCM and stash Files stage('build') { node {

    git 'https://github.com/beedemo/mobile-deposit-api.git' writeFile encoding: 'UTF-8', file: 'config', text: 'version=1' stash includes: 'pom.xml,config', name: 'pom-config' } } stage('test'){ node { unstash 'pom-config' configValue = readFile encoding: 'UTF-8', file: 'config' echo configValue } } 54
  27. try up to N times retry(5) { // some block

    } wait for a condition waitUntil { // some block } Flow Control wait for a set time sleep time: 1000, unit:'NANOSECONDS' timeout timeout(time: 30, unit: 'SECONDS'){ // some block }
  28. You can also rely on Groovy control flow syntax! while(something)

    { // do something if (something_else) { // do something else } } Flow Control 56 try{ //some things }catch(e){ // }
  29. Advanced Flow Control input - pause for manual or automated

    approval parallel - allows simultaneous execution of build steps on the current node or across nodes, thus increasing build speed parallel 'quality scan': { node {sh 'mvn sonar:sonar'} }, 'integration test': { node {sh 'mvn verify'} } checkpoint - capture the workspace state so it can be reused as a starting point for subsequent runs
  30. Input Approval & Checkpoints 59 stage('deploy') { input message: 'Do

    you want to deploy?' node { echo 'deployed' } }
  31. Input Approval & Checkpoints 60 checkpoint 'testing-complete' stage('approve') { mail

    body: "Approval needed for '${env.JOB_NAME}' at '${env.BUILD_URL}'", subject: "${env.JOB_NAME} Approval", to: "[email protected]" timeout(time: 7, unit: 'DAYS') { input message: 'Do you want to deploy?', parameters: [string(defaultValue: '', description: 'Provide any comments regarding decision.', name: 'Comments')], submitter: 'ops' } }
  32. tool Step •Binds a tool installation to a variable •The

    tool home directory is returned •Only tools already configured are available def mvnHome = tool 'M3' sh "${mvnHome}/bin/mvn -B verify" 62
  33. Use Docker Containers •The CloudBees Docker Pipeline plugin allows running

    steps inside a container •You can even build the container as part of the same Pipeline docker.image('maven:3.3.3-jdk-8').inside() { sh 'mvn -B verify' } 63
  34. More Advanced Steps Send email mail body: 'Uh oh.', subject:

    'Build Failed!', to: '[email protected]' step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: '[email protected]']) Deploy to Amazon Elastic Beanstalk wrap([$class: 'AmazonAwsCliBuildWrapper', credentialsId: 'aws-beanstalk- credentials', defaultRegion: 'us-east-1']) { sh 'aws elasticbeanstalk create-application-version' } Integrate with Jira jiraComment(issueKey: "EX-111", body: "Job '${env.JOB_NAME}' ($ {env.BUILD_NUMBER}) built. Please go to ${env.BUILD_URL}.")
  35. Accelerating CD with Containers Workflow CD Pipeline Triggers: ✓ New

    application code (feature, bug fix, etc.) ✓ Updated certified stack (security fix in Linux, etc.) ✓ Will lead to a new gold image being built and available for … TESTING … STAGING … PRODUCTION ✓ All taking place in a standardized/similar/consistent OS environment + Jenkins Pipeline TEST STAG E PRODUCTIO N App <code> (git, etc.) Gold Docker Image (~per app) <OS config> Certified Docker Images (Ubuntu, etc.) <OS config>
  36. Jenkins Pipeline Docker Commands • withRegistry Specifies which Docker Registry

    to use for pushing/pulling • withServer Specifies which Server to issue Docker commands against • Build Builds container from Dockerfile • Image.run Runs a container • Image.inside Runs container and allows you to execute command inside the container • Image.push Pushed image to Docker Registry with specified credentials
  37. Pipeline - Docker Example stage 'Build' node('docker-cloud') { checkout scm

    docker.image('java:jdk-8').inside('-v /data:/data') { sh "mvn clean package" }} stage 'Quality Analysis' node('docker-cloud') { unstash 'pom' parallel( JRE8Test: { docker.image('java:jre-8').inside('-v /data:/data') { sh 'mvn -Dmaven.repo.local=/data/mvn/repo verify' } },JRE7Test: { docker.image('java:jre-7').inside('-v /data:/data') { sh 'mvn -Dmaven.repo.local=/data/mvn/repo verify' } }, failFast: true )}
  38. Pipeline - Docker Example node('docker-cloud') { stage 'Build Docker Image'

    dir('target') { mobileDepositApiImage = docker.build "beedemo/mobile-deposit-api:$ {dockerTag}" } stage 'Publish Docker Image' withDockerRegistry(registry: [credentialsId: 'docker-hub-beedemo']) { mobileDepositApiImage.push() } }