Andreas Caternberg on Jenkins Pipelines

Slide 1

Slide 1 text

Let’s Build a Jenkins Pipeline Andreas Caternberg

Slide 2

Slide 2 text

Download Slides Now https://gist.github.com/cccaternberg/3984e4a3d1a84ec6faffbd728ce6c247

Slide 3

Slide 3 text

About Your Guide Andreas Caternberg Senior Consultant at CloudBees @acaternberg

Slide 4

Slide 4 text

Plan for today

Slide 5

Slide 5 text

Slide 6

Slide 6 text

What we are going to cover • What is a pipeline • Compare scripted pipeline to declarative pipeline • Create a declarative pipeline • Learn about Blue Ocean • Share pipeline resources • Extra scripted pipeline information © 2017 CloudBees, Inc. All Rights Reserved

Slide 7

Slide 7 text

What is a Jenkins pipeline • Formerly known as Jenkins Workflow, Jenkins Pipeline was introduced in 2014 • New job type – a single Groovy script using an embedded DSL - no need to jump between multiple job configuration screens to see what is going on • Durable - keeps running even if Jenkins master is not • Distributable – a Pipeline job may be run across an unlimited number of nodes, including in parallel • Pausable – allows waiting for input from users before continuing to the next step • Visualized - Pipeline Stage View provides status at-a-glance dashboards to include trending © 2017 CloudBees, Inc. All Rights Reserved

Slide 8

Slide 8 text

What is a declarative pipeline? • A syntax for Jenkins Pipelines designed to… –Have smarter default behavior for most use cases –Make Pipelines more structured –Provide better error reporting and handling © 2017 CloudBees, Inc. All Rights Reserved

Slide 9

Slide 9 text

Have a smarter behavior • Default SCM checkout • Built with Docker and containers in mind from day one • No more need for try/catch for notifications, etc. • More intuitive ways to specify job properties, triggers and parameters • Streamlined syntax for conditional execution of stages © 2017 CloudBees, Inc. All Rights Reserved

Slide 10

Slide 10 text

More structure in Pipelines • Separate configuration from actual steps to be executed • Everything happens in a stage • Easier to look at any Declarative Pipeline and understand what it’s doing - more consistency! • Enables round-tripping with the visual editor (more on this later!) • Built with and for Blue Ocean visualization © 2017 CloudBees, Inc. All Rights Reserved

Slide 11

Slide 11 text

Better error reporting ● Syntax and validation errors reported at the very beginning of the build ● No more waiting until a build has gone an hour to discover that you misspelled “timeout”! ● Errors report as compilation errors, pointing to the specific code or text that’s causing problems ● Eliminates many potentially confusing stack traces (though not all!) ● Validates things like syntax, required configuration, step parameter types, and more © 2017 CloudBees, Inc. All Rights Reserved

Slide 12

Slide 12 text

What happens to Scripted Pipeline? ● In short? Nothing. ● We’re now calling “traditional” Pipeline “Scripted Pipeline” to distinguish it from Declarative Pipeline ● All one execution engine behind the scenes ○Declarative is built on top of Scripted Pipeline, not replacing it ○Blue Ocean and Stage View don’t distinguish between a Scripted Pipeline or a Declarative Pipeline ● Scripted Pipeline still used *inside* Declarative Pipeline in steps blocks, post blocks ○You can copy the contents of a Declarative steps block into a Scripted Pipeline (inside a node block, of course!) and it’ll Just Work. © 2017 CloudBees, Inc. All Rights Reserved

Slide 13

Slide 13 text

Benefits of Declarative for new users ● Lower barrier of entry than Scripted Pipelines ● Human readable ● Does not require Groovy-specific expertise ● Create and edit Declarative Pipelines via the upcoming UI Editor © 2017 CloudBees, Inc. All Rights Reserved

Slide 14

Slide 14 text

Benefits of Declarative for existing users ● Uses same engine as Scripted, so existing investments still pay off ● Expand usage and usability ○Less burden on “experts” ○Easier empowerment of other users ● Easier collaboration and code review ● Lintable! ● Separation of Jenkins infrastructure-related configuration from steps to execute © 2017 CloudBees, Inc. All Rights Reserved

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Pipeline Comparison © 2017 CloudBees, Inc. All Rights Reserved Scripted Pipeline node { try { stage('Build') { checkout scm docker.image('ubuntu').inside { sh 'mvn clean install' } } } catch(exc) { mail to:'[email protected]', subject:'FAILURE:' } finally { deleteDir() } } Declarative Pipeline pipeline { agent { docker 'ubuntu' } stages { stage('Build') { steps { sh 'mvn clean install' } } } post { always { deleteDir() } failure { mail to:'[email protected]', subject:'FAILURE:' } } }

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

A basic pipeline pipeline { agent any stages { stage('Build') { steps { echo 'Hello from Build' } } stage('Test') { steps { echo 'Testing Testing 123' } } stage('Deploy') { steps { echo 'Deploy some things' } } } } © 2017 CloudBees, Inc. All Rights Reserved

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Stash Files © 2017 CloudBees, Inc. All Rights Reserved stages { stage('Build') { steps { echo 'Hello from Build’ writeFile(file: 'config', text: 'version=1', encoding: 'UTF-8') stash(name: 'pom-config', includes: 'config') } } stage('Test') { environment { configValue = readFile(encoding: 'UTF-8', file: 'config') } steps { echo 'Testing Testing 123’ unstash 'pom-config' sh 'echo "$configValue"' } }

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Slide 38

Slide 38 text

Blue Ocean Visualization ● Some special smarts for optimized visualization ● Blue Ocean keeps built-in stages for things like checkout, Docker image prep, post-build actions out of your way unless there’s a problem ● Special marking of stages that have been skipped due to either an unsatisfied when condition or an earlier failed stage © 2017 CloudBees, Inc. All Rights Reserved

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

Resources on Declarative • Documentation https://jenkins.io/doc/ • Examples https://github.com/jenkinsci/pipeline-examples • Plugin source https://github.com/jenkinsci/pipeline-model-definition-plugin © 2017 CloudBees, Inc. All Rights Reserved

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Pipeline: a new job type -

Slide 47

Slide 47 text

Key Benefits ✓ Long-running ✓ Durable ✓ Scriptable ✓ One-place for Everything Pipeline: a new job type ✓ Makes building Pipelines Simple

Slide 48

Slide 48 text

Domain Specific Language Simple, Groovy-based DSL (“Domain Specific Language”) to manage build step orchestration Can be defined as DSL in Jenkins or as Jenkinsfile in SCM Groovy is widely used in Jenkins ecosystem. You can leverage the large Jenkins Groovy experience If you don’t like/care about Groovy, just consider the DSL as a dedicated simple syntax

Slide 49

Slide 49 text

Snippet Generator 49

Slide 50

Slide 50 text

Create a Pipeline - core steps 50 stage - group your steps into its component parts and control concurrency node - schedules the steps to run by adding them to Jenkins' build queue and creates a workspace specific to this pipeline sh (or bat) - execute shell (*nix) or batch scripts (Windows) just like freestyle jobs, calling any tools in the agent

Slide 51

Slide 51 text

Create a Pipeline stage('build') { echo 'hello from jenkins master' node { sh 'echo hello from jenkins agent' } } stage('test') { echo 'test some things' } stage('deploy') { echo 'deploy some things' } 51

Slide 52

Slide 52 text

Lab Exercise: Checkout from SCM

Slide 53

Slide 53 text

Files stash - store some files for later use unstash - retrieve previously stashed files (even across nodes) writeFile - write a file to the workspace readFile - read a file from the workspace 53

Slide 54

Slide 54 text

Checkout from SCM and stash Files stage('build') { node { git 'https://github.com/beedemo/mobile-deposit-api.git' writeFile encoding: 'UTF-8', file: 'config', text: 'version=1' stash includes: 'pom.xml,config', name: 'pom-config' } } stage('test'){ node { unstash 'pom-config' configValue = readFile encoding: 'UTF-8', file: 'config' echo configValue } } 54

Slide 55

Slide 55 text

try up to N times retry(5) { // some block } wait for a condition waitUntil { // some block } Flow Control wait for a set time sleep time: 1000, unit:'NANOSECONDS' timeout timeout(time: 30, unit: 'SECONDS'){ // some block }

Slide 56

Slide 56 text

You can also rely on Groovy control flow syntax! while(something) { // do something if (something_else) { // do something else } } Flow Control 56 try{ //some things }catch(e){ // }

Slide 57

Slide 57 text

Advanced Flow Control input - pause for manual or automated approval parallel - allows simultaneous execution of build steps on the current node or across nodes, thus increasing build speed parallel 'quality scan': { node {sh 'mvn sonar:sonar'} }, 'integration test': { node {sh 'mvn verify'} } checkpoint - capture the workspace state so it can be reused as a starting point for subsequent runs

Slide 58

Slide 58 text

Lab Exercise: Input and Checkpoints

Slide 59

Slide 59 text

Input Approval & Checkpoints 59 stage('deploy') { input message: 'Do you want to deploy?' node { echo 'deployed' } }

Slide 60

Slide 60 text

Input Approval & Checkpoints 60 checkpoint 'testing-complete' stage('approve') { mail body: "Approval needed for '${env.JOB_NAME}' at '${env.BUILD_URL}'", subject: "${env.JOB_NAME} Approval", to: "[email protected]" timeout(time: 7, unit: 'DAYS') { input message: 'Do you want to deploy?', parameters: [string(defaultValue: '', description: 'Provide any comments regarding decision.', name: 'Comments')], submitter: 'ops' } }

Slide 61

Slide 61 text

Lab Exercise: Tool Management

Slide 62

Slide 62 text

tool Step ●Binds a tool installation to a variable ●The tool home directory is returned ●Only tools already configured are available def mvnHome = tool 'M3' sh "${mvnHome}/bin/mvn -B verify" 62

Slide 63

Slide 63 text

Use Docker Containers ●The CloudBees Docker Pipeline plugin allows running steps inside a container ●You can even build the container as part of the same Pipeline docker.image('maven:3.3.3-jdk-8').inside() { sh 'mvn -B verify' } 63

Slide 64

Slide 64 text

Lab Exercise: Pipeline as Code

Slide 65

Slide 65 text

Pipeline script in SCM 65

Slide 66

Slide 66 text

Pipeline Multibranch 66

Slide 67

Slide 67 text

Pipeline Shared Libraries

Slide 68

Slide 68 text

What Next?

Slide 69

Slide 69 text

More Advanced Steps Send email mail body: 'Uh oh.', subject: 'Build Failed!', to: '[email protected]' step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: '[email protected]']) Deploy to Amazon Elastic Beanstalk wrap([$class: 'AmazonAwsCliBuildWrapper', credentialsId: 'aws-beanstalk- credentials', defaultRegion: 'us-east-1']) { sh 'aws elasticbeanstalk create-application-version' } Integrate with Jira jiraComment(issueKey: "EX-111", body: "Job '${env.JOB_NAME}' ($ {env.BUILD_NUMBER}) built. Please go to ${env.BUILD_URL}.")

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

Gartner: “Using Docker to Run Build Nodes is Ideal.” Agent Master z Agent Agent

Slide 72

Slide 72 text

Accelerating CD with Containers Workflow CD Pipeline Triggers: ✓ New application code (feature, bug fix, etc.) ✓ Updated certified stack (security fix in Linux, etc.) ✓ Will lead to a new gold image being built and available for … TESTING … STAGING … PRODUCTION ✓ All taking place in a standardized/similar/consistent OS environment + Jenkins Pipeline TEST STAG E PRODUCTIO N App


(git, etc.)
Gold
Docker
Image
(~per app)

Certified
Docker
Images
(Ubuntu, etc.)

Slide 73

Slide 73 text

Jenkins Pipeline Docker Commands ● withRegistry Specifies which Docker Registry to use for pushing/pulling ● withServer Specifies which Server to issue Docker commands against ● Build Builds container from Dockerfile ● Image.run Runs a container ● Image.inside Runs container and allows you to execute command inside the container ● Image.push Pushed image to Docker Registry with specified credentials

Slide 74

Slide 74 text

Pipeline - Docker Example stage 'Build' node('docker-cloud') { checkout scm docker.image('java:jdk-8').inside('-v /data:/data') { sh "mvn clean package" }} stage 'Quality Analysis' node('docker-cloud') { unstash 'pom' parallel( JRE8Test: { docker.image('java:jre-8').inside('-v /data:/data') { sh 'mvn -Dmaven.repo.local=/data/mvn/repo verify' } },JRE7Test: { docker.image('java:jre-7').inside('-v /data:/data') { sh 'mvn -Dmaven.repo.local=/data/mvn/repo verify' } }, failFast: true )}

Slide 75

Slide 75 text

Pipeline - Docker Example node('docker-cloud') { stage 'Build Docker Image' dir('target') { mobileDepositApiImage = docker.build "beedemo/mobile-deposit-api:$ {dockerTag}" } stage 'Publish Docker Image' withDockerRegistry(registry: [credentialsId: 'docker-hub-beedemo']) { mobileDepositApiImage.push() } }

Slide 76

Slide 76 text

No content