Andreas Caternberg on Jenkins Pipelines

Transcript

1. Let’s Build a Jenkins Pipeline
   Andreas Caternberg
   

   View Slide

2. Download Slides Now
   https://gist.github.com/cccaternberg/3984e4a3d1a84ec6faffbd728ce6c247
   

   View Slide

3. About Your Guide
   Andreas Caternberg
   Senior Consultant at CloudBees
   @acaternberg
   

   View Slide

4. Plan for today
   

   View Slide

5. CloudBees Jenkins Enterprise
   © 2017 CloudBees, Inc. All Rights
   Reserved
   • Community Innovation
   • Jenkins for the
   Enterprise
   

   View Slide

6. What we are going to cover
   • What is a pipeline
   • Compare scripted pipeline to declarative
   pipeline
   • Create a declarative pipeline
   • Learn about Blue Ocean
   • Share pipeline resources
   • Extra scripted pipeline information
   © 2017 CloudBees, Inc. All Rights
   Reserved
   

   View Slide

7. What is a Jenkins pipeline
   • Formerly known as Jenkins Workflow, Jenkins Pipeline was
   introduced in 2014
   • New job type – a single Groovy script using an embedded DSL - no
   need to jump between multiple job configuration screens to see
   what is going on
   • Durable - keeps running even if Jenkins master is not
   • Distributable – a Pipeline job may be run across an unlimited
   number of nodes, including in parallel
   • Pausable – allows waiting for input from users before continuing to
   the next step
   • Visualized - Pipeline Stage View provides status at-a-glance
   dashboards to include trending
   © 2017 CloudBees, Inc. All Rights
   Reserved
   

   View Slide

8. What is a declarative pipeline?
   • A syntax for Jenkins Pipelines designed to…
   –Have smarter default behavior for most use
   cases
   –Make Pipelines more structured
   –Provide better error reporting and handling
   © 2017 CloudBees, Inc. All Rights
   Reserved
   

   View Slide

9. Have a smarter behavior
   • Default SCM checkout
   • Built with Docker and containers in mind from day one
   • No more need for try/catch for notifications, etc.
   • More intuitive ways to specify job properties, triggers
   and parameters
   • Streamlined syntax for conditional execution of stages
   © 2017 CloudBees, Inc. All Rights
   Reserved
   

   View Slide

10. More structure in Pipelines
    • Separate configuration from actual steps to be executed
    • Everything happens in a stage
    • Easier to look at any Declarative Pipeline and understand
    what it’s doing - more consistency!
    • Enables round-tripping with the visual editor (more on this
    later!)
    • Built with and for Blue Ocean visualization
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

11. Better error reporting
    ● Syntax and validation errors reported at the very beginning of the
    build
    ● No more waiting until a build has gone an hour to discover that you
    misspelled “timeout”!
    ● Errors report as compilation errors, pointing to the specific code or
    text that’s causing problems
    ● Eliminates many potentially confusing stack traces (though not all!)
    ● Validates things like syntax, required configuration, step parameter
    types, and more
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

12. What happens to Scripted Pipeline?
    ● In short? Nothing.
    ● We’re now calling “traditional” Pipeline “Scripted Pipeline” to distinguish it from
    Declarative Pipeline
    ● All one execution engine behind the scenes
    ○Declarative is built on top of Scripted Pipeline, not replacing it
    ○Blue Ocean and Stage View don’t distinguish between a Scripted Pipeline or a
    Declarative Pipeline
    ● Scripted Pipeline still used *inside* Declarative Pipeline in steps blocks, post
    blocks
    ○You can copy the contents of a Declarative steps block into a Scripted Pipeline (inside
    a node block, of course!) and it’ll Just Work.
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

13. Benefits of Declarative for new users
    ● Lower barrier of entry than Scripted Pipelines
    ● Human readable
    ● Does not require Groovy-specific expertise
    ● Create and edit Declarative Pipelines via the
    upcoming UI Editor
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

14. Benefits of Declarative for existing users
    ● Uses same engine as Scripted, so existing investments still pay off
    ● Expand usage and usability
    ○Less burden on “experts”
    ○Easier empowerment of other users
    ● Easier collaboration and code review
    ● Lintable!
    ● Separation of Jenkins infrastructure-related configuration from
    steps to execute
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

15. Syntax Overview
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

16. Pipeline Comparison
    © 2017 CloudBees, Inc. All Rights
    Reserved
    Scripted Pipeline
    node {
    try {
    stage('Build') {
    checkout scm
    docker.image('ubuntu').inside {
    sh 'mvn clean install'
    }
    }
    }
    catch(exc) {
    mail to:'me@ex.com', subject:'FAILURE:'
    }
    finally {
    deleteDir()
    }
    }
    Declarative Pipeline
    pipeline {
    agent {
    docker 'ubuntu'
    }
    stages {
    stage('Build') {
    steps {
    sh 'mvn clean install'
    }
    }
    }
    post {
    always {
    deleteDir()
    }
    failure {
    mail to:'me@ex.com', subject:'FAILURE:'
    }
    }
    }
    

    View Slide

17. First example
    © 2017 CloudBees, Inc. All Rights
    Reserved
    https://gist.github.com/cccaternberg/1d2169d445d04d5d6264a241fbc53e8e
    

    View Slide

18. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

19. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

20. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

21. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

22. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

23. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

24. Second Example
    © 2017 CloudBees, Inc. All Rights
    Reserved
    https://gist.github.com/cccaternberg/b45e58163725c3ca81f76a55e1328009
    

    View Slide

25. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

26. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

27. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

28. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

29. Time to build a pipeline
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

30. Get Ready
    • Log into our CloudBees Jenkins Operations Center
    http://dpa.beedemo.net/pipeline-
    workshop/
    • Create an account
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

31. Get Ready
    • Log in
    • Go to the LBAJP server
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

32. Create a Pipeline Job
    © 2017 CloudBees, Inc. All Rights
    Reserved
    • Select ”New Item” • Name your job
    

    View Slide

33. A basic pipeline
    pipeline {
    agent any
    stages {
    stage('Build') {
    steps {
    echo 'Hello from Build'
    }
    }
    stage('Test') {
    steps {
    echo 'Testing Testing 123'
    }
    }
    stage('Deploy') {
    steps {
    echo 'Deploy some things'
    }
    }
    }
    }
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

34. Add an agent
    © 2017 CloudBees, Inc. All Rights
    Reserved
    pipeline {
    agent any
    stages {
    pipeline {
    agent {
    label ‘docker’
    }
    stages {
    

    View Slide

35. Stash Files
    © 2017 CloudBees, Inc. All Rights
    Reserved
    stages {
    stage('Build') {
    steps {
    echo 'Hello from Build’
    writeFile(file: 'config', text: 'version=1', encoding: 'UTF-8')
    stash(name: 'pom-config', includes: 'config')
    }
    }
    stage('Test') {
    environment {
    configValue = readFile(encoding: 'UTF-8', file: 'config')
    }
    steps {
    echo 'Testing Testing 123’
    unstash 'pom-config'
    sh 'echo "$configValue"'
    }
    }
    

    View Slide

36. Create an input step
    stage('Deploy') {
    steps {
    echo 'Deploy some things'
    input 'Do you want to deploy?'
    echo 'deployed'
    }
    }
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

37. Blue Ocean
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

38. Blue Ocean Visualization
    ● Some special smarts for optimized visualization
    ● Blue Ocean keeps built-in stages for things like
    checkout, Docker image prep, post-build actions
    out of your way unless there’s a problem
    ● Special marking of stages that have been skipped
    due to either an unsatisfied when condition or an
    earlier failed stage
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

39. Blue Ocean Editor
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

40. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

41. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

42. © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

43. Reference Card
    https://www.cloudbees.com/sites/default/files/declarative-pipeline-refcard.pdf
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

44. Resources on Declarative
    • Documentation
    https://jenkins.io/doc/
    • Examples
    https://github.com/jenkinsci/pipeline-examples
    • Plugin source
    https://github.com/jenkinsci/pipeline-model-definition-plugin
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

45. Scripted Pipeline
    © 2017 CloudBees, Inc. All Rights
    Reserved
    

    View Slide

46. Pipeline: a new job type
    -
    

    View Slide

47. Key Benefits
    ✓ Long-running
    ✓ Durable
    ✓ Scriptable
    ✓ One-place for
    Everything
    Pipeline: a new job type
    ✓ Makes building Pipelines
    Simple
    

    View Slide

48. Domain Specific Language
    Simple, Groovy-based DSL (“Domain
    Specific Language”) to manage build
    step orchestration
    Can be defined as DSL in Jenkins or as
    Jenkinsfile in SCM
    Groovy is widely used in Jenkins
    ecosystem. You can leverage the large
    Jenkins Groovy experience
    If you don’t like/care about Groovy, just
    consider the DSL as a dedicated simple
    syntax
    

    View Slide

49. Snippet Generator
    49
    

    View Slide

50. Create a Pipeline - core steps
    50
    stage - group your steps into its component
    parts and control concurrency
    node - schedules the steps to run by adding
    them to Jenkins' build queue and creates a
    workspace specific to this pipeline
    sh (or bat) - execute shell (*nix) or batch
    scripts (Windows) just like freestyle jobs,
    calling any tools in the agent
    

    View Slide

51. Create a Pipeline
    stage('build') {
    echo 'hello from jenkins master'
    node {
    sh 'echo hello from jenkins agent'
    }
    }
    stage('test') {
    echo 'test some things'
    }
    stage('deploy') {
    echo 'deploy some things'
    }
    51
    

    View Slide

52. Lab Exercise:
    Checkout from SCM
    

    View Slide

53. Files
    stash - store some files for later use
    unstash - retrieve previously stashed
    files (even across nodes)
    writeFile - write a file to the workspace
    readFile - read a file from the
    workspace
    53
    

    View Slide

54. Checkout from SCM and stash Files
    stage('build') {
    node {
    git 'https://github.com/beedemo/mobile-deposit-api.git'
    writeFile encoding: 'UTF-8', file: 'config', text: 'version=1'
    stash includes: 'pom.xml,config', name: 'pom-config'
    }
    }
    stage('test'){
    node {
    unstash 'pom-config'
    configValue = readFile encoding: 'UTF-8', file: 'config'
    echo configValue
    }
    }
    54
    

    View Slide

55. try up to N
    times
    retry(5) {
    // some block
    }
    wait for a
    condition
    waitUntil {
    // some block
    }
    Flow Control
    wait for a set time
    sleep time: 1000, unit:'NANOSECONDS'
    timeout
    timeout(time: 30, unit: 'SECONDS'){
    // some block
    }
    

    View Slide

56. You can also rely on Groovy control
    flow syntax!
    while(something) {
    // do something
    if (something_else) {
    // do something else
    }
    }
    Flow Control
    56
    try{
    //some things
    }catch(e){
    //
    }
    

    View Slide

57. Advanced Flow Control
    input - pause for manual or automated approval
    parallel - allows simultaneous execution of build steps on the
    current node or across nodes, thus increasing build speed
    parallel 'quality scan': {
    node {sh 'mvn sonar:sonar'}
    }, 'integration test': {
    node {sh 'mvn verify'}
    }
    checkpoint - capture the workspace
    state so it can be reused as a starting
    point for subsequent runs
    

    View Slide

58. Lab Exercise:
    Input and Checkpoints
    

    View Slide

59. Input Approval & Checkpoints
    59
    stage('deploy') {
    input message: 'Do you want to
    deploy?'
    node {
    echo 'deployed'
    }
    }
    

    View Slide

60. Input Approval & Checkpoints
    60
    checkpoint 'testing-complete'
    stage('approve') {
    mail body: "Approval needed for '${env.JOB_NAME}' at '${env.BUILD_URL}'",
    subject: "${env.JOB_NAME} Approval",
    to: "ops@yourcompanyhere.com"
    timeout(time: 7, unit: 'DAYS') {
    input message: 'Do you want to deploy?', parameters: [string(defaultValue: '',
    description: 'Provide any comments regarding decision.', name: 'Comments')], submitter:
    'ops'
    }
    }
    

    View Slide

61. Lab Exercise:
    Tool Management
    

    View Slide

62. tool Step
    ●Binds a tool installation to a variable
    ●The tool home directory is returned
    ●Only tools already configured are
    available
    def mvnHome = tool 'M3'
    sh "${mvnHome}/bin/mvn -B verify"
    62
    

    View Slide

63. Use Docker Containers
    ●The CloudBees Docker Pipeline plugin
    allows running steps inside a container
    ●You can even build the container as part
    of the same Pipeline
    docker.image('maven:3.3.3-jdk-8').inside() {
    sh 'mvn -B verify'
    }
    63
    

    View Slide

64. Lab Exercise:
    Pipeline as Code
    

    View Slide

65. Pipeline script in SCM
    65
    

    View Slide

66. Pipeline Multibranch
    66
    

    View Slide

67. Pipeline Shared Libraries
    

    View Slide

68. What Next?
    

    View Slide

69. More Advanced Steps
    Send email
    mail body: 'Uh oh.', subject: 'Build Failed!', to: 'dev@cloudbees.com'
    step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients:
    'info@cloudbees.com'])
    Deploy to Amazon Elastic Beanstalk
    wrap([$class: 'AmazonAwsCliBuildWrapper', credentialsId: 'aws-beanstalk-
    credentials', defaultRegion: 'us-east-1']) {
    sh 'aws elasticbeanstalk create-application-version'
    }
    Integrate with Jira
    jiraComment(issueKey: "EX-111", body: "Job '${env.JOB_NAME}' ($
    {env.BUILD_NUMBER}) built. Please go to ${env.BUILD_URL}.")
    

    View Slide

70. View Slide

71. Gartner: “Using Docker to Run Build Nodes is Ideal.”
    Agent
    Master
    z
    Agent
    Agent
    

    View Slide

72. Accelerating CD with Containers
    Workflow CD Pipeline Triggers:
    ✓ New application code (feature, bug fix, etc.)
    ✓ Updated certified stack (security fix in Linux, etc.)
    ✓ Will lead to a new gold image being built and
    available for … TESTING … STAGING …
    PRODUCTION
    ✓ All taking place in a standardized/similar/consistent
    OS environment
    +
    Jenkins Pipeline
    TEST
    STAG
    E
    PRODUCTIO
    N
    App
    
    (git, etc.)
    Gold
    Docker
    Image
    (~per app)
    config>
    Certified
    Docker
    Images
    (Ubuntu, etc.)
    
    

    View Slide

73. Jenkins Pipeline Docker Commands
    ● withRegistry
    Specifies which Docker Registry to use for pushing/pulling
    ● withServer
    Specifies which Server to issue Docker commands against
    ● Build
    Builds container from Dockerfile
    ● Image.run
    Runs a container
    ● Image.inside
    Runs container and allows you to execute command inside the
    container
    ● Image.push
    Pushed image to Docker Registry with specified credentials
    

    View Slide

74. Pipeline - Docker Example
    stage 'Build'
    node('docker-cloud') {
    checkout scm
    docker.image('java:jdk-8').inside('-v /data:/data') {
    sh "mvn clean package" }}
    stage 'Quality Analysis'
    node('docker-cloud') {
    unstash 'pom'
    parallel(
    JRE8Test: {
    docker.image('java:jre-8').inside('-v /data:/data') {
    sh 'mvn -Dmaven.repo.local=/data/mvn/repo verify' }
    },JRE7Test: {
    docker.image('java:jre-7').inside('-v /data:/data') {
    sh 'mvn -Dmaven.repo.local=/data/mvn/repo verify' }
    }, failFast: true )}
    

    View Slide

75. Pipeline - Docker Example
    node('docker-cloud') {
    stage 'Build Docker Image'
    dir('target') {
    mobileDepositApiImage = docker.build "beedemo/mobile-deposit-api:$
    {dockerTag}"
    }
    stage 'Publish Docker Image'
    withDockerRegistry(registry: [credentialsId: 'docker-hub-beedemo']) {
    mobileDepositApiImage.push()
    }
    }
    

    View Slide

76. View Slide