Under the hood of serverless Jenkins. Jenkinsfile Runner

Under the hood of serverless Jenkins. Jenkinsfile Runner

There were significant changes in the Jenkins project over last year. It is still one of the most popular automation servers in the world, but it is not just a “server” anymore. In 2019 Jenkins is a universal automation engine, which actively evolves towards the new Cloud Native architecture and which can be used in any CI/CD flows. It can be used not only as a server, but also as a “serverless” single-shot execution. And it is not only about Jenkins X, but also about more classic usages.

In my talk I will present Jenkinsfile Runner - an engine which allows running Jenkins Pipelines in a single-shot container. I will show how it is used in Jenkins X, GitHub Actions and other services. And, finally, I will show how to package your own Jenkinsfile Runner images and run it in any environments, CI systems and Cloud providers.

NOTE: This slidedeck will be periodically updated to reflect the latest version of the talk

568e3391c8b528f2b255443e4cca27ca?s=128

Oleg Nenashev

May 14, 2019
Tweet

Transcript

  1. © 2019 CloudBees, Inc. All Rights Reserved. Under the hood

    of serverless Jenkins. Jenkinsfile Runner Oleg Nenashev, CloudBees @oleg_nenashev May 14, 2019 1
  2. © 2019 CloudBees, Inc. All Rights Reserved. TL;DR “You can

    run Jenkins Pipelines as single-shot container in any environment” This talk: How to do it? 2 These slides: https://twitter.com/oleg_nenashev http://bit.ly/devopsdays_zurich_jfr
  3. © 2019 CloudBees, Inc. All Rights Reserved. > whoami 3

    @oleg_nenashev oleg-nenashev • Principal SW Engineer, CloudBees • Jenkins maintainer & ambassador • Swiss Jenkins Meetup organizer
  4. © 2019 CloudBees, Inc. All Rights Reserved. About you 4

    These slides: https://twitter.com/oleg_nenashev http://bit.ly/devopsdays_zurich_jfr
  5. © 2019 CloudBees, Inc. All Rights Reserved. 1. It’s a

    framework => flexible 2. It’s open source => customizable 3. More than 1600 plugins 4. Big community 5. Commercial support is available Who is Mr. Jenkins?
  6. © 2019 CloudBees, Inc. All Rights Reserved. 6 Jenkins in

    2019 Jenkins Pipeline Config-as-Code Plugin Modern plugins Modern packaging Jenkins X
  7. © 2019 CloudBees, Inc. All Rights Reserved. 7 Permanent agents

    On-demand agents from cloud providers Jenkins Master Web UI
  8. © 2019 CloudBees, Inc. All Rights Reserved. Master as a

    SPoF Macroservice 8 Permanent agents On-demand agents from cloud providers Web UI Jenkins Master
  9. © 2018 CloudBees, Inc. All Rights Reserved. 9 (┛ಠ_ಠ)┛彡┻━┻ What

    if Jenkins master was “on-demand”? 9
  10. © 2018 CloudBees, Inc. All Rights Reserved. Aug 31, 2018

    – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ Slides: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 10
  11. © 2018 CloudBees, Inc. All Rights Reserved. Cloud Native Jenkins

    • Best service for each need • Pay per use • “Infinite” scaling • Easy to use • Fast to develop https://flic.kr/p/75ncBF 11
  12. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner https://github.com/jenkinsci/jenkinsfile-runner

    Started by Kohsuke in Mar 2018 3 maintainers, 19 contributors Status: Beta (1.0-beta-8) 12
  13. © 2019 CloudBees, Inc. All Rights Reserved. JFR TL;DR •

    Binary and base Docker image • Runs Pipeline and shuts down • Prints log output to STDOUT • Workspace mapping as a volume • No Web UI included https://github.com/jenkinsci/jenkinsfile-runner 13
  14. © 2019 CloudBees, Inc. All Rights Reserved. Usage • Pass

    workspace with project/Jenkinsfile • Pass cache for local data (if needed) docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner 14
  15. © 2019 CloudBees, Inc. All Rights Reserved. DEMO https://github.com/jenkinsci/jenkinsfile-runner/tree/master/demo/cwp use

    the 1.0-beta-8 tag 15
  16. © 2019 CloudBees, Inc. All Rights Reserved. Demo. Our Jenkinsfile

    16 docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner
  17. © 2019 CloudBees, Inc. All Rights Reserved. 17

  18. © 2019 CloudBees, Inc. All Rights Reserved. Features Startup in

    few seconds • Performance tweaks Advanced options • Script sandbox can be disabled (--ns) • Pipeline parameters support Java 11 support ◦ Better garbage collector + optimization ◦ cgroups support => containers resource limits 18
  19. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors

    19 Vanilla: Docker/Binary in the official repo • https://github.com/jenkinsci/jenkinsfile-runner • https://hub.docker.com/r/jenkins4eval/jenkinsfile-runner
  20. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors

    Vanilla: Docker/Binary in the official repo • https://github.com/jenkinsci/jenkinsfile-runner • https://hub.docker.com/r/jenkins4eval/jenkinsfile-runner Custom JFR packages • https://jenkins.io/blog/2018/10/16/custom-war-packager/ • Ready-to-fly Docker images with plugins and configs • Way to go in 95% of cases 20
  21. © 2019 CloudBees, Inc. All Rights Reserved. Building custom Jenkinsfile

    Runner images Custom WAR Packager Jenkinsfile Runner Core and Plugins Configuration-as-code Packaging Ready-to-fly Docker image Base image https://jenkins.io/blog/2018/10/16/custom-war-packager/ 21
  22. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager

    and Self-configuration 1. Custom WAR Packager is managed by a config YAML 2. Docker image: plugins.txt, Java flags, etc. 3. Groovy Init Scripts 4. AND: New Configuration-as-Code Plugin https://plugins.jenkins.io/configuration-as-code Allows configuring Jenkins from YAML 22 https://github.com/jenkinsci/custom-war-packager
  23. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager

    Demos • https://github.com/jenkinsci/custom-war-packager#demo • https://github.com/jenkinsci/ci.jenkins.io-runner/ 23
  24. © 2019 CloudBees, Inc. All Rights Reserved. Just building the

    stuff is not enough 24
  25. © 2019 CloudBees, Inc. All Rights Reserved. 25

  26. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test

    Framework https://github.com/jenkinsci/jenkinsfile-runner-test-framework • Custom WAR Packager • Docker • shUnit2 26
  27. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test

    Framework Jenkinsfile Runner packages Jenkinsfiles Pipeline libraries 27 Tests
  28. © 2019 CloudBees, Inc. All Rights Reserved. Examples Jenkins X,

    GitHub Actions, CloudBees CodeShip 28
  29. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner. Usage

    examples Jenkins X, Serverless mode (before Jenkins X 2.0 with Tekton) • https://github.com/jenkins-x/jenkins-x-serverless ci.jenkins.io-runner • https://github.com/jenkinsci/ci.jenkins.io-runner “FaaS” packaging • GitHub Actions: https://github.com/jonico/jenkinsfile-runner-github-actions • Codeship: https://github.com/oleg-nenashev/codeship-jenkinsfile • AWS Lambda: https://github.com/carlossg/jenkinsfile-runner-lambda • Project Fn: https://github.com/carlossg/jenkinsfile-runner-fn 29
  30. © 2019 CloudBees, Inc. All Rights Reserved. 30 https://jenkins-x.io

  31. © 2019 CloudBees, Inc. All Rights Reserved. 31

  32. © 2019 CloudBees, Inc. All Rights Reserved. Details about Jenkins

    X Paolo Carta, Matteo Baiguini, Continuous Deployment of Polyglot Microservices with Istio • https://devopsdays.org/events/2019-zurich/program/workshop-with- matteo-baiguini-and-paolo-carta/ Cosmin Cojocar, Jenkins X contributor • CloudBees Booth on Wednesday • Both Jenkinsfile runner and Tekton flows 32
  33. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins X. Modes

    and engines 33 Static Masters Serverless build engines Jenkinsfile Runner Tekton (default in 2.0) Classic Jenkins
  34. © 2018 CloudBees, Inc. All Rights Reserved. Jenkins X. Serverless

    Mode (before April 2019) 34 https://medium.com/@jdrawlings/ser verless-jenkins-with-jenkins-x-9134cb fe6870 Serverless Jenkins with Jenkins X
  35. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins X Serverless

    Build flow Single-shot masters build flow 35
  36. © 2019 CloudBees, Inc. All Rights Reserved. https://medium.com/@jdrawlings/serverless-jenki ns-with-jenkins-x-9134cbfe6870 Jenkinsfile

    Runner is Here 36 Jenkinsfile Runner
  37. © 2018 CloudBees, Inc. All Rights Reserved. CommentOps - communication

    with users 37
  38. © 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions …

    and Jenkinsfile Runner 38
  39. © 2019 CloudBees, Inc. All Rights Reserved. https://github.com/features/actions 39

  40. © 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions! Not

    designed to be CI, useful for short tasks/workflows https://github.com/jonico/jenkinsfile-runner-github-actions 40
  41. © 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions is

    in Limited Beta … and I am still waiting for my limited Beta access to be approved 41
  42. © 2019 CloudBees, Inc. All Rights Reserved. And CodeShip? 42

  43. © 2019 CloudBees, Inc. All Rights Reserved. CloudBees CodeShip •

    Jenkins has no good SaaS • CodeShip Pro - CI SaaS with native Docker support • Free tiers for open-source projects • What if we run Jenkins there? 43 https://codeship.com/features/pro
  44. © 2019 CloudBees, Inc. All Rights Reserved. Demo 1 44

    codeship-steps.yml codeship-services.yml Jenkinsfile https://github.com/oleg-nenashev/codeship-jenkinsfile
  45. © 2019 CloudBees, Inc. All Rights Reserved. 45

  46. © 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building

    a Jenkins plugin in CodeShip https://github.com/oleg-nenashev/mailer-plugin/tree/codeship-demo • Using ci.jenkins.io-runner ◦ https://github.com/jenkinsci/ci.jenkins.io-runner • Using the standard Jenkins Pipeline library ◦ https://github.com/jenkins-infra/pipeline-library • Using standard Jenkinsfile ◦ buildPlugin() 46
  47. © 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building

    a Jenkins plugin in CodeShip codeship-services.yml codeship-steps.yml 47 Jenkinsfile
  48. © 2019 CloudBees, Inc. All Rights Reserved. 48

  49. © 2019 CloudBees, Inc. All Rights Reserved. CodeShip. Build history

    49
  50. © 2019 CloudBees, Inc. All Rights Reserved. So... 50

  51. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner runs

    everywhere 51 docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner • Docker images • CLI Tool - on Java 8 or Java 11
  52. © 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner is

    not a silver bullet! 52
  53. © 2019 CloudBees, Inc. All Rights Reserved. Limitations Jenkinsfile Runner

    is not fully compatible • No Web UI, no way to extract Jenkins reports • No native support of external storage • No build throttling, cross-master communication, etc. Single container, single agent • Docker and Kubernetes plugins cannot be easily used to produce sidecar containers • https://github.com/jenkinsci/remoting/pull/286 53
  54. © 2019 CloudBees, Inc. All Rights Reserved. Future work Waiting

    for user feedback Integrating patches • https://github.com/jenkinsci/jenkinsfile-runner/blob/master/CONTRIBUTING.md (?) Jenkinsfile Runner 1.0 (?) Sidecar containers Experiments with GraalVM and Quarkus 54
  55. © 2019 CloudBees, Inc. All Rights Reserved. Takeaways • serverless

    Jenkins is real • you can run your Pipelines everywhere • there are tools for that! • use Jenkins X if applicable (K8s/GitOps) 55
  56. © 2019 CloudBees, Inc. All Rights Reserved. Demos and Questions

    CloudBees Booth Demos... • Jenkins • Jenkinsfile Runner • Jenkins X - Wed • CodeShip & Co • Config-as-Code • Pipeline devel 56 We are here
  57. © 2019 CloudBees, Inc. All Rights Reserved. Contacts: E-mail: onenashev@cloudbees.com

    GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com 57
  58. © 2019 CloudBees, Inc. All Rights Reserved. > whoami -cloudbees

    • CloudBees Jenkins Distribution • CloudBees Jenkins Support • Community (Java 11 support, Jenkinsfile Runner, Security) https://www.cloudbees.com 58
  59. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins as you

    may knew it… in 2012 Complex configuration Mega-masters, limited scalability Painful upgrades Masters are always running . . . 59