Tweet
Tweet

Slide 1

Slide 1 text

© 2019 CloudBees, Inc. All Rights Reserved. Under the hood of serverless Jenkins. Jenkinsfile Runner Oleg Nenashev, CloudBees @oleg_nenashev May 14, 2019 1

Slide 2

Slide 2 text

© 2019 CloudBees, Inc. All Rights Reserved. TL;DR “You can run Jenkins Pipelines as single-shot container in any environment” This talk: How to do it? 2 These slides: https://twitter.com/oleg_nenashev http://bit.ly/devopsdays_zurich_jfr

Slide 3

Slide 3 text

© 2019 CloudBees, Inc. All Rights Reserved. > whoami 3 @oleg_nenashev oleg-nenashev • Principal SW Engineer, CloudBees • Jenkins maintainer & ambassador • Swiss Jenkins Meetup organizer

Slide 4

Slide 4 text

© 2019 CloudBees, Inc. All Rights Reserved. About you 4 These slides: https://twitter.com/oleg_nenashev http://bit.ly/devopsdays_zurich_jfr

Slide 5

Slide 5 text

© 2019 CloudBees, Inc. All Rights Reserved. 1. It’s a framework => flexible 2. It’s open source => customizable 3. More than 1600 plugins 4. Big community 5. Commercial support is available Who is Mr. Jenkins?

Slide 6

Slide 6 text

© 2019 CloudBees, Inc. All Rights Reserved. 6 Jenkins in 2019 Jenkins Pipeline Config-as-Code Plugin Modern plugins Modern packaging Jenkins X

Slide 7

Slide 7 text

© 2019 CloudBees, Inc. All Rights Reserved. 7 Permanent agents On-demand agents from cloud providers Jenkins Master Web UI

Slide 8

Slide 8 text

© 2019 CloudBees, Inc. All Rights Reserved. Master as a SPoF Macroservice 8 Permanent agents On-demand agents from cloud providers Web UI Jenkins Master

Slide 9

Slide 9 text

© 2018 CloudBees, Inc. All Rights Reserved. 9 (┛ಠ_ಠ)┛彡┻━┻ What if Jenkins master was “on-demand”? 9

Slide 10

Slide 10 text

© 2018 CloudBees, Inc. All Rights Reserved. Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ Slides: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 10

Slide 11

Slide 11 text

© 2018 CloudBees, Inc. All Rights Reserved. Cloud Native Jenkins ● Best service for each need ● Pay per use ● “Infinite” scaling ● Easy to use ● Fast to develop https://flic.kr/p/75ncBF 11

Slide 12

Slide 12 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner https://github.com/jenkinsci/jenkinsfile-runner Started by Kohsuke in Mar 2018 3 maintainers, 19 contributors Status: Beta (1.0-beta-8) 12

Slide 13

Slide 13 text

© 2019 CloudBees, Inc. All Rights Reserved. JFR TL;DR ● Binary and base Docker image ● Runs Pipeline and shuts down ● Prints log output to STDOUT ● Workspace mapping as a volume ● No Web UI included https://github.com/jenkinsci/jenkinsfile-runner 13

Slide 14

Slide 14 text

© 2019 CloudBees, Inc. All Rights Reserved. Usage ● Pass workspace with project/Jenkinsfile ● Pass cache for local data (if needed) docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner 14

Slide 15

Slide 15 text

© 2019 CloudBees, Inc. All Rights Reserved. DEMO https://github.com/jenkinsci/jenkinsfile-runner/tree/master/demo/cwp use the 1.0-beta-8 tag 15

Slide 16

Slide 16 text

© 2019 CloudBees, Inc. All Rights Reserved. Demo. Our Jenkinsfile 16 docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner

Slide 17

Slide 17 text

© 2019 CloudBees, Inc. All Rights Reserved. 17

Slide 18

Slide 18 text

© 2019 CloudBees, Inc. All Rights Reserved. Features Startup in few seconds ● Performance tweaks Advanced options ● Script sandbox can be disabled (--ns) ● Pipeline parameters support Java 11 support ○ Better garbage collector + optimization ○ cgroups support => containers resource limits 18

Slide 19

Slide 19 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors 19 Vanilla: Docker/Binary in the official repo ● https://github.com/jenkinsci/jenkinsfile-runner ● https://hub.docker.com/r/jenkins4eval/jenkinsfile-runner

Slide 20

Slide 20 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors Vanilla: Docker/Binary in the official repo ● https://github.com/jenkinsci/jenkinsfile-runner ● https://hub.docker.com/r/jenkins4eval/jenkinsfile-runner Custom JFR packages ● https://jenkins.io/blog/2018/10/16/custom-war-packager/ ● Ready-to-fly Docker images with plugins and configs ● Way to go in 95% of cases 20

Slide 21

Slide 21 text

© 2019 CloudBees, Inc. All Rights Reserved. Building custom Jenkinsfile Runner images Custom WAR Packager Jenkinsfile Runner Core and Plugins Configuration-as-code Packaging Ready-to-fly Docker image Base image https://jenkins.io/blog/2018/10/16/custom-war-packager/ 21

Slide 22

Slide 22 text

© 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager and Self-configuration 1. Custom WAR Packager is managed by a config YAML 2. Docker image: plugins.txt, Java flags, etc. 3. Groovy Init Scripts 4. AND: New Configuration-as-Code Plugin https://plugins.jenkins.io/configuration-as-code Allows configuring Jenkins from YAML 22 https://github.com/jenkinsci/custom-war-packager

Slide 23

Slide 23 text

© 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager Demos ● https://github.com/jenkinsci/custom-war-packager#demo ● https://github.com/jenkinsci/ci.jenkins.io-runner/ 23

Slide 24

Slide 24 text

© 2019 CloudBees, Inc. All Rights Reserved. Just building the stuff is not enough 24

Slide 25

Slide 25 text

© 2019 CloudBees, Inc. All Rights Reserved. 25

Slide 26

Slide 26 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test Framework https://github.com/jenkinsci/jenkinsfile-runner-test-framework ● Custom WAR Packager ● Docker ● shUnit2 26

Slide 27

Slide 27 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test Framework Jenkinsfile Runner packages Jenkinsfiles Pipeline libraries 27 Tests

Slide 28

Slide 28 text

© 2019 CloudBees, Inc. All Rights Reserved. Examples Jenkins X, GitHub Actions, CloudBees CodeShip 28

Slide 29

Slide 29 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner. Usage examples Jenkins X, Serverless mode (before Jenkins X 2.0 with Tekton) ● https://github.com/jenkins-x/jenkins-x-serverless ci.jenkins.io-runner ● https://github.com/jenkinsci/ci.jenkins.io-runner “FaaS” packaging ● GitHub Actions: https://github.com/jonico/jenkinsfile-runner-github-actions ● Codeship: https://github.com/oleg-nenashev/codeship-jenkinsfile ● AWS Lambda: https://github.com/carlossg/jenkinsfile-runner-lambda ● Project Fn: https://github.com/carlossg/jenkinsfile-runner-fn 29

Slide 30

Slide 30 text

© 2019 CloudBees, Inc. All Rights Reserved. 30 https://jenkins-x.io

Slide 31

Slide 31 text

© 2019 CloudBees, Inc. All Rights Reserved. 31

Slide 32

Slide 32 text

© 2019 CloudBees, Inc. All Rights Reserved. Details about Jenkins X Paolo Carta, Matteo Baiguini, Continuous Deployment of Polyglot Microservices with Istio ● https://devopsdays.org/events/2019-zurich/program/workshop-with- matteo-baiguini-and-paolo-carta/ Cosmin Cojocar, Jenkins X contributor ● CloudBees Booth on Wednesday ● Both Jenkinsfile runner and Tekton flows 32

Slide 33

Slide 33 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins X. Modes and engines 33 Static Masters Serverless build engines Jenkinsfile Runner Tekton (default in 2.0) Classic Jenkins

Slide 34

Slide 34 text

© 2018 CloudBees, Inc. All Rights Reserved. Jenkins X. Serverless Mode (before April 2019) 34 https://medium.com/@jdrawlings/ser verless-jenkins-with-jenkins-x-9134cb fe6870 Serverless Jenkins with Jenkins X

Slide 35

Slide 35 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins X Serverless Build flow Single-shot masters build flow 35

Slide 36

Slide 36 text

© 2019 CloudBees, Inc. All Rights Reserved. https://medium.com/@jdrawlings/serverless-jenki ns-with-jenkins-x-9134cbfe6870 Jenkinsfile Runner is Here 36 Jenkinsfile Runner

Slide 37

Slide 37 text

© 2018 CloudBees, Inc. All Rights Reserved. CommentOps - communication with users 37

Slide 38

Slide 38 text

© 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions … and Jenkinsfile Runner 38

Slide 39

Slide 39 text

© 2019 CloudBees, Inc. All Rights Reserved. https://github.com/features/actions 39

Slide 40

Slide 40 text

© 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions! Not designed to be CI, useful for short tasks/workflows https://github.com/jonico/jenkinsfile-runner-github-actions 40

Slide 41

Slide 41 text

© 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions is in Limited Beta … and I am still waiting for my limited Beta access to be approved 41

Slide 42

Slide 42 text

© 2019 CloudBees, Inc. All Rights Reserved. And CodeShip? 42

Slide 43

Slide 43 text

© 2019 CloudBees, Inc. All Rights Reserved. CloudBees CodeShip ● Jenkins has no good SaaS ● CodeShip Pro - CI SaaS with native Docker support ● Free tiers for open-source projects ● What if we run Jenkins there? 43 https://codeship.com/features/pro

Slide 44

Slide 44 text

© 2019 CloudBees, Inc. All Rights Reserved. Demo 1 44 codeship-steps.yml codeship-services.yml Jenkinsfile https://github.com/oleg-nenashev/codeship-jenkinsfile

Slide 45

Slide 45 text

© 2019 CloudBees, Inc. All Rights Reserved. 45

Slide 46

Slide 46 text

© 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building a Jenkins plugin in CodeShip https://github.com/oleg-nenashev/mailer-plugin/tree/codeship-demo ● Using ci.jenkins.io-runner ○ https://github.com/jenkinsci/ci.jenkins.io-runner ● Using the standard Jenkins Pipeline library ○ https://github.com/jenkins-infra/pipeline-library ● Using standard Jenkinsfile ○ buildPlugin() 46

Slide 47

Slide 47 text

© 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building a Jenkins plugin in CodeShip codeship-services.yml codeship-steps.yml 47 Jenkinsfile

Slide 48

Slide 48 text

© 2019 CloudBees, Inc. All Rights Reserved. 48

Slide 49

Slide 49 text

© 2019 CloudBees, Inc. All Rights Reserved. CodeShip. Build history 49

Slide 50

Slide 50 text

© 2019 CloudBees, Inc. All Rights Reserved. So... 50

Slide 51

Slide 51 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner runs everywhere 51 docker run --rm \ -v $(shell pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner ● Docker images ● CLI Tool - on Java 8 or Java 11

Slide 52

Slide 52 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner is not a silver bullet! 52

Slide 53

Slide 53 text

© 2019 CloudBees, Inc. All Rights Reserved. Limitations Jenkinsfile Runner is not fully compatible ● No Web UI, no way to extract Jenkins reports ● No native support of external storage ● No build throttling, cross-master communication, etc. Single container, single agent ● Docker and Kubernetes plugins cannot be easily used to produce sidecar containers ● https://github.com/jenkinsci/remoting/pull/286 53

Slide 54

Slide 54 text

© 2019 CloudBees, Inc. All Rights Reserved. Future work Waiting for user feedback Integrating patches ● https://github.com/jenkinsci/jenkinsfile-runner/blob/master/CONTRIBUTING.md (?) Jenkinsfile Runner 1.0 (?) Sidecar containers Experiments with GraalVM and Quarkus 54

Slide 55

Slide 55 text

© 2019 CloudBees, Inc. All Rights Reserved. Takeaways ● serverless Jenkins is real ● you can run your Pipelines everywhere ● there are tools for that! ● use Jenkins X if applicable (K8s/GitOps) 55

Slide 56

Slide 56 text

© 2019 CloudBees, Inc. All Rights Reserved. Demos and Questions CloudBees Booth Demos... ● Jenkins ● Jenkinsfile Runner ● Jenkins X - Wed ● CodeShip & Co ● Config-as-Code ● Pipeline devel 56 We are here

Slide 57

Slide 57 text

© 2019 CloudBees, Inc. All Rights Reserved. Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com 57

Slide 58

Slide 58 text

© 2019 CloudBees, Inc. All Rights Reserved. > whoami -cloudbees ● CloudBees Jenkins Distribution ● CloudBees Jenkins Support ● Community (Java 11 support, Jenkinsfile Runner, Security) https://www.cloudbees.com 58

Slide 59

Slide 59 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins as you may knew it… in 2012 Complex configuration Mega-masters, limited scalability Painful upgrades Masters are always running . . . 59