Slide 1

Slide 1 text

© 2019 CloudBees, Inc. All Rights Reserved. Under the hood of serverless Jenkins Jenkinsfile Runner Oleg Nenashev (@oleg_nenashev) CloudBees, Inc. St. Petersburg, Oct 30, 2019

Slide 2

Slide 2 text

© 2019 CloudBees, Inc. All Rights Reserved. TL;DR “You can run Jenkins Pipelines as single-shot container in any environment” 2 This talk: How to do it?

Slide 3

Slide 3 text

© 2019 CloudBees, Inc. All Rights Reserved. > whoami 3 @oleg_nenashev oleg-nenashev • Based in Neuchatel, Switzerland • Principal SW Engineer, CloudBees • Jenkins core maintainer

Slide 4

Slide 4 text

© 2019 CloudBees, Inc. All Rights Reserved. > whoami -jenkins 4 @oleg_nenashev oleg-nenashev

Slide 5

Slide 5 text

© 2019 CloudBees, Inc. All Rights Reserved. ● CloudBees Jenkins Distribution ● CloudBees Jenkins Support ● Community, e.g. Java 11 support, Jenkinsfile Runner, JCasC > whoami -cloudbees 5 https://www.cloudbees.com

Slide 6

Slide 6 text

© 2019 CloudBees, Inc. All Rights Reserved. About you 6 Disclaimer: Opinions are mine; typos are mine

Slide 7

Slide 7 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Automation server/framework ● Open-source ● More than 1700 plugins ● Big community ● Commercial support is available Who is Mr. Jenkins? 7

Slide 8

Slide 8 text

© 2019 CloudBees, Inc. All Rights Reserved. Hudson/Jenkins as you knew it… in 2011 Batteries are not included Complex configuration Mega-masters, limited scalability Painful upgrades Maintenance is required 8

Slide 9

Slide 9 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Pipeline-as-Code ● Configuration-as-Code ● New plugins and integrations ● Modern packaging ● Jenkins X Jenkins in 2019 9

Slide 10

Slide 10 text

© 2019 CloudBees, Inc. All Rights Reserved. 10 Permanent agents On-demand agents from cloud providers Jenkins Master Web UI

Slide 11

Slide 11 text

© 2019 CloudBees, Inc. All Rights Reserved. Master as a SPoF Macroservice 11 Permanent agents On-demand agents from cloud providers Web UI Jenkins Master

Slide 12

Slide 12 text

© 2019 CloudBees, Inc. All Rights Reserved. Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ 12

Slide 13

Slide 13 text

© 2019 CloudBees, Inc. All Rights Reserved. Shifting Gears: Making Changes 13

Slide 14

Slide 14 text

© 2019 CloudBees, Inc. All Rights Reserved. “General purpose CI/CD engine that runs on Kubernetes and embraces fundamentally different architecture & extensibility mechanisms” Cloud Native Jenkins 14 https://flic.kr/p/75ncBF Kohsuke Kawaguchi, 2018

Slide 15

Slide 15 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Best service for each need ● Pay per use ● “Infinite” scaling ● Easy to use ● Easy to maintain ● Fast to develop Cloud Native Jenkins 15 https://flic.kr/p/75ncBF

Slide 16

Slide 16 text

© 2019 CloudBees, Inc. All Rights Reserved. Founded in July 2018 https://jenkins.io/sigs/cloud-native Goal: Improve Jenkins to run on cloud environments as a "Cloud Native" application Cloud Native Special Interest Group 16 ?

Slide 17

Slide 17 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins X? 17

Slide 18

Slide 18 text

© 2019 CloudBees, Inc. All Rights Reserved. 18 https://jenkins-x.io

Slide 19

Slide 19 text

© 2019 CloudBees, Inc. All Rights Reserved. 19

Slide 20

Slide 20 text

© 2019 CloudBees, Inc. All Rights Reserved. 20 ● AWESOME (c) Jenkins X 2.x is…

Slide 21

Slide 21 text

© 2019 CloudBees, Inc. All Rights Reserved. 21 ● AWESOME (c) ● Cloud native ● K8s native ● Scalable ● Designed for CD ● Moving fast Jenkins X 2.x is…

Slide 22

Slide 22 text

© 2019 CloudBees, Inc. All Rights Reserved. 22 ● AWESOME (c) ● Cloud native ● K8s native ● Scalable ● Designed for CD ● Moving fast ● Kubernetes-only ● GitOps-only ● Limited SCMs support ● Limited CI capabilities ● No auto-migration from Jenkins Jenkins X 2.x is… Limitations

Slide 23

Slide 23 text

© 2019 CloudBees, Inc. All Rights Reserved. 23 ● AWESOME (c) ● Cloud native ● K8s native ● Scalable ● Designed for CD ● Moving fast ● Kubernetes-only ● GitOps-only ● Limited SCMs support ● Limited CI capabilities Jenkins X 2.x is… Limitations Try out Jenkins X if it fits your environment OR if you plan to go all-in Kubernetes. Jenkins is still around for other cases

Slide 24

Slide 24 text

© 2019 CloudBees, Inc. All Rights Reserved. Going Back? 24

Slide 25

Slide 25 text

© 2019 CloudBees, Inc. All Rights Reserved. Master as a SPoF Macroservice 25 Permanent agents On-demand agents from cloud providers Web UI Jenkins Master

Slide 26

Slide 26 text

© 2019 CloudBees, Inc. All Rights Reserved. (┛ಠ_ಠ)┛彡┻━┻ What if Jenkins master was “on-demand”?

Slide 27

Slide 27 text

© 2019 CloudBees, Inc. All Rights Reserved. 27

Slide 28

Slide 28 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner https://github.com/jenkinsci/jenkinsfile-runner Started by Kohsuke in Mar 2018 30 contributors Status: Beta (1.0-beta-11) 28

Slide 29

Slide 29 text

© 2019 CloudBees, Inc. All Rights Reserved. JFR TL;DR ● Runs Pipeline and shuts down ● Fat JAR or a thin JAR pack ● Docker image ● Runs everywhere ● Startup in milliseconds/seconds 29 https://github.com/jenkinsci/jenkinsfile-runner

Slide 30

Slide 30 text

© 2019 CloudBees, Inc. All Rights Reserved. 30 STDOUT/STDERR Workspace External storage Workspace Jenkinsfile Configuration (WAR, plugins, settings)

Slide 31

Slide 31 text

© 2019 CloudBees, Inc. All Rights Reserved. Usage: jenkinsfile-runner [options] [params] Options: -f (--file) FILE : Path to Jenkinsfile (or directory containing a Jenkinsfile) to run -w (--jenkins-war) FILE : Path to the exploded jenkins.war directory -p (--plugins) FILE : plugins.txt file or a /plugins installation directory Advanced: --runHome FILE : Path to the empty Jenkins Home directory to use for this run --runWorkspace FILE : Path to the workspace to be used within node{} context. -a (--arg) : Parameters to be passed to workflow job -b (--build-number) N : Build number of the run -c (--cause) VAL : Cause of the run -n (--job-name) VAL : Name of the job the run belongs to -ns (--no-sandbox) : Disable the sandbox environment 31

Slide 32

Slide 32 text

© 2019 CloudBees, Inc. All Rights Reserved. Usage in Docker 32 docker run --rm \ -v $(pwd)/Jenkinsfile:/workspace/Jenkinsfile \ jenkins4eval/jenkinsfile-runner* * or another image….

Slide 33

Slide 33 text

© 2019 CloudBees, Inc. All Rights Reserved. Demo. Our Jenkinsfile 33

Slide 34

Slide 34 text

© 2019 CloudBees, Inc. All Rights Reserved. 34

Slide 35

Slide 35 text

© 2019 CloudBees, Inc. All Rights Reserved. 35

Slide 36

Slide 36 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Not a CI/CD solution on its own Limitations 36 Triggers Webhook receivers Log services Result Storage Web UI

Slide 37

Slide 37 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Not a CI/CD solution on its own ● Jenkinsfile Runner is not 100% Jenkins ● No Web UI ● No build throttling, cross-master communication, etc. ● Designed for a single-container ● Slow agent connection ● No out-of-the-box sidecar containers support Limitations 37

Slide 38

Slide 38 text

© 2019 CloudBees, Inc. All Rights Reserved. But 38 Triggers Webhook receivers Log services Result Storage Web UI

Slide 39

Slide 39 text

© 2019 CloudBees, Inc. All Rights Reserved. But 39 Triggers Webhook receivers Log services Result Storage Web UI

Slide 40

Slide 40 text

© 2019 CloudBees, Inc. All Rights Reserved. “Packaging”? 40

Slide 41

Slide 41 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors 41 Vanilla: Docker/Binary in the official repo ● Basic plugins: Pipeline, JCasC, etc. ● GitHub: jenkinsci/jenkinsfile-runner ● DockerHub: jenkins4eval/jenkinsfile-runner

Slide 42

Slide 42 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner flavors 42 Custom JFR packages ● Way to go in 95% of cases ● Ready-to-fly Docker images with plugins and configs ● Extending the Vanilla Docker image ● https://jenkins.io/blog/2018/10/16/custom-war-packager/ Vanilla: Docker/Binary in the official repo ● Basic plugins: Pipeline, JCasC, etc. ● GitHub: jenkinsci/jenkinsfile-runner ● DockerHub: jenkins4eval/jenkinsfile-runner

Slide 43

Slide 43 text

© 2019 CloudBees, Inc. All Rights Reserved. Building custom Jenkinsfile Runner images 43 https://jenkins.io/blog/2018/10/16/custom-war-packager/ Custom WAR Packager Jenkinsfile Runner Core and Plugins Configuration-as-code Packaging Ready-to-fly Docker image Base image Y A M L

Slide 44

Slide 44 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner. Usage examples Jenkins X 1.x, serverless mode ● jenkins-x/jenkins-x-serverless ci.jenkins.io-runner ● jenkinsci/ci.jenkins.io-runner “FaaS” packaging ● GitHub Actions: jonico/jenkinsfile-runner-github-actions ● Codeship: oleg-nenashev/codeship-jenkinsfile ● AWS Lambda: carlossg/jenkinsfile-runner-lambda ● Project Fn: carlossg/jenkinsfile-runner-fn 44

Slide 45

Slide 45 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins X 1.0 Serverless Engine example 45

Slide 46

Slide 46 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins X. Serverless Mode (before April 2019) 46 https://medium.com/@jdrawlings/ser verless-jenkins-with-jenkins-x- 9134cbfe6870 Serverless Jenkins with Jenkins X

Slide 47

Slide 47 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins X 2.x. Modes and engines 47 Static Masters Serverless build engines Jenkinsfile Runner Tekton (default in 2.0) Classic Jenkins @Deprecated

Slide 48

Slide 48 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins X 1.x. Serverless Mode packaging 48 Single-shot masters build flow

Slide 49

Slide 49 text

49

Slide 50

Slide 50 text

50 Jenkinsfile Runner Jenkinsfile Runner is Here

Slide 51

Slide 51 text

© 2019 CloudBees, Inc. All Rights Reserved. CommentOps - communication with users 51

Slide 52

Slide 52 text

© 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions … and Jenkinsfile Runner 52

Slide 53

Slide 53 text

© 2019 CloudBees, Inc. All Rights Reserved. https://github.com/features/actions 53

Slide 54

Slide 54 text

© 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions! 54 https://github.com/jonico/jenkinsfile-runner-github-actions

Slide 55

Slide 55 text

© 2019 CloudBees, Inc. All Rights Reserved. Sample definition 55

Slide 56

Slide 56 text

© 2019 CloudBees, Inc. All Rights Reserved. 56

Slide 57

Slide 57 text

© 2019 CloudBees, Inc. All Rights Reserved. 57

Slide 58

Slide 58 text

© 2019 CloudBees, Inc. All Rights Reserved. GitHub Actions is in Limited Beta GA - November 13! 58

Slide 59

Slide 59 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfiles on CodeShip?

Slide 60

Slide 60 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner on CloudBees CodeShip ● Jenkins has no good SaaS (now) ● CodeShip Pro - CI SaaS with native Docker support ● Free tiers for open-source projects ● What if we run Jenkins there? 60 https://codeship.com/features/pro

Slide 61

Slide 61 text

© 2019 CloudBees, Inc. All Rights Reserved. Demo 1 61 codeship-steps.yml codeship-services.yml Jenkinsfile https://github.com/oleg-nenashev/codeship-jenkinsfile

Slide 62

Slide 62 text

© 2019 CloudBees, Inc. All Rights Reserved. 62

Slide 63

Slide 63 text

© 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building a Jenkins plugin in CodeShip https://github.com/oleg-nenashev/mailer-plugin/tree/codeship-demo ● Using ci.jenkins.io-runner ○ https://github.com/jenkinsci/ci.jenkins.io-runner ● Using the standard Jenkins Pipeline library ○ https://github.com/jenkins-infra/pipeline-library ● Using standard Jenkinsfile ○ buildPlugin() 63

Slide 64

Slide 64 text

© 2019 CloudBees, Inc. All Rights Reserved. Example 2. Building a Jenkins plugin in CodeShip 64 codeship-services.yml codeship-steps.yml Jenkinsfile

Slide 65

Slide 65 text

© 2019 CloudBees, Inc. All Rights Reserved. 65

Slide 66

Slide 66 text

© 2019 CloudBees, Inc. All Rights Reserved. CodeShip. Build history 66

Slide 67

Slide 67 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Jenkins X ● GitHub Actions ● CloudBees CodeShip We talked about… 67

Slide 68

Slide 68 text

© 2019 CloudBees, Inc. All Rights Reserved. Last but not least? 68

Slide 69

Slide 69 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkins 69

Slide 70

Slide 70 text

© 2019 CloudBees, Inc. All Rights Reserved. Running JFR from Jenkins Well, “docker run”… 70

Slide 71

Slide 71 text

© 2019 CloudBees, Inc. All Rights Reserved. 71

Slide 72

Slide 72 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test Framework https://github.com/jenkinsci/jenkinsfile-runner-test-framework ● Custom WAR Packager ● Docker ● shUnit2 72

Slide 73

Slide 73 text

© 2019 CloudBees, Inc. All Rights Reserved. If Pipeline Unit is not enough… https://github.com/jenkinsci/JenkinsPipelineUnit Works for Mock Unit tests JFR Test framework for integration tests 73

Slide 74

Slide 74 text

© 2019 CloudBees, Inc. All Rights Reserved. Jenkinsfile Runner Test Framework Jenkinsfile Runner packages Jenkinsfiles Pipeline libraries 74 Tests

Slide 75

Slide 75 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Waiting for user feedback/patches ● TBD - Jenkinsfile Runner 1.0 ● Packaging for common cases ● Lightweight sidecar containers ● Jenkinsfile Runner Jenkins Plugin? ● Experiments with GraalVM and Quarkus Future work 75

Slide 76

Slide 76 text

© 2019 CloudBees, Inc. All Rights Reserved. 76 !=

Slide 77

Slide 77 text

© 2019 CloudBees, Inc. All Rights Reserved. ● Jenkins-as-a-function is real ● You can run your Pipelines everywhere ● There are tools for that ● Use Jenkins X if applicable (K8s/GitOps) ● Consider Jenkinsfile Runner if not Takeaways 77

Slide 78

Slide 78 text

© 2019 CloudBees, Inc. All Rights Reserved. ● jenkinsci/jenkinsfile-runner ● CONTRIBUTING.md in the repository ● 1 day of Hacktoberfest left ;) ○ https://jenkins.io/events/hacktoberfest/ It is a great time to contribute! 78

Slide 79

Slide 79 text

© 2019 CloudBees, Inc. All Rights Reserved. Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS?