DevOops 2019 - Under the hood of serverless Jenkins. Jenkinsfile Runner

There were significant changes in the Jenkins project over last year. It is still one of the most popular automation servers in the world, but it is not just a "server" anymore. In 2019 Jenkins is a universal automation engine, which actively evolves towards the new Cloud Native architecture and which can be used in any CI/CD flows. It can be used not only as a server, but also as a FaaS. And it is not only about Jenkins X in Kubernetes, but also about more classic usages.

In this talk Oleg will present Jenkinsfile Runner — an engine which allows running Jenkins Pipelines in a single-shot container in any environment. Running Jenkins pipelines in Travis? Easy. In FaaS like AWS Lambda? We can do it as well. Speaker will show examples for these services few other use-cases (Jenkins X, CodeShip, Tekton). And, finally, he will show how to package your own Jenkinsfile Runner images with several plug-ins and preconfiguration.

Transcript

1. © 2019 CloudBees, Inc. All Rights Reserved.
   Under the hood of serverless Jenkins
   Jenkinsfile Runner
   Oleg Nenashev (@oleg_nenashev)
   CloudBees, Inc.
   St. Petersburg, Oct 30, 2019
   

   View Slide

2. © 2019 CloudBees, Inc. All Rights Reserved.
   TL;DR
   “You can run Jenkins Pipelines as single-shot container
   in any environment”
   2
   This talk: How to do it?
   

   View Slide

3. © 2019 CloudBees, Inc. All Rights Reserved.
   > whoami
   3
   @oleg_nenashev
   oleg-nenashev
   • Based in Neuchatel, Switzerland
   • Principal SW Engineer, CloudBees
   • Jenkins core maintainer
   

   View Slide

4. © 2019 CloudBees, Inc. All Rights Reserved.
   > whoami -jenkins
   4
   @oleg_nenashev
   oleg-nenashev
   

   View Slide

5. © 2019 CloudBees, Inc. All Rights Reserved.
   ● CloudBees Jenkins Distribution
   ● CloudBees Jenkins Support
   ● Community, e.g. Java 11 support, Jenkinsfile Runner, JCasC
   > whoami -cloudbees
   5
   https://www.cloudbees.com
   

   View Slide

6. © 2019 CloudBees, Inc. All Rights Reserved.
   About you
   6
   Disclaimer: Opinions are mine;
   typos are mine
   

   View Slide

7. © 2019 CloudBees, Inc. All Rights Reserved.
   ● Automation server/framework
   ● Open-source
   ● More than 1700 plugins
   ● Big community
   ● Commercial support is available
   Who is Mr. Jenkins?
   7
   

   View Slide

8. © 2019 CloudBees, Inc. All Rights Reserved.
   Hudson/Jenkins as you knew it… in 2011
   Batteries are not included
   Complex configuration
   Mega-masters, limited scalability
   Painful upgrades
   Maintenance is required
   8
   

   View Slide

9. © 2019 CloudBees, Inc. All Rights Reserved.
   ● Pipeline-as-Code
   ● Configuration-as-Code
   ● New plugins and integrations
   ● Modern packaging
   ● Jenkins X
   Jenkins in 2019
   9
   

   View Slide

10. © 2019 CloudBees, Inc. All Rights Reserved.
    10
    Permanent agents
    On-demand agents
    from cloud providers
    Jenkins Master
    Web UI
    

    View Slide

11. © 2019 CloudBees, Inc. All Rights Reserved.
    Master as a SPoF Macroservice
    11
    Permanent agents
    On-demand agents
    from cloud providers
    Web UI
    Jenkins
    Master
    

    View Slide

12. © 2019 CloudBees, Inc. All Rights Reserved.
    Aug 31, 2018 – Kohsuke Kawaguchi,
    “Jenkins: Shifting Gears”
    https://jenkins.io/blog/2018/08/31/shifting-gears/
    12
    

    View Slide

13. © 2019 CloudBees, Inc. All Rights Reserved.
    Shifting Gears: Making Changes
    13
    

    View Slide

14. © 2019 CloudBees, Inc. All Rights Reserved.
    “General purpose CI/CD
    engine that runs on
    Kubernetes and embraces
    fundamentally different
    architecture & extensibility
    mechanisms”
    Cloud Native Jenkins
    14
    https://flic.kr/p/75ncBF
    Kohsuke Kawaguchi, 2018
    

    View Slide

15. © 2019 CloudBees, Inc. All Rights Reserved.
    ● Best service for each need
    ● Pay per use
    ● “Infinite” scaling
    ● Easy to use
    ● Easy to maintain
    ● Fast to develop
    Cloud Native Jenkins
    15
    https://flic.kr/p/75ncBF
    

    View Slide

16. © 2019 CloudBees, Inc. All Rights Reserved.
    Founded in July 2018
    https://jenkins.io/sigs/cloud-native
    Goal: Improve Jenkins to run on cloud
    environments as a "Cloud Native"
    application
    Cloud Native Special Interest Group
    16
    ?
    

    View Slide

17. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkins X?
    17
    

    View Slide

18. © 2019 CloudBees, Inc. All Rights Reserved.
    18
    https://jenkins-x.io
    

    View Slide

19. © 2019 CloudBees, Inc. All Rights Reserved.
    19
    

    View Slide

20. © 2019 CloudBees, Inc. All Rights Reserved.
    20
    ● AWESOME (c)
    Jenkins X 2.x is…
    

    View Slide

21. © 2019 CloudBees, Inc. All Rights Reserved.
    21
    ● AWESOME (c)
    ● Cloud native
    ● K8s native
    ● Scalable
    ● Designed for CD
    ● Moving fast
    Jenkins X 2.x is…
    

    View Slide

22. © 2019 CloudBees, Inc. All Rights Reserved.
    22
    ● AWESOME (c)
    ● Cloud native
    ● K8s native
    ● Scalable
    ● Designed for CD
    ● Moving fast
    ● Kubernetes-only
    ● GitOps-only
    ● Limited SCMs support
    ● Limited CI capabilities
    ● No auto-migration from Jenkins
    Jenkins X 2.x is… Limitations
    

    View Slide

23. © 2019 CloudBees, Inc. All Rights Reserved.
    23
    ● AWESOME (c)
    ● Cloud native
    ● K8s native
    ● Scalable
    ● Designed for CD
    ● Moving fast
    ● Kubernetes-only
    ● GitOps-only
    ● Limited SCMs support
    ● Limited CI capabilities
    Jenkins X 2.x is… Limitations
    Try out Jenkins X if it fits your environment
    OR if you plan to go all-in Kubernetes.
    Jenkins is still around for other cases
    

    View Slide

24. © 2019 CloudBees, Inc. All Rights Reserved.
    Going Back?
    24
    

    View Slide

25. © 2019 CloudBees, Inc. All Rights Reserved.
    Master as a SPoF Macroservice
    25
    Permanent agents
    On-demand agents
    from cloud providers
    Web UI
    Jenkins
    Master
    

    View Slide

26. © 2019 CloudBees, Inc. All Rights Reserved.
    (┛ಠ_ಠ)┛彡┻━┻
    What if Jenkins master was
    “on-demand”?
    

    View Slide

27. © 2019 CloudBees, Inc. All Rights Reserved.
    27
    

    View Slide

28. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfile Runner
    https://github.com/jenkinsci/jenkinsfile-runner
    Started by Kohsuke in Mar 2018
    30 contributors
    Status: Beta (1.0-beta-11)
    28
    

    View Slide

29. © 2019 CloudBees, Inc. All Rights Reserved.
    JFR TL;DR
    ● Runs Pipeline and shuts down
    ● Fat JAR or a thin JAR pack
    ● Docker image
    ● Runs everywhere
    ● Startup in milliseconds/seconds
    29
    https://github.com/jenkinsci/jenkinsfile-runner
    

    View Slide

30. © 2019 CloudBees, Inc. All Rights Reserved.
    30
    STDOUT/STDERR
    Workspace
    External storage
    Workspace
    Jenkinsfile
    Configuration (WAR,
    plugins, settings)
    

    View Slide

31. © 2019 CloudBees, Inc. All Rights Reserved.
    Usage: jenkinsfile-runner [options] [params]
    Options:
    -f (--file) FILE : Path to Jenkinsfile (or directory containing a Jenkinsfile) to run
    -w (--jenkins-war) FILE : Path to the exploded jenkins.war directory
    -p (--plugins) FILE : plugins.txt file or a /plugins installation directory
    Advanced:
    --runHome FILE : Path to the empty Jenkins Home directory to use for this run
    --runWorkspace FILE : Path to the workspace to be used within node{} context.
    -a (--arg) : Parameters to be passed to workflow job
    -b (--build-number) N : Build number of the run
    -c (--cause) VAL : Cause of the run
    -n (--job-name) VAL : Name of the job the run belongs to
    -ns (--no-sandbox) : Disable the sandbox environment
    31
    

    View Slide

32. © 2019 CloudBees, Inc. All Rights Reserved.
    Usage in Docker
    32
    docker run --rm \
    -v $(pwd)/Jenkinsfile:/workspace/Jenkinsfile \
    jenkins4eval/jenkinsfile-runner*
    * or another image….
    

    View Slide

33. © 2019 CloudBees, Inc. All Rights Reserved.
    Demo. Our Jenkinsfile
    33
    

    View Slide

34. © 2019 CloudBees, Inc. All Rights Reserved.
    34
    

    View Slide

35. © 2019 CloudBees, Inc. All Rights Reserved.
    35
    

    View Slide

36. © 2019 CloudBees, Inc. All Rights Reserved.
    ● Not a CI/CD solution on its own
    Limitations
    36
    Triggers
    Webhook
    receivers
    Log services
    Result Storage
    Web UI
    

    View Slide

37. © 2019 CloudBees, Inc. All Rights Reserved.
    ● Not a CI/CD solution on its own
    ● Jenkinsfile Runner is not 100% Jenkins
    ● No Web UI
    ● No build throttling, cross-master communication, etc.
    ● Designed for a single-container
    ● Slow agent connection
    ● No out-of-the-box sidecar containers support
    Limitations
    37
    

    View Slide

38. © 2019 CloudBees, Inc. All Rights Reserved.
    But
    38
    Triggers
    Webhook
    receivers Log services
    Result Storage
    Web UI
    

    View Slide

39. © 2019 CloudBees, Inc. All Rights Reserved.
    But
    39
    Triggers
    Webhook
    receivers Log services
    Result Storage
    Web UI
    

    View Slide

40. © 2019 CloudBees, Inc. All Rights Reserved.
    “Packaging”?
    40
    

    View Slide

41. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfile Runner flavors
    41
    Vanilla: Docker/Binary in the official repo
    ● Basic plugins: Pipeline, JCasC, etc.
    ● GitHub: jenkinsci/jenkinsfile-runner
    ● DockerHub: jenkins4eval/jenkinsfile-runner
    

    View Slide

42. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfile Runner flavors
    42
    Custom JFR packages
    ● Way to go in 95% of cases
    ● Ready-to-fly Docker images with plugins and configs
    ● Extending the Vanilla Docker image
    ● https://jenkins.io/blog/2018/10/16/custom-war-packager/
    Vanilla: Docker/Binary in the official repo
    ● Basic plugins: Pipeline, JCasC, etc.
    ● GitHub: jenkinsci/jenkinsfile-runner
    ● DockerHub: jenkins4eval/jenkinsfile-runner
    

    View Slide

43. © 2019 CloudBees, Inc. All Rights Reserved.
    Building custom Jenkinsfile Runner images
    43
    https://jenkins.io/blog/2018/10/16/custom-war-packager/
    Custom WAR
    Packager
    Jenkinsfile
    Runner
    Core and Plugins
    Configuration-as-code
    Packaging
    Ready-to-fly
    Docker image
    Base image
    Y
    A
    M
    L
    

    View Slide

44. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfile Runner. Usage examples
    Jenkins X 1.x, serverless mode
    ● jenkins-x/jenkins-x-serverless
    ci.jenkins.io-runner
    ● jenkinsci/ci.jenkins.io-runner
    “FaaS” packaging
    ● GitHub Actions: jonico/jenkinsfile-runner-github-actions
    ● Codeship: oleg-nenashev/codeship-jenkinsfile
    ● AWS Lambda: carlossg/jenkinsfile-runner-lambda
    ● Project Fn: carlossg/jenkinsfile-runner-fn
    44
    

    View Slide

45. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkins X 1.0
    Serverless Engine example
    45
    

    View Slide

46. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkins X. Serverless Mode (before April 2019)
    46
    https://medium.com/@jdrawlings/ser
    verless-jenkins-with-jenkins-x-
    9134cbfe6870
    Serverless Jenkins with Jenkins X
    

    View Slide

47. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkins X 2.x. Modes and engines
    47
    Static
    Masters
    Serverless
    build engines
    Jenkinsfile
    Runner
    Tekton
    (default in 2.0)
    Classic
    Jenkins
    @Deprecated
    

    View Slide

48. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkins X 1.x. Serverless Mode packaging
    48
    Single-shot
    masters build flow
    

    View Slide

49. View Slide

50. 50
    Jenkinsfile
    Runner
    Jenkinsfile
    Runner is Here
    

    View Slide

51. © 2019 CloudBees, Inc. All Rights Reserved.
    CommentOps - communication with users
    51
    

    View Slide

52. © 2019 CloudBees, Inc. All Rights Reserved.
    GitHub Actions
    … and Jenkinsfile Runner
    52
    

    View Slide

53. © 2019 CloudBees, Inc. All Rights Reserved.
    https://github.com/features/actions 53
    

    View Slide

54. © 2019 CloudBees, Inc. All Rights Reserved.
    GitHub Actions!
    54
    https://github.com/jonico/jenkinsfile-runner-github-actions
    

    View Slide

55. © 2019 CloudBees, Inc. All Rights Reserved.
    Sample definition
    55
    

    View Slide

56. © 2019 CloudBees, Inc. All Rights Reserved.
    56
    

    View Slide

57. © 2019 CloudBees, Inc. All Rights Reserved.
    57
    

    View Slide

58. © 2019 CloudBees, Inc. All Rights Reserved.
    GitHub Actions is in
    Limited Beta
    GA - November 13!
    58
    

    View Slide

59. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfiles on CodeShip?
    

    View Slide

60. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfile Runner on CloudBees CodeShip
    ●
    Jenkins has no good SaaS (now)
    ●
    CodeShip Pro - CI SaaS with native Docker support
    ●
    Free tiers for open-source projects
    ●
    What if we run Jenkins there?
    60
    https://codeship.com/features/pro
    

    View Slide

61. © 2019 CloudBees, Inc. All Rights Reserved.
    Demo 1
    61
    codeship-steps.yml
    codeship-services.yml
    Jenkinsfile
    https://github.com/oleg-nenashev/codeship-jenkinsfile
    

    View Slide

62. © 2019 CloudBees, Inc. All Rights Reserved.
    62
    

    View Slide

63. © 2019 CloudBees, Inc. All Rights Reserved.
    Example 2. Building a Jenkins plugin in CodeShip
    https://github.com/oleg-nenashev/mailer-plugin/tree/codeship-demo
    ● Using ci.jenkins.io-runner
    ○ https://github.com/jenkinsci/ci.jenkins.io-runner
    ● Using the standard Jenkins Pipeline library
    ○ https://github.com/jenkins-infra/pipeline-library
    ● Using standard Jenkinsfile
    ○ buildPlugin()
    63
    

    View Slide

64. © 2019 CloudBees, Inc. All Rights Reserved.
    Example 2. Building a Jenkins plugin in CodeShip
    64
    codeship-services.yml codeship-steps.yml
    Jenkinsfile
    

    View Slide

65. © 2019 CloudBees, Inc. All Rights Reserved.
    65
    

    View Slide

66. © 2019 CloudBees, Inc. All Rights Reserved.
    CodeShip. Build history
    66
    

    View Slide

67. © 2019 CloudBees, Inc. All Rights Reserved.
    ● Jenkins X
    ● GitHub Actions
    ● CloudBees CodeShip
    We talked about…
    67
    

    View Slide

68. © 2019 CloudBees, Inc. All Rights Reserved.
    Last but not least?
    68
    

    View Slide

69. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkins
    69
    

    View Slide

70. © 2019 CloudBees, Inc. All Rights Reserved.
    Running JFR from Jenkins
    Well, “docker run”…
    70
    

    View Slide

71. © 2019 CloudBees, Inc. All Rights Reserved.
    71
    

    View Slide

72. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfile Runner Test Framework
    https://github.com/jenkinsci/jenkinsfile-runner-test-framework
    ● Custom WAR Packager
    ● Docker
    ● shUnit2
    72
    

    View Slide

73. © 2019 CloudBees, Inc. All Rights Reserved.
    If Pipeline Unit is not enough…
    https://github.com/jenkinsci/JenkinsPipelineUnit
    Works for Mock Unit tests
    JFR Test framework for integration tests
    73
    

    View Slide

74. © 2019 CloudBees, Inc. All Rights Reserved.
    Jenkinsfile Runner Test Framework
    Jenkinsfile Runner packages
    Jenkinsfiles
    Pipeline libraries
    74
    Tests
    

    View Slide

75. © 2019 CloudBees, Inc. All Rights Reserved.
    ● Waiting for user feedback/patches
    ● TBD - Jenkinsfile Runner 1.0
    ● Packaging for common cases
    ● Lightweight sidecar containers
    ● Jenkinsfile Runner Jenkins Plugin?
    ● Experiments with GraalVM and Quarkus
    Future work
    75
    

    View Slide

76. © 2019 CloudBees, Inc. All Rights Reserved.
    76
    !=
    

    View Slide

77. © 2019 CloudBees, Inc. All Rights Reserved.
    ● Jenkins-as-a-function is real
    ● You can run your Pipelines everywhere
    ● There are tools for that
    ● Use Jenkins X if applicable (K8s/GitOps)
    ● Consider Jenkinsfile Runner if not
    Takeaways
    77
    

    View Slide

78. © 2019 CloudBees, Inc. All Rights Reserved.
    ● jenkinsci/jenkinsfile-runner
    ● CONTRIBUTING.md in the repository
    ● 1 day of Hacktoberfest left ;)
    ○ https://jenkins.io/events/hacktoberfest/
    It is a great time to contribute!
    78
    

    View Slide

79. © 2019 CloudBees, Inc. All Rights Reserved.
    Contacts:
    E-mail: onenashev@cloudbees.com
    GitHub: oleg-nenashev
    Twitter: @oleg_nenashev
    QUESTIONS?
    

    View Slide