GBG JAM - Managing Jenkins with Groovy Hooks and JCasC

Slide 1

Slide 1 text

Managing Jenkins with Groovy Hooks And JCasC plugin Oleg Nenashev CloudBees, Inc. Gothenburg Jenkins Meetup May 28, 2018

Slide 2

Slide 2 text

2 @oleg_nenashev, #codeconf > whoami librecores.org Meetups SPbSPU @oleg_nenashev oleg-nenashev

Slide 3

Slide 3 text

3 @oleg_nenashev, #codeconf > whoami -jenkins Hudson/Jenkins user since 2008 Jenkins contributor since 2012 Now: • Maintainer: Core, Remoting, plugins • Security Team member • Meetup organizer • Google Summer of Code org admin @oleg_nenashev oleg-nenashev

Slide 4

Slide 4 text

4 @oleg_nenashev, #codeconf About you

Slide 5

Slide 5 text

5 @oleg_nenashev, #codeconf • Scripting languages are fine • YAML/JSON/XML are fine as well • You can use them together This talk is NOT about holywars

Slide 6

Slide 6 text

6 @oleg_nenashev, #codeconf üGROOVY HOOKS üGROOVY HOOKS & CASC PLUGIN üTIPS & TRICKS üSOME LIVE DEMOS Outline Slides:

Slide 7

Slide 7 text

7 @oleg_nenashev, #codeconf Configuration as Code in Jenkins Jobs System configuration

Slide 8

Slide 8 text

8 @oleg_nenashev, #codeconf Configuration as Code in Jenkins Jobs System configuration

Slide 9

Slide 9 text

9 @oleg_nenashev, #codeconf “Infrastructure as Code” in Jenkins API Clients Jenkins CLI and REST API python- jenkins jenkins-client (java) Configuration Management Ansible, Chef, … Docker, Docker Compose ... Jenkins- internal solutions Groovy Boot Hooks Scriptler Plugin Configuration -as-Code Plugin

Slide 10

Slide 10 text

10 @oleg_nenashev, #codeconf State in 2016 – Stockholm JAM • No default solution recommended by the Jenkins Project • No active work on this front • System Config DSL, was abandoned • Configuration Management – outside Jenkins Project • Time lag in Features • Security fun • Compatibility (e.g. Jenkins 2)

Slide 11

Slide 11 text

11 @oleg_nenashev, #codeconf Configuration-as-Code Plugin http://bit.ly/casc-plugin-slides (older version) • Presented by Ewelina • The plugin is in alpha • Final version may differ

Slide 12

Slide 12 text

12 @oleg_nenashev, #codeconf Limitations (temporary?). Features • Not all plugins are supported • No Folders support – cannot set project structure • No debugging support • Export from UI is not working well

Slide 13

Slide 13 text

13 @oleg_nenashev, #codeconf Limitations (temporary?). Architecture • Forward/backward compatibility is not guaranteed • YAML - No dynamic logic / scripting • Startup Chicken&Egg for plugins • Race conditions for jobs • Handling of removed sections

Slide 14

Slide 14 text

14 @oleg_nenashev, #codeconf Limitations. Forward/backward compatibility • Config many break… • … even if a core/plugin change in a compatible way • Example: Work directory support in 2.73.x • New fields were added to the configuration in the core • Class @DataBoundConstructor interface changed, but old one is in place • workDir settings must be always set in CasC now • Ticket: JCasC Issue #167

Slide 15

Slide 15 text

15 @oleg_nenashev, #codeconf Limitations. Jobs loading JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED CasC Initializer PLUGINS_STARTED . . . • CasC happens in parallel with Jobs loading! • CasC may happen in parallel with initializers • Race conditions if job loading depends on plugin settings • Extra milestones are needed in the core

Slide 16

Slide 16 text

16 @oleg_nenashev, #codeconf

Slide 17

Slide 17 text

17 @oleg_nenashev, #codeconf Groovy!

Slide 18

Slide 18 text

18 @oleg_nenashev, #codeconf

Slide 19

Slide 19 text

19 @oleg_nenashev, #codeconf Groovy Hooks [1/2] • Standard Groovy engine • Triggered by Jenkins • Direct access to the runtime • Extensible hook engine • init, boot-failure – in the core https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script

Slide 20

Slide 20 text

20 @oleg_nenashev, #codeconf Groovy Hooks [2/2] • Script locations: •WEB-INF/HOOK.groovy in jenkins.war •WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war •$JENKINS_HOME/HOOK.groovy •$JENKINS_HOME/HOOK.groovy.d/*.groovy • Execution in alphabetical order https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script

Slide 21

Slide 21 text

21 @oleg_nenashev, #codeconf Groovy Init Hooks JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() [CasC if installed] PLUGINS_STARTED . . .

Slide 22

Slide 22 text

22 @oleg_nenashev, #codeconf Example https://github.com/oleg-nenashev/demo-jenkins-config-as-code

Slide 23

Slide 23 text

23 @oleg_nenashev, #codeconf https://github.com/oleg-nenashev/demo- jenkins-config-as-code

Slide 24

Slide 24 text

24 @oleg_nenashev, #codeconf Demo. Local Pipeline Development Env Intellij IDEA Filesystem SCM Plugin • Documentation, Syntax • Static analysis • Debug (only for hooks) Local Jenkins instance Source Code (local .git repos) • Configuration-as-Code • Same as production • Repos with libs • Jenkinsfile JENKINS_HOME (volume)

Slide 25

Slide 25 text

25 @oleg_nenashev, #codeconf More examples • https://github.com/oleg-nenashev/demo-jenkins-config-as-code • https://github.com/Praqma/JenkinsAsCodeReference • https://github.com/librecores/librecores-ci

Slide 26

Slide 26 text

26 @oleg_nenashev, #codeconf Groovy! #ornot

Slide 27

Slide 27 text

27 @oleg_nenashev, #codeconf Groovy Hook Limitations. General • Jenkins API is scattered and complex • Javadoc is not user-friendly • Public API is not enough sometimes • No export from UI Solution: • Configuration as Code Plugin

Slide 28

Slide 28 text

28 @oleg_nenashev, #codeconf Groovy Hook Limitations. Missing Features • No class support • No shared libraries support • Error propagation: Jenkins does not stop on script error • Restart is required to apply changes Solution: • See below

Slide 29

Slide 29 text

Groovy Hooks and CasC - make them work together

Slide 30

Slide 30 text

30 @oleg_nenashev, #codeconf CasC and Groovy hooks JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() CasC PLUGINS_STARTED . . . • Groovy hooks run after CasC • Hooks can fine-tune the configuration: • Missing integrations • Dynamic scripting • Job configurations and migrations

Slide 31

Slide 31 text

31 @oleg_nenashev, #codeconf https://github.com/oleg-nenashev/demo- jenkins-config-as-code/tree/casc-plugin

Slide 32

Slide 32 text

Groovy Hooks – Tips & Tricks • Class support & Error handling • Development and debugging • Static configurations

Slide 33

Slide 33 text

33 @oleg_nenashev, #codeconf Class support & Error handling • They say: •Hard to share/reuse, no class support •No error propagation

Slide 34

Slide 34 text

34 @oleg_nenashev, #codeconf

Slide 35

Slide 35 text

35 @oleg_nenashev, #codeconf Let’s run Groovy engine from… a Groovy script Groovy Bootstrap

Slide 36

Slide 36 text

36 @oleg_nenashev, #codeconf https://github.com/oleg-nenashev/jenkins-config-as-code-demo/ blob/master/init_scripts/src/main/groovy/GroovyBootstrap.groovy More class examples: https://github.com/librecores/librecores-ci

Slide 37

Slide 37 text

37 @oleg_nenashev, #codeconf What can be done? Almost anything… • Class support (in demo) • Error propagation (in demo) • Library loading… ▸@Grab or other library management ▸Library bundling into WAR / Docker image • Partial CasC plugin invocation • Apply config on API call • …

Slide 38

Slide 38 text

38 @oleg_nenashev, #codeconf Groovy Hook Development?

Slide 39

Slide 39 text

39 @oleg_nenashev, #codeconf 1. Use your favorite IDE with Groovy support 2. Create Maven project 3. Use virtual Jenkins “plugin” with deps • Static analysis • Test automation Groovy Hook Development?

Slide 40

Slide 40 text

40 @oleg_nenashev, #codeconf Groovy Hook Debugging?

Slide 41

Slide 41 text

41 @oleg_nenashev, #codeconf 1. Start Jenkins with Debug flags 2. Just attach your IDE Groovy Hook Debugging? https://github.com/oleg-nenashev/demo- jenkins-config-as-code/blob/master/jenkins2.sh

Slide 42

Slide 42 text

42 @oleg_nenashev, #codeconf •Read-only configurations •No Jenkins/Administer •But… no read-only admin UI L Static configuration? It’s possible!

Slide 43

Slide 43 text

43 @oleg_nenashev, #codeconf Static configuration in Jenkins • Base: https://github.com/jenkinsci/docker • plugins.txt Docker • Jenkins configuration • Setup of folders and jobs Groovy Init Hooks • Jenkinsfile in SCM • Organization Folder Pipeline

Slide 44

Slide 44 text

What does not “just work”?

Slide 45

Slide 45 text

45 @oleg_nenashev, #codeconf Comparison JCasC Plugin Groovy Hooks YAML – standard declarative definition Low barrier to entry Limited integrations Compatibility issues External rollback Apply changes w/o restart No debugging, YAGNI? Future: Export from UI Groovy – general-purpose OOP language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support

Slide 46

Slide 46 text

46 @oleg_nenashev, #codeconf What is missing in BOTH engines? • Rollback support •Job/Build entries may be migrated with any update •Update is possible, but there may be breaking changes •Rollback – only via backups

Slide 47

Slide 47 text

47 @oleg_nenashev, #codeconf What is missing in BOTH engines? • Rollback support •Job/Build entries may be migrated with any update •Update is possible, but there may be breaking changes •Rollback – only via backups Solution: • External config management logic (change config, test initialization, rollback) • Backup plugin with CLI / REST API for Restore (e.g. Periodic Backup Plugin)

Slide 48

Slide 48 text

48 @oleg_nenashev, #codeconf What is missing in BOTH engines? • Rollback support • Plugin installation Chicken & Egg •New plugin installation is possible •Other cases: Reload may be needed ▸upgrade, downgrade, uninstall ▸CasC plugin self-upgrade? JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() CasC PLUGINS_STARTED . . .

Slide 49

Slide 49 text

49 @oleg_nenashev, #codeconf What is missing in BOTH engines? • Rollback support • Plugin installation Chicken & Egg •New plugin installation is possible •Other cases: Reload may be needed JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() CasC PLUGINS_STARTED . . . Solutions: A. CasC – work in progress (by Nicolas de Loof) B. External pre-startup tools • plugins.txt in stock Docker images • Custom WAR Packager (WiP by Oleg)

Slide 50

Slide 50 text

50 @oleg_nenashev, #codeconf What is missing in BOTH engines? • Rollback support • Plugin installation Chicken & Egg • UX: Lack of support in the Jenkins core •Admin UI is must-have for diagnostics •But: ▸No read-only admin UI ▸No read-only job configurations for admins ▸No read-only support in API

Slide 51

Slide 51 text

51 @oleg_nenashev, #codeconf Read-only UI. Partial solution • “sudo” mode in Matrix Auth or Role Strategy plugin •Task for Role Strategy plugin (JENKINS-51567) • Admin permissions are granted only when required • Job with System Groovy script to grant/revoke permissions • OR: REST API for Admin

Slide 52

Slide 52 text

So…

Slide 53

Slide 53 text

53 @oleg_nenashev, #codeconf Takeaways • Two engines, no silver bullet JCasC Plugin Groovy Hooks YAML – standard declarative definition Low barrier to entry Limited integrations Compatibility issues External rollback Apply changes w/o restart No debugging, YAGNI? Future: Export from UI Groovy – general-purpose OOP language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support

Slide 54

Slide 54 text

54 @oleg_nenashev, #codeconf Takeaways • Select an engine depending on your needs and expertise • New to Jenkins? Use JCasC

Slide 55

Slide 55 text

55 @oleg_nenashev, #codeconf Takeaways • Select an engine depending on your needs and expertise • New to Jenkins? Use JCasC • Combine engines if needed

Slide 56

Slide 56 text

56 @oleg_nenashev, #codeconf Takeaways • It’s a great time to contribute ;) •https://jenkins.io/participate

Slide 57

Slide 57 text

57 @oleg_nenashev, #codeconf Links Groovy hooks: • https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script • These slides: https://speakerdeck.com/onenashev/gbg-jam-managing- jenkins-with-groovy-hooks-and-jcasc Demo and examples: • https://hub.docker.com/r/onenashev/demo-jenkins-config-as-code/ • https://github.com/oleg-nenashev/demo-jenkins-config-as-code • https://github.com/Praqma/JenkinsAsCodeReference • https://github.com/librecores/librecores-ci Configuration-as-Code Plugin • https://github.com/jenkinsci/configuration-as-code-plugin

Slide 58

Slide 58 text

58 @oleg_nenashev, #codeconf Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com

Slide 59

Slide 59 text

Software at the speed of ideas THANK YOU! www.cloudbees.com