Codifying the Build and Release Process with a Jenkins Pipeline Shared Library - Jenkins World 2017

Have you ever been asked to migrate old legacy applications/servers/code to a new datacenter? Have you ever had issues figuring out how to replicate the old environment in the new one?

This war story is about our migration from a Jenkins 1.x instance in an old legacy environment that was being decommissioned, into a brand new Jenkins 2.x instance in a new datacenter. We were given a 60-day window to complete the migration of all 150+ Freestyle jobs from the old Jenkins implementation to the new one.

While copying the XML job configuration file from the old server and moving it to the new one would be easy, replicating the hand-built monolithic build servers is difficult. Building and reusing template jobs would be a secondary option, but this would lead to technical debt in the future. Finally, developers were not enthused about constantly replicating basic global configuration changes such as email addresses between jobs.

The DevOps team quickly identified similarities across all jobs and sought to DRY up the code by building a shared library to abstract Jenkins Pipeline configuration and cover a vast majority of jobs. This made global changes much easier and we were able to pare down Jenkinsfiles to less than 10 lines of configuration parameters using wrappers. This saved hours of developers’ time learning Pipeline syntax and Groovy. To address the monolithic build servers, we created Dockerfiles in each app repository with the exact dependencies needed to build that particular app. Now, each repository has a Jenkinsfile that describes how to build, test and release the app, and a Dockerfile along with a Docker Compose file, that builds the environment in which the pipeline runs. This gives us confidence that future Jenkins migrations would be less cumbersome – as long as there was access to GitHub, agents where Docker containers could run and network connectivity to Artifactory where we store RPMs, all the jobs could be picked up and moved over with minimal effort.

There were many resources out already in the Jenkins community about these efforts such as:
- https://www.cloudbees.com/blog/need-jenkins-pipeline
and-
https://github.com/jenkinsci/simple-build-for-pipeline-plugin/
...but there was no case study about the successes and failures nor tricks and tips of how to implement this. We may not have done it the right way nor the most efficient way, but by sharing my story at FireEye, I hope to give others a valuable starting point they can cherry-pick ideas from.

This talk concludes with lessons learned of why we were successful, opinions of other Jenkins tools we tried such as Jenkins Job Builder and raw, scripted Pipeline, deployments with Fabric, and our plans to move to Declarative Pipeline in the future.

Transcript

1. Codifying the Build and Release Process with a Jenkins Pipeline Shared Library Alvin Huang
2. Codifying the Build and Release Process with a Jenkins Pipeline Shared Library Alvin Huang DevOps Engineer, FireEye
3. Who Am I • DevOps Engineer, GSI @ FireEye • Twitter: @RealAlvinHuang, IRC: ahuang • alvin.huang@fireeye.com
4. Agenda • Overview of GSI • The Great Migration • Why Pipeline? • Why Docker? • Shared Library Examples • How we did this • Lessons Learned • Benefits for the Future
5. Overview of GSI • Unified global organization combining Mandiant, FireEye iSIGHT Intelligence, and FireEye as a Service operations • DevOps Goal: Make the feedback loop between intelligence on the front lines to capabilities in software, smaller
6. What We Build
7. How We Build Code Test Build Deliver Deploy
8. The Great Migration
9. Scenario • Consolidate datacenters • Migrate Jenkins off a shared instance • ~60 days • ~150 Freestyle jobs • ~20 different build agents • Upgrade from 1.x to 2.x
10. Legacy Jenkins Jobs • Jenkins primarily used to run tests and build RPMs • SCP to publish RPMs to yum server • Ad-hoc Github triggers • No owner/SME for Jenkins
11. Goals • Standardize and make consistent CI pipelines • Jenkins SME = DevOps team • Make new app onboarding easy – Build abstraction that covers ~90% of jobs • Jenkins jobs tracked in git • Decouple tools from Jenkins/shell => Docker
12. Migration Strategies Migration Strategy LOE Advantages Disadvantages Copy XMLs from old server to new server Minimal - Easy, no extra work - Tech debt - Global changes are hard - Job config not in SCM Create Freestyle Jenkins job templates Moderate - Global changes are easier - Hide much of the configuration from end users - Make Freestyle jobs DRY - Tech debt - Debugging a job is hard - Job config not in SCM - Global changes can break all builds Convert Freestyle jobs to Pipeline jobs Significant - Job configuration in SCM - Learn pipeline syntax - Large job footprint - Global changes are hard - Too complex for many users - Subject to repetition and tech debt Create a Jenkins Shared Library for Pipeline jobs Moderate - Jobs in SCM - Standardized and consistent - Easy on-boarding, just needs parameters - Job configuration hidden from user - Learn pipeline syntax - Need owner for library - Global changes can break all builds
13. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle Jobs Jobs were not DRY
14. Solution: Build Wrapper
15. Solution: Use Github Organizations • Index every branch of every repo that has a Jenkinsfile • Configure one Github hook on the organization level
16. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper
17. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift
18. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift Jenkinsfiles in SCM
19. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift Jenkinsfiles in SCM Difficult to view job configurations and build steps
20. Solution
21. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift Jenkinsfiles in SCM Difficult to view job configurations and build steps Jenkinsfiles + Stage View
22. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift Jenkinsfiles in SCM Difficult to view job configurations and build steps Jenkinsfiles + Stage View Very difficult to make global changes
23. Solution yourBuildWrapper.groovy
24. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift Jenkinsfiles in SCM Difficult to view job configurations and build steps Jenkinsfiles + Stage View Very difficult to make global changes *Library changes have an effect on all builds
25. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift Jenkinsfiles in SCM Difficult to view job configurations and build steps Jenkinsfiles + Stage View Very difficult to make global changes *Library changes have an effect on all builds Monolithic build servers
26. Migrating Monolithic Build Agents • No configuration management • What packages and versions were installed? • Who runs on what agent? • Changes on a node may affect another team sharing that same agent
27. Solution to Monolithic Build Agents
28. Sample Dockerfile
29. Why Pipeline and Shared Libraries? Problem Solution Overhead with Freestyle jobs Use a pipeline library wrapper; Github Organizations Jobs were not DRY Use a pipeline library wrapper Job configuration drift Jenkinsfiles in SCM Difficult to view job configurations and build steps Jenkinsfiles + Stage View Very difficult to make global changes *Library changes have an effect on all builds Monolithic build servers Use Docker
30. What about… • Jenkins Job Builder – Simple jobs defined by relatively complex YAML – Verbose – Needs abstraction for novice users • Job DSL Plugin – Programmatically seeds/creates Jenkins jobs – Jenkins Pipeline + Github Organizations can be used instead – No logic between jobs (ie: no Pipeline ‘input’ step)
31. Starting Point • New Jenkins 2.x instance • Stable build nodes built with Puppet – CentOS 6/7 hosts • Working Docker integration
32. Before Pipeline
33. Phase 1 (Pre-migration)
34. Phase 2 (Pre-migration)
35. Visualization
36. The NEW World
37. Inspiration • https://github.com/jenkinsci/simple-build-for-pipeline- plugin -michaelneale • Build a wrapper called simpleBuild that takes in parameters as a closure and runs logic
38. Jenkins Pipeline Workflow
39. Example Jenkinsfile
40. Example Dockerfile
41. Resulting Pipeline
42. Demo
43. How do we do this? • Find a *passionate* liaison from each team • Learn their workflow – There’s probably a freestyle job already • Start pipelining their workflow in modular pieces: – Checkout code – Run build script – Archive artifacts – Publish to Artifactory – Promote artifact – Send email • Build a wrapper to call those functions • Repeat for another team/app, reusing what you just made
44. Reap the Benefits • Give Devs/QA their time back • Standardization of jobs and workflows • Standardization of build and test environments • Jenkins maintenance and resilience made easy by decoupling tools
45. Lessons Learned
46. Use Github Organizations • Index every branch of every repo that has a Jenkinsfile • Configure one Github hook on the organization level
47. Make Steps Modular • What does this step do? – Does A ‘AND’ B; break it apart • Namespace them
48. Leverage the Snippet Generator
49. Benefits of Shared Libraries For Our Future • Migration of Github • Upgrade to Puppet 5 • Consolidation of Artifactory • Application stacks in Openshift • Deployments through Rundeck, Ansible or Fabric • Scripts in ‘vars’ can be called in Declarative Pipeline
50. Upgrade to Puppet 5 Dockerfile Jenkinsfile
51. Go Forth and Pipeline!
52. Thanks! • alvin.huang@fireeye.com | @RealAlvinHuang | ahuang