Jenkins Online Meetup - Jenkins Templating Engine

Jenkins Templating Engine allows you to separate the business logic of your pipeline (what should happen, and when) from the technical implementation by creating pipeline templates and separating the implementation of the pipeline actions defined in the template out into pipeline libraries.

The idea is that regardless of which specific tools are being used there are common steps that often take place, such as unit testing, static code analysis, packaging an artifact, and deploying that artifact to an application environment.

Come join us for a session where Steven Terrana will share insights on the development, evolution, and future of the Jenkins Templating Engine. He'll share success stories and answer questions from the audience as well.

Transcript

1. Jenkins Templating Engine - Pipeline Templates for Jenkins Jenkins Online Meetup Steven Terrana
2. steven_terrana steven-terrana Introductions • Senior Lead Technologist at Booz Allen Hamilton • Principal Engineer for Booz Allen’s DevSecOps capability development • Platform Engineer – Kubernetes/OpenShift • Helped build DevOps platforms for multiple federal agencies supporting multiple 2
3. steven_terrana steven-terrana 3 • Shift-left many security and compliance activities as a shared responsibility of the whole team • Educate and automate security vigilance to establish early detection, confidence, and trust required for Continuous Delivery • Perform vulnerability and compliance inspection of dependencies, code, container images, and running applications ACCESSIBILITY ASSURANCE Crawl web pages for compliance with section 508 standards to give developers early warning and opportunity to improve the site while accelerating manual 508 testing. STATIC CODE ANALYSIS Analyze the code written by developers for inadvertent technical and logical flaws that make it vulnerable. CONTINUOUS COMPLIANCE Routinely scan the configuration of hosts or containers in their packaged image state or at runtime for compliance with security policy groups (NIST, CIS, FISMA, STIG, etc.), for required patches, or for configuration drift. DEPENDENCIES Prevent introduction of vulnerabilities from the outside. Scan libraries in dependency repos, source code repos, and on disk for known vulnerabilities. IMAGE SCANNING Unpack and scan dependencies and configuration of the image to be used at runtime for vulnerabilities, out-of-date patching, and to ensure a trusted pedigree. DYNAMIC APPLICATION SECURITY TESTING Perform automated penetration testing to see how your application will withstand common attacks at runtime. Security and compliance are indicative of the same software delivery sins that spawned the DevOps movement. Work piles up because it is tedious, foreign, or difficult. Security pros are alienated and left to burn down the pile in isolation, as an afterthought. True concerns then become hugely disruptive, which breeds further discontent within the team. AS WITH QUALITY ASSURANCE, SECURITY ASSURANCE AND COMPLIANCE CAN BE INTEGRATED INTO YOUR SOFTWARE DEVELOPMENT LIFECYCLE DevSecOps: Build a Trusted Software Supply Chain CONTINUOUS SECURITY & COMPLIANCE IS PERVASIVE IN OUR DEVOPS APPROACH. IT CROSS-CUTS EVERY PRACTICE AREA
4. Example DevSecOps Pipeline Flow 4 Security Profile Compliance ENVIRONMENT PROMOTION Production Readiness / Maturity Select Story for Development Pull source code from SCM repo Unit Test Code Coverage Build Container Image CVEs Tag Image with Git SHA Publish Image Static Code Analysis Functional Testing Regression Testing Integration Testing Download Build Dependencies Generate BOM Sign Image Development Test / QA Staging Production Load Testing Soak Testing Validate Image Signature Validate Runtime Configs Validate Image Signature Validate Runtime Configs Validate Image Signature Validate Runtime Configs Log Aggregation Resource Utilization Continuous Runtime Security Scan Build Dependencies Accessibility Compliance Scan Penetration Testing ACAS & HBSS Scan Fuzzing DEPLOY TO PRODUCTION MONITOR SYSTEM TESTING DEPLOY TO STAGING SYSTEM TESTING DEPLOY TO TEST PUBLISH ARTIFACT SCAN ARTIFACT BUILD ARTIFACT TEST PLAN & DEVELOP Scan Image
5. steven_terrana steven-terrana Challenges at Scale 1/21/20 5 Jenkins Templating Engine Time Creating a mature DevSecOps pipeline for an application can take months. Onboarding new applications requires manual intervention. Complexity Different types of applications will utilize different tools and different teams may leverage different testing frameworks. Standardization Each application’s source code repository requires a Jenkinsfile, making it difficult to ensure common processes are adhered to. Continuous Improvement Making a change to the pipeline requires changing Jenkinsfiles distributed across every branch in every source code repository.
6. steven_terrana steven-terrana Challenges at Scale: Multiplied 1/21/20 6 Jenkins Templating Engine applications supported clients supported Client 1 Client 2 Client 3 Client 4 As IT Consultants at Booz Allen, these challenges are multiplied as we encounter them again and again across different client engagements.
7. steven_terrana steven-terrana Our Goals 1/21/20 7 Jenkins Templating Engine Decrease the time it takes to instrument a mature DevSecOps pipeline Lower the technical barrier to entry for teams to automate their software delivery processes • Writing a Jenkins pipeline is largely undifferentiated work • Ought to be able to modularize development to reuse pipeline code • Teams should configure – not build – their pipelines Bring standardization & governance to our software delivery by creating reusable pipeline templates
8. steven_terrana steven-terrana How We Scaled: Pipeline Templates 1/21/20 8 Jenkins Templating Engine stage("Maven: Build"){ docker.image("maven").inside{ sh "mvn clean package" } } stage('SonarQube: Static Code Analysis') { node { def scannerHome = tool 'SonarScanner 4.0’; withSonarQubeEnv('My SonarQube Server') { sh "${scannerHome}/bin/sonar-scanner" } } } Example Jenkinsfile for an application using Maven stage(”Gradle: Build"){ docker.image(”gradle").inside{ sh ”gradle clean build" } } stage('SonarQube: Static Code Analysis') { node { def scannerHome = tool 'SonarScanner 4.0’; withSonarQubeEnv('My SonarQube Server') { sh "${scannerHome}/bin/sonar-scanner" } } } Example Jenkinsfile for an application using Gradle build() static_code_analysis() Pipeline Template Regardless of what tools are being used, the flow remains the same.
9. steven_terrana steven-terrana Pipeline Configuration Repository 1/21/20 9 Jenkins Templating Engine . ├── libraries │ ├── maven │ │ └── build.groovy │ ├── gradle │ │ └── build.groovy │ └── sonarqube │ └── static_code_analysis.groovy └── pipeline-configuration └── Jenkinsfile Pipeline Configuration Repository libraries{ maven sonarqube } maven application pipeline_config.groovy libraries{ gradle sonarqube } gradle application pipeline_config.groovy build() static_code_analysis() Pipeline Template Jenkinsfile void call(){ stage("Maven: Build"){ docker.image("maven").inside{ sh "mvn clean package" } } } void call(){ stage(”Gradle: Build"){ docker.image(”gradle").inside{ sh ”gradle clean build" } } } void call(){ stage('SonarQube: Static Code Analysis') { node { def scannerHome = tool 'SonarScanner 4.0’; withSonarQubeEnv('My SonarQube Server') { sh "${scannerHome}/bin/sonar-scanner" } } } } Library Steps
10. steven_terrana steven-terrana Making Libraries Configurable 1/21/20 Jenkins Templating Engine 10 void call(){ stage('SonarQube: Static Code Analysis') { // parse configuration String scannerVersion = config.scanner_version ?: "SonarScanner 4.0" String serverName = config.server_name ?: "My SonarQube Server" Boolean enforceQualityGate = config.containsKey("enforce_quality_gate") ? config.enforce_quality_gate : true node { def scannerHome = tool(scannerVersion) withSonarQubeEnv(serverName) { sh "${scannerHome}/bin/sonar-scanner" } } timeout(time: 1, unit: 'HOURS') { def qg = waitForQualityGate() if (qg.status != 'OK') { if(enforceQualityGate){ error "Pipeline aborted due to quality gate failure: ${qg.status}" } else { warning "Quality gate failure: ${qg.status}" } } } } } libraries{ maven sonarqube{ scanner_version = “SonarScanner 3.0” enforce_quality_gate = false } } Library steps autowired with a config variable populated with values from the pipeline configuration. Libraries become reusable building blocks used configure pipelines. Libraries can externalize their configuration to optimize reusability.
11. steven_terrana steven-terrana Hierarchical Pipeline Configurations 1/21/20 Jenkins Templating Engine 11 libraries{ maven sonarqube } maven application libraries{ gradle sonarqube } gradle application libraries{ maven } libraries{ gradle } Organizational Pipeline Configurations Application-Specific Pipeline Configurations Aggregated Pipeline Configurations libraries{ merge = true sonarqube } libraries{ merge = true sonarqube }
12. steven_terrana steven-terrana 1/21/20 Jenkins Templating Engine 12 DEMO
13. steven_terrana steven-terrana Key Takeaways 1/21/20 13 Jenkins Templating Engine • The Jenkins Templating Engine is a framework for developing tool-agnostic, templated workflows that can be reused by multiple teams simultaneously – regardless of the tools they are using. • This approach separates the business logic (pipeline template) from the technical implementation (pipeline libraries) allowing teams to configure their pipelines instead of build them from scratch Apply Organizational Governance By centralizing your pipeline definition to a common place you can standardize your software delivery processes across teams Optimize Pipeline Code Reuse Create modularized tool- integrations called pipeline libraries that can be reused and collectively maintained Simplify Pipeline Maintainability Each application’s source code repository requires a Jenkinsfile, making it difficult to ensure common processes are adhered to. At Booz Allen, we’ve seen pipeline development decrease from 5 months to 5 days for new projects when leveraging existing tool integrations
14. steven_terrana steven-terrana Learn More! 1/21/20 Jenkins Templating Engine (JTE) 14 Documentation Learning Labs Gitter Channel Documentation: https://jenkinsci.github.io/templating-engine-plugin Learning Labs: https://boozallen.github.io/sdp-docs/html/pages/learning-labs/jte-the-basics/index.html Gitter Channel: https://gitter.im/jenkinsci/templating-engine-plugin