EnterJS: DIY Full-Stack JavaScript CI CD

EnterJS Conference Julie Ng / Allianz Deutschland AG / 21
June 2018 DIY FULL-STACK DIY FULL-STACK JAVASCRIPT CI/CD JAVASCRIPT CI/CD

Julie Ng | DIY Full Stack JavaScript CI/CD | EnterJS
| Slides Version 26 June 2018 Designer, Developer, and now Designer, Developer, and now ENTERPRISE ARCHITECT ENTERPRISE ARCHITECT Building for the web since 1999 Designer, Developer, Failed Entrepreneuer Joined Allianz as Developer in 2016 Enterprise Architect since May 2017 Climber :)

CORPORATE CORPORATE MONOLITH MONOLITH 01

Revenue Operating Profit ALLIANZ GERMANY ALLIANZ GERMANY Financial Year 2017
Financial Year 2017 € 34.8 Billion € 2.3 Billion – Allianz Deutschland AG Zahlen, Daten und Fakten

• New Business Models • New Markets • 100% Digital
CHALLENGES FROM OUTSIDE CHALLENGES FROM OUTSIDE

HELLO LEMONADE HELLO LEMONADE Insurance claims are processed in 3
Minutes and paid out

| Slides Version 26 June 2018 How to processing claims in minutes? How to processing claims in minutes? NETFLIX ARCHITECTURES NETFLIX ARCHITECTURES — Ruslan Meshenberg, "Microservices at Netflix Scale: Principles, Tradeoffs & Lessons Learned", (GOTO 2016) 0:00 / 0:04

ALLIANZ ALLIANZ GOES AGILE GOES AGILE 02

| Slides Version 26 June 2018 Open Office & Team Spaces Open Office & Team Spaces AGILE TRAINING CENTER AGILE TRAINING CENTER

| Slides Version 26 June 2018 Agile Setup Agile Setup SCRUM, MACS AND PAIR PROGRAMMING SCRUM, MACS AND PAIR PROGRAMMING

| Slides Version 26 June 2018 SchadenOnline - Kraft (2017) SchadenOnline - Kraft (2017) SINGLE PAGE APPLICATIONS SINGLE PAGE APPLICATIONS

| Slides Version 26 June 2018 SchadenOnline - Kraft (2017) SchadenOnline - Kraft (2017) RICH USER INTERFACES RICH USER INTERFACES

| Slides Version 26 June 2018 SchadenOnline - Kraft (2017) SchadenOnline - Kraft (2017) FORMS WITH CONTEXT FORMS WITH CONTEXT

| Slides Version 26 June 2018 Frontend and Backend Frontend and Backend NODE.JS IN THE CLOUD NODE.JS IN THE CLOUD ...is cheaper (per Pivotal pricing) — Source: run.pivotal.io/pricing

| Slides Version 26 June 2018 Not just in the Frontend… Not just in the Frontend… NODE.JS IN THE BACKEND NODE.JS IN THE BACKEND Gently introduce new technologies Node.js a good candidate for worker processes

| Slides Version 26 June 2018 Node.js in the Backend Node.js in the Backend WORKER EXAMPLE WORKER EXAMPLE Actual app.js only has 50 lines. Queue is wrapper around busboy streaming-processor library. import express from 'express'; queue.on('connect', () => { // connected, but no routes http.listen(port, () => { console.log(`=== [${workerName}] listening on ${port}… ===` ); }); }); queue.on('message', (attrs) => { let route = `/api/upload/${attrs.id}`; queue.transmit('ready', Object.assign({ route: route }, attrs)); // dynamic routes per request http.post(`/upload/${attrs.id}`, (req, res) => { let upload = new Upload(attrs.id, req.headers); queue.transmit('begin', attrs); upload.on('file:end', () => { queue.transmit('done', attrs); res.writeHead(200, { 'Connection': 'close' }); res.end("That's all folks!"); }); return req.pipe(upload); }); });

| Slides Version 26 June 2018 Modern and awesome, but complicated Modern and awesome, but complicated CLOUD ARCHITECTURES CLOUD ARCHITECTURES CI/CD can help!

| Slides Version 26 June 2018 Continuous Integration Continuous Integration AND CONTINUOUS DELIVERY? AND CONTINUOUS DELIVERY? What if you're building distributed monoliths? Is it green? Is it red?

| Slides Version 26 June 2018 BENEFITS OF CI/CD BENEFITS OF CI/CD Low risk releases Faster time to market Higher quality Lower Costs Better products Happier teams

| Slides Version 26 June 2018 Section Summary Section Summary ALLIANZ GOES AGILE ALLIANZ GOES AGILE Allianz has Agile Training Centers (ATC) in Munich and Stuttgart. ATC Teams practice DevOps. They build it. They run it. ATCs practiced continuous integration and delivery from the beginning (2016). JavaScript was introduced first via frontends, especially Single Page Applications. ATC Stuttgart is full-stack JavaScript-centric. Java still dominates in the Allianz. Julie is trying to push JavaScript more in the Backend: Node.js applications are slimmer and therefore cheaper. When migrating to the cloud, look at architecture and find opportunity to split off processes into slim Node.js Workers.

AZD-CLOUD AZD-CLOUD STACK STACK 03

| Slides Version 26 June 2018 GITHUB GITHUB Source Control with GitHub Enterprise

| Slides Version 26 June 2018 JENKINS JENKINS The automation server, which controls the workflows aka pipelines.

| Slides Version 26 June 2018 ARTIFACTORY ARTIFACTORY Proxies and caches other registries, e.g. Maven, npmjs.com Hosts private libraries Archives builds artifacts Offers artifact promotion for CD workflows

| Slides Version 26 June 2018 SELENIUM HUB SELENIUM HUB Grab a browser for E2E testing Avoids the "works on my machine" problem Simultaneously test many browser versions

| Slides Version 26 June 2018 PIVOTAL CLOUD FOUNDRY PIVOTAL CLOUD FOUNDRY Internal PaaS Preferred application platform Utilizes Buildpacks cf push and done

| Slides Version 26 June 2018 Section Summary Section Summary AZD-CLOUD STACK AZD-CLOUD STACK AZD-Cloud is a hybrid cloud. Stack supports Java and JavaScript Ecosystems with: pre-configured Jenkins Tools. buildpacks on Cloud Foundry (PaaS). For Security and Privacy Reasons (§ 203 StGB Verletzung von Privatgeheimnissen), we use on- premise solutions, a good reason to take the do-it-yourself (DIY) approach. Allianz uses open-source software, e.g. Jenkins, Selenium Hub. Listed Tools are just a small slice of the AZD Cloud Landscape.

CONTINUOUS CONTINUOUS INTEGRATION INTEGRATION AT AZD AT AZD 04

| Slides Version 26 June 2018 WHAT IS DEVOPS? WHAT IS DEVOPS? You Build it. You run it. — Werner Vogels, CTO of Amazon “

| Slides Version 26 June 2018 Continuous Integration Continuous Integration AUTONOMOUS TEAMS DECIDE AUTONOMOUS TEAMS DECIDE Architecture Deployment Git Workﬂow

| Slides Version 26 June 2018 Continuous Integration Continuous Integration TEAMS BUILD OWN PIPELINES TEAMS BUILD OWN PIPELINES Teams define their own pipelines in code via a Jenkinsfile .

| Slides Version 26 June 2018 Continuous Integration Continuous Integration TESTS RUN AUTOMATICALLY TESTS RUN AUTOMATICALLY with every git push

| Slides Version 26 June 2018 Continuous Integration Continuous Integration PIPELINE AS CODE PIPELINE AS CODE Are pipelines really this complicated? @Library('jenkins-pipeline-library') _ String cfOrg = 'APP-Product' String buildDescription node { def downloadInfo def server def hipChat = new HipChat("119", this) try { buildDescription = "set up workspace" stage(setBuildDescription(buildDescription)) { cleanWorkspace('.git') } buildDescription = "Get artifacts from Artifactory" stage(setBuildDescription(buildDescription)) { String downloadSpec = """{ "files": [ { "pattern": "static-snapshot-local/de/allianz/app/product/app-product-frontend-${FrontendVersion}*.zip", "target": "./" }, {

| Slides Version 26 June 2018 Continuous Integration Continuous Integration DECLARATIVE PIPELINE DECLARATIVE PIPELINE Pipelines can also be clean. package.json FTW pipeline { tools { nodejs 'node-10' } stages { stage('Checkout') { steps { checkout scm } } stage('NPM Install') { steps { sh 'npm install' } } stage('Lint Code') { steps { sh 'npm run lint' } } stage('Unit Tests') { steps {

| Slides Version 26 June 2018 Continuous Integration Continuous Integration TEAMS OWN BUILD MONITOR TEAMS OWN BUILD MONITOR

| Slides Version 26 June 2018 Section Summary Section Summary CONTINUOUS INTEGRATION AT ALLIANZ CONTINUOUS INTEGRATION AT ALLIANZ DEUTSCHLAND DEUTSCHLAND ATC Teams are autonomous and practice DevOps. They code, deploy and maintain their own applications. Each team picks its own architecture and gitflow. Deployment strategy is dependent on those factors. Teams have their own Jenkins. Tests are run automatically with each git push , pull-request, etc. Teams build their own pipelines. Pipeline as Code: lots of configuration. Declarative Pipelines: easy to read and understand. JavaScript pipelines are clean, just a bunch of npm scripts . What is clean code? Can someone who is not the author extend it? Julie prefers declarative pipelines.

LIVE DEMO: LIVE DEMO: CI WITH AZD CI WITH AZD
STACK STACK 05

| Slides Version 26 June 2018 Continuous Integration Continuous Integration LIVE DEMO PREVIEW LIVE DEMO PREVIEW In this live demo, we'll look at: Introduction Jenkins UI Configuring Node.js Installations Demo: Adding Chrome for Unit Tests Multibranch Integration Note the following slides summarize what was shown in the demo.

| Slides Version 26 June 2018 JenkinsLive Demo JenkinsLive Demo JENKINS UI - BLUE OCEAN JENKINS UI - BLUE OCEAN Jenkins offers for a modern UI/UX. The UI is clean and integrates logs with each step. But it is difficult to see build history and health. For example, the build is red. Ok, how long has it been broken? Blue Ocean Plugin

| Slides Version 26 June 2018 JenkinsLive Demo JenkinsLive Demo JENKINS UI - CLASSIC JENKINS UI - CLASSIC Classic Jenkins: Dear Julie, your last 3 builds have been broken.

| Slides Version 26 June 2018 Jenkins Live Demo Jenkins Live Demo CONFIGURING NODE.JS CONFIGURING NODE.JS Under Manage Tools > Global Tool Configuration, you can setup multiple Node.js Installations. In Screenshot, we've installed version 8.11.3 and labeled it node-8 We preconfigure Node.js installations in the image. See example how at https://github.com/allianz-de/cidemo-jenkins

| Slides Version 26 June 2018 Jenkins Live Demo Jenkins Live Demo INSTALL CHROME FOR UNIT TESTS INSTALL CHROME FOR UNIT TESTS Docker lets your customize your Jenkins Images easily, for example: N.B. That is a generic install. In real-life, you probably want to install a specific version and do a checksum validation. FROM jenkins/jenkins:lts # Install Chrome as root USER root RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \ sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' && \ apt-get update && \ apt-get install apt-transport-https google-chrome-stable -y # Drop back to jenkins user USER jenkins ENV CHROME_BIN=/usr/bin/google-chrome-stable

| Slides Version 26 June 2018 Jenkins Live Demo Jenkins Live Demo USING CHROME FOR HEADLESS TESTS USING CHROME FOR HEADLESS TESTS Once you install Chrome on Jenkins, you can use it headlessly as you would locally. Excerpt Karma Configuration NPM Script NPM Script View Demo Project at customLaunchers: { ChromeHeadless: { base: 'Chrome', flags: [ '--headless', '--disable-gpu', '--no-sandbox', // Without a remote debugging port, Google Chrome exits immediately. '--remote-debugging-port=9222', ], } } "scripts": { "test:headless": "ng test --single-run --progress false --browser ChromeHeadless", } https://github.com/allianz-de/cidemo-angular

| Slides Version 26 June 2018 Jenkins Live Demo Jenkins Live Demo MULTIBRANCH PIPELINES - AUTOMATE MULTIBRANCH PIPELINES - AUTOMATE FEATURE BUILDS FEATURE BUILDS Leverage automate building all your code Just do git push origin mybranch Jenkins will create separate builds per branch automatically Each branch builds based on the Jenkinsfile in that branch multibranch pipelines

| Slides Version 26 June 2018 Jenkins Live Demo Jenkins Live Demo BRANCH BUILD STATUS ON GITHUB BRANCH BUILD STATUS ON GITHUB Jenkins can also report build status ✅ or ❌ to GitHub! Install the . Requires little/no configuration. Make sure you GitHub user tokens have enough permissions. See example results at Note the "unconfigured-jenkins-location" in the checkmark links from Julie's local Docker installation GitHub Plugin github.com/allianz-de/cidemo-angular/branches

| Slides Version 26 June 2018 Section Summary Section Summary DIY CONTINUOUS INTEGRATION DIY CONTINUOUS INTEGRATION Customize your Jenkins with Docker Preinstall plug-ins Preconfigure credentials with placeholders Install tools, e.g. Chrome for headless unit tests Configure and manage multiple Node.js installations Let Jenkins automatically build every branch with multibranch pipelines Multibranch pipelines and conventions key to automation Jenkins can automatically notify GitHub

CONTINUOUS CONTINUOUS DELIVERY (CD) DELIVERY (CD) 06

| Slides Version 26 June 2018 Continuous Delivery Continuous Delivery WHAT IS CONTINUOUS DELIVERY? WHAT IS CONTINUOUS DELIVERY? Extension of Continuous Integration Automated Release Process Application can be deployed with 1 click Continuous deployment extends delivery further, automatically releases to production.

| Slides Version 26 June 2018 NODE MODULES NODE MODULES 30 percent of all modules rely indirectly on native modules. — “ Node.js Foundation

| Slides Version 26 June 2018 Continuous Delivery Continuous Delivery NODE MODULES & ENVIRONMENTS NODE MODULES & ENVIRONMENTS Does the node_modules folder change across environments? Developer machine Jenkins (runs tests, builds code) Platform (where app is deployed) Julie asks are you confident in your code? Yes? Stop worrying. No? Then build and deploy Docker images.

| Slides Version 26 June 2018 Continuous Delivery Continuous Delivery MULTIBRANCH DELIVERY MULTIBRANCH DELIVERY Challenges at Allianz Challenges at Allianz Dependent on Legacy System Release Cycle, only a few per year Review with business stakeholders can take a long time Can you automatically correlate multibranch deployments?

LIVE DEMO: LIVE DEMO: CD WITH AZD CD WITH AZD
STACK STACK 07

| Slides Version 26 June 2018 Continuous Delivery Continuous Delivery LIVE DEMO PREVIEW LIVE DEMO PREVIEW In this live demo, we'll look at: Multibranch Deployments Integrating E2E Tests (pulled from Frontend) Matching branch deployments in a distributed monolith

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo BRANCH SPECIFIC STAGES BRANCH SPECIFIC STAGES When using a multibranch pipeline, you can with the when directive and branch property: You can also use advanced expressions with code: where isFeatureBranch() is a pipeline library method. filter branches stage('Release') { when { when { branch 'release' } } steps { // Do a release } } stage('Feature') { when { expression { return isFeatureBranch() || someOtherLibrarMethod() } } steps { // Do something } }

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo BRANCH DEPLOYMENT NAMING PATTERNS BRANCH DEPLOYMENT NAMING PATTERNS Using the conventions mentioned before and in the , we will create the following pattern. Branch Name App Name master demo-app release demo-app-prod feat/story-1 demo-app-feat-story-1 feat/story-2 demo-app-feat-story-2 where each app is deployed at its own URL, for example cidemo https://demo-app-feat-story-1.cfapps.io

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo HOW TO DEPLOY BRANCHES HOW TO DEPLOY BRANCHES 1. Read application name from manifest.yml, the Cloud Foundry configuration file 2. Adjust name as necessary (per conventions) before deploying View helper method source at stage('Deploy') { when { expression { return isDeployBranch() } } steps { script { String appName = isFeatureBranch() ? appNameFromManifest(append: env.BRANCH_NAME) // result: demo-feat-1.cfapps.io : appNameFromManifest() // result: demo.cfapps.io // specifying name overrides manifest.yml cfPush([ appName: appName, apiUrl: params.CF_API, org: params.CF_ORG, space: params.CF_SPACE, credentialsId: 'pcf' ]) } } } https://github.com/allianz-de/cidemo-pipeline-library

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo CAN WE MATCH FEATURE DEPLOYMENTS? CAN WE MATCH FEATURE DEPLOYMENTS? Assume frontend is a Single Page Application served by an container… Assume all /api/ URLs are handled by a middleware application… How do you correlate frontend-feature-1 with middleware-feature-1 ? nginx

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo HACK... CLOUD FOUNDRY BUILDPACKS HACK... CLOUD FOUNDRY BUILDPACKS Knowing Cloud Foundry , we can 1. add our own nginx.conf 2. include our own variables in the location blocks, for example: 3. and then use cf set-env and cf restage in our pipelines to correlate the deployments. N.B. This demo was built before introduction of the nginx buildpack. If you are trying this new, do not hack the staticfile buildpack. Use the nginx buildpack instead. See for details. evaluates embedded ruby before storing the nginx.conf location /api/ { proxy_pass <%= ENV["API_ENDPOINT"] %>/api/; } https://github.com/cloudfoundry/staticfile-buildpack/issues/103

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo THAT'S NOT MICROSERVICES! THAT'S NOT MICROSERVICES! Correct. That is a distributed monolith. Why?! Not all organizations are microservices-ready. … There's some justification for not starting with a microservices architecture until the domain and the proper contexts are more clear. — “ Rebecca Parisons, CTO, ThoughtWorks

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo PROTIP: CI FOR CI PROTIP: CI FOR CI How do you test a custom CI/CD stack? With unit tests? No. With the stack itself Create new GitHub organization with repositories per stack or toolchain. Jenkins can autodiscover repositories and branches within an organization.

| Slides Version 26 June 2018 Continuous Delivery - Live Demo Continuous Delivery - Live Demo PROTIP: CI FOR CI PROTIP: CI FOR CI Leverage multibranch pipelines to test toolchains and versions. This demo repository serves as integration testing for most common pipeline stages, e.g. production builds, deployments, etc. These integration tests are slower. But they are also easy, low maintenance and meaningful ✌

| Slides Version 26 June 2018 Section Summary Section Summary CD & DISTRIBUTED MONOLITHS CD & DISTRIBUTED MONOLITHS It's acceptable to start with distributed monoliths and move to Microservices as a goal. How do you automate CD of distributed monoliths? 1. Assume we correlate deployments via branch names 2. Leverage multibranch pipelines 3. Target branches with when directive and branch property 4. Overwrite app name when deploying 5. Configure communication endpoints dynamically in pipeline* How do you automate testing of your CI stack? With the stack itself Leverage repository autodiscovery for GitHub organizations Leverage autobuilds in multibranch pipelines *This demo showed a workaround conceived mid-2017. Since then there is an official nginx buildpack that should be used instead. You can also use a Node.js Express app to serve the frontend. The same principle of using endpoints set by environment variables applies.

THANK YOU THANK YOU We're Hiring! We're Hiring! Julie Ng
Enterprise Architect [email protected]

EnterJS: DIY Full-Stack JavaScript CI CD

EnterJS: DIY Full-Stack JavaScript CI CD

More Decks by jng

Other Decks in Technology

Featured

Transcript