EnterJS: DIY Full-Stack JavaScript CI CD

EnterJS Conference
Julie Ng / Allianz Deutschland AG / 21 June 2018
DIY FULL-STACK
DIY FULL-STACK
JAVASCRIPT CI/CD
JAVASCRIPT CI/CD

View Slide

Julie Ng | DIY Full Stack JavaScript CI/CD | EnterJS | Slides Version 26 June 2018
Designer, Developer, and now
Designer, Developer, and now
ENTERPRISE ARCHITECT
ENTERPRISE ARCHITECT
Building for the web since 1999
Designer, Developer, Failed
Entrepreneuer
Joined Allianz as Developer in 2016
Enterprise Architect since May 2017
Climber :)

View Slide

CORPORATE
CORPORATE
MONOLITH
MONOLITH
01

View Slide

Revenue
Operating Profit
ALLIANZ GERMANY
ALLIANZ GERMANY
Financial Year 2017
Financial Year 2017
€ 34.8 Billion
€ 2.3 Billion
– Allianz Deutschland AG Zahlen, Daten und Fakten

View Slide

• New Business Models
• New Markets
• 100% Digital
CHALLENGES FROM OUTSIDE
CHALLENGES FROM OUTSIDE

View Slide

HELLO LEMONADE
HELLO LEMONADE
Insurance claims
are processed in 3 Minutes
and paid out

View Slide

How to processing claims in minutes?
How to processing claims in minutes?
NETFLIX ARCHITECTURES
NETFLIX ARCHITECTURES
— Ruslan Meshenberg, "Microservices at Netflix Scale: Principles, Tradeoffs & Lessons Learned", (GOTO 2016)
0:00 / 0:04

View Slide

ALLIANZ
ALLIANZ
GOES AGILE
GOES AGILE
02

View Slide

Open Office & Team Spaces
Open Office & Team Spaces
AGILE TRAINING CENTER
AGILE TRAINING CENTER

View Slide

Agile Setup
Agile Setup
SCRUM, MACS AND PAIR PROGRAMMING
SCRUM, MACS AND PAIR PROGRAMMING

View Slide

SchadenOnline - Kraft (2017)
SINGLE PAGE APPLICATIONS
SINGLE PAGE APPLICATIONS

View Slide

RICH USER INTERFACES
RICH USER INTERFACES

View Slide

FORMS WITH CONTEXT
FORMS WITH CONTEXT

View Slide

Frontend and Backend
Frontend and Backend
NODE.JS IN THE CLOUD
NODE.JS IN THE CLOUD
...is cheaper (per Pivotal pricing)
— Source: run.pivotal.io/pricing

View Slide

Not just in the Frontend…
Not just in the Frontend…
NODE.JS IN THE BACKEND
NODE.JS IN THE BACKEND
Gently introduce new technologies
Node.js a good candidate for worker processes

View Slide

Node.js in the Backend
Node.js in the Backend
WORKER EXAMPLE
WORKER EXAMPLE
Actual app.js only has 50 lines.
Queue is wrapper around busboy streaming-processor library.
import express from 'express';
queue.on('connect', () => {
// connected, but no routes
http.listen(port, () => { console.log(`=== [${workerName}] listening on ${port}… ===` ); });
});
queue.on('message', (attrs) => {
let route = `/api/upload/${attrs.id}`;
queue.transmit('ready', Object.assign({ route: route }, attrs));
// dynamic routes per request
http.post(`/upload/${attrs.id}`, (req, res) => {
let upload = new Upload(attrs.id, req.headers);
queue.transmit('begin', attrs);
upload.on('file:end', () => {
queue.transmit('done', attrs);
res.writeHead(200, { 'Connection': 'close' });
res.end("That's all folks!");
});
return req.pipe(upload);
});
});

View Slide

Modern and awesome, but complicated
Modern and awesome, but complicated
CLOUD ARCHITECTURES
CLOUD ARCHITECTURES
CI/CD can help!

View Slide

Continuous Integration
AND CONTINUOUS DELIVERY?
AND CONTINUOUS DELIVERY?
What if you're building distributed monoliths?
Is it green? Is it red?

View Slide

BENEFITS OF CI/CD
BENEFITS OF CI/CD
Low risk releases
Faster time to market
Higher quality
Lower Costs
Better products
Happier teams

View Slide

Section Summary
Section Summary
ALLIANZ GOES AGILE
ALLIANZ GOES AGILE
Allianz has Agile Training Centers (ATC) in Munich and Stuttgart.
ATC Teams practice DevOps. They build it. They run it.
ATCs practiced continuous integration and delivery from the beginning (2016).
JavaScript was introduced first via frontends, especially Single Page Applications.
ATC Stuttgart is full-stack JavaScript-centric.
Java still dominates in the Allianz.
Julie is trying to push JavaScript more in the Backend:
Node.js applications are slimmer and therefore cheaper.
When migrating to the cloud, look at architecture and find opportunity to split off processes into
slim Node.js Workers.

View Slide

AZD-CLOUD
AZD-CLOUD
STACK
STACK
03

View Slide

View Slide

GITHUB
GITHUB
Source Control with GitHub Enterprise

View Slide

JENKINS
JENKINS
The automation server, which controls the workflows aka pipelines.

View Slide

ARTIFACTORY
ARTIFACTORY
Proxies and caches other registries, e.g. Maven, npmjs.com
Hosts private libraries
Archives builds artifacts
Offers artifact promotion for CD workflows

View Slide

SELENIUM HUB
SELENIUM HUB
Grab a browser for E2E testing
Avoids the "works on my machine" problem
Simultaneously test many browser versions

View Slide

PIVOTAL CLOUD FOUNDRY
PIVOTAL CLOUD FOUNDRY
Internal PaaS
Preferred application platform
Utilizes Buildpacks
cf push and done

View Slide

Section Summary
Section Summary
AZD-CLOUD STACK
AZD-CLOUD STACK
AZD-Cloud is a hybrid cloud.
Stack supports Java and JavaScript Ecosystems with:
pre-configured Jenkins Tools.
buildpacks on Cloud Foundry (PaaS).
For Security and Privacy Reasons (§ 203 StGB Verletzung von Privatgeheimnissen), we use on-
premise solutions, a good reason to take the do-it-yourself (DIY) approach.
Allianz uses open-source software, e.g. Jenkins, Selenium Hub.
Listed Tools are just a small slice of the AZD Cloud Landscape.

View Slide

CONTINUOUS
CONTINUOUS
INTEGRATION
INTEGRATION
AT AZD
AT AZD
04

View Slide

WHAT IS DEVOPS?
WHAT IS DEVOPS?
You Build it. You run it.
— Werner Vogels, CTO of Amazon
“

View Slide

AUTONOMOUS TEAMS DECIDE
AUTONOMOUS TEAMS DECIDE
Architecture
Deployment
Git Workﬂow

View Slide

TEAMS BUILD OWN PIPELINES
TEAMS BUILD OWN PIPELINES
Teams define their own pipelines in code via a Jenkinsfile .

View Slide

TESTS RUN AUTOMATICALLY
TESTS RUN AUTOMATICALLY
with every git push

View Slide

PIPELINE AS CODE
PIPELINE AS CODE
Are pipelines really this complicated?
@Library('jenkins-pipeline-library') _
String cfOrg = 'APP-Product'
String buildDescription
node {
def downloadInfo
def server
def hipChat = new HipChat("119", this)
try {
buildDescription = "set up workspace"
stage(setBuildDescription(buildDescription)) {
cleanWorkspace('.git')
}
buildDescription = "Get artifacts from Artifactory"
stage(setBuildDescription(buildDescription)) {
String downloadSpec = """{
"files": [
{
"pattern": "static-snapshot-local/de/allianz/app/product/app-product-frontend-${FrontendVersion}*.zip",
"target": "./"
},
{

View Slide

DECLARATIVE PIPELINE
DECLARATIVE PIPELINE
Pipelines can also be clean. package.json FTW
pipeline {
tools {
nodejs 'node-10'
}
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('NPM Install') {
steps {
sh 'npm install'
}
}
stage('Lint Code') {
steps {
sh 'npm run lint'
}
}
stage('Unit Tests') {
steps {

View Slide

TEAMS OWN BUILD MONITOR
TEAMS OWN BUILD MONITOR

View Slide

Section Summary
Section Summary
CONTINUOUS INTEGRATION AT ALLIANZ
CONTINUOUS INTEGRATION AT ALLIANZ
DEUTSCHLAND
DEUTSCHLAND
ATC Teams are autonomous and practice DevOps. They code, deploy and maintain their own
applications.
Each team picks its own architecture and gitflow. Deployment strategy is dependent on those
factors.
Teams have their own Jenkins.
Tests are run automatically with each git push , pull-request, etc.
Teams build their own pipelines.
Pipeline as Code: lots of configuration.
Declarative Pipelines: easy to read and understand.
JavaScript pipelines are clean, just a bunch of npm scripts .
What is clean code? Can someone who is not the author extend it? Julie prefers declarative
pipelines.

View Slide

LIVE DEMO:
LIVE DEMO:
CI WITH AZD
CI WITH AZD
STACK
STACK
05

View Slide

LIVE DEMO PREVIEW
LIVE DEMO PREVIEW
In this live demo, we'll look at:
Introduction Jenkins UI
Configuring Node.js Installations
Demo: Adding Chrome for Unit Tests
Multibranch Integration
Note the following slides summarize what was shown in the demo.

View Slide

JenkinsLive Demo
JenkinsLive Demo
JENKINS UI - BLUE OCEAN
JENKINS UI - BLUE OCEAN
Jenkins offers for a modern UI/UX.
The UI is clean and integrates logs with each step. But it is difficult to see build history and health.
For example, the build is red. Ok, how long has it been broken?
Blue Ocean Plugin

View Slide

JenkinsLive Demo
JenkinsLive Demo
JENKINS UI - CLASSIC
JENKINS UI - CLASSIC
Classic Jenkins: Dear Julie, your last 3 builds have been broken.

View Slide

Jenkins Live Demo
Jenkins Live Demo
CONFIGURING NODE.JS
CONFIGURING NODE.JS
Under Manage Tools > Global Tool Configuration, you can setup multiple Node.js Installations.
In Screenshot, we've installed version 8.11.3 and labeled it node-8
We preconfigure Node.js installations in the image.
See example how at https://github.com/allianz-de/cidemo-jenkins

View Slide

Jenkins Live Demo
Jenkins Live Demo
INSTALL CHROME FOR UNIT TESTS
INSTALL CHROME FOR UNIT TESTS
Docker lets your customize your Jenkins Images easily, for example:
N.B. That is a generic install. In real-life, you probably want to install a specific version and do a
checksum validation.
FROM jenkins/jenkins:lts
# Install Chrome as root
USER root
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - && \
sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list' && \
apt-get update && \
apt-get install apt-transport-https google-chrome-stable -y
# Drop back to jenkins user
USER jenkins
ENV CHROME_BIN=/usr/bin/google-chrome-stable

View Slide

Jenkins Live Demo
Jenkins Live Demo
USING CHROME FOR HEADLESS TESTS
USING CHROME FOR HEADLESS TESTS
Once you install Chrome on Jenkins, you can use it headlessly as you would locally.
Excerpt Karma Configuration
NPM Script
NPM Script
View Demo Project at
customLaunchers: {
ChromeHeadless: {
base: 'Chrome',
flags: [
'--headless',
'--disable-gpu',
'--no-sandbox',
// Without a remote debugging port, Google Chrome exits immediately.
'--remote-debugging-port=9222',
],
}
}
"scripts": {
"test:headless": "ng test --single-run --progress false --browser ChromeHeadless",
}
https://github.com/allianz-de/cidemo-angular

View Slide

Jenkins Live Demo
Jenkins Live Demo
MULTIBRANCH PIPELINES - AUTOMATE
MULTIBRANCH PIPELINES - AUTOMATE
FEATURE BUILDS
FEATURE BUILDS
Leverage automate building all your code
Just do git push origin mybranch
Jenkins will create separate builds per branch automatically
Each branch builds based on the Jenkinsfile in that branch
multibranch pipelines

View Slide

Jenkins Live Demo
Jenkins Live Demo
BRANCH BUILD STATUS ON GITHUB
BRANCH BUILD STATUS ON GITHUB
Jenkins can also report build status ✅ or ❌ to GitHub!
Install the . Requires little/no configuration. Make sure you GitHub user tokens have
enough permissions.
See example results at
Note the "unconfigured-jenkins-location" in the checkmark links from Julie's local Docker installation
GitHub Plugin
github.com/allianz-de/cidemo-angular/branches

View Slide

Section Summary
Section Summary
DIY CONTINUOUS INTEGRATION
DIY CONTINUOUS INTEGRATION
Customize your Jenkins with Docker
Preinstall plug-ins
Preconfigure credentials with placeholders
Install tools, e.g. Chrome for headless unit tests
Configure and manage multiple Node.js installations
Let Jenkins automatically build every branch with multibranch pipelines
Multibranch pipelines and conventions key to automation
Jenkins can automatically notify GitHub

View Slide

CONTINUOUS
CONTINUOUS
DELIVERY (CD)
DELIVERY (CD)
06

View Slide

Continuous Delivery
Continuous Delivery
WHAT IS CONTINUOUS DELIVERY?
WHAT IS CONTINUOUS DELIVERY?
Extension of Continuous Integration
Automated Release Process
Application can be deployed with 1 click
Continuous deployment extends delivery further, automatically releases to production.

View Slide

NODE MODULES
NODE MODULES
30 percent of all modules rely indirectly on native modules.
—
“
Node.js Foundation

View Slide

Continuous Delivery
Continuous Delivery
NODE MODULES & ENVIRONMENTS
NODE MODULES & ENVIRONMENTS
Does the node_modules folder change across environments?
Developer machine
Jenkins (runs tests, builds code)
Platform (where app is deployed)
Julie asks are you confident in your code?
Yes? Stop worrying.
No? Then build and deploy Docker images.

View Slide

Continuous Delivery
Continuous Delivery
MULTIBRANCH DELIVERY
MULTIBRANCH DELIVERY
Challenges at Allianz
Challenges at Allianz
Dependent on Legacy System Release Cycle, only a few per year
Review with business stakeholders can take a long time
Can you automatically correlate multibranch deployments?

View Slide

LIVE DEMO:
LIVE DEMO:
CD WITH AZD
CD WITH AZD
STACK
STACK
07

View Slide

Continuous Delivery
Continuous Delivery
LIVE DEMO PREVIEW
LIVE DEMO PREVIEW
In this live demo, we'll look at:
Multibranch Deployments
Integrating E2E Tests (pulled from Frontend)
Matching branch deployments in a distributed monolith

View Slide

Continuous Delivery - Live Demo
BRANCH SPECIFIC STAGES
BRANCH SPECIFIC STAGES
When using a multibranch pipeline, you can with the when directive and branch
property:
You can also use advanced expressions with code:
where isFeatureBranch() is a pipeline library method.
filter branches
stage('Release') {
when {
when { branch 'release' }
}
steps {
// Do a release
}
}
stage('Feature') {
when {
expression {
return isFeatureBranch() || someOtherLibrarMethod()
}
}
steps {
// Do something
}
}

View Slide

BRANCH DEPLOYMENT NAMING PATTERNS
BRANCH DEPLOYMENT NAMING PATTERNS
Using the conventions mentioned before and in the , we will create the following pattern.
Branch Name App Name
master demo-app
release demo-app-prod
feat/story-1 demo-app-feat-story-1
feat/story-2 demo-app-feat-story-2
where each app is deployed at its own URL, for example
cidemo
https://demo-app-feat-story-1.cfapps.io

View Slide

HOW TO DEPLOY BRANCHES
HOW TO DEPLOY BRANCHES
1. Read application name from manifest.yml, the Cloud Foundry configuration file
2. Adjust name as necessary (per conventions) before deploying
View helper method source at
stage('Deploy') {
when {
expression { return isDeployBranch() }
}
steps {
script {
String appName = isFeatureBranch()
? appNameFromManifest(append: env.BRANCH_NAME) // result: demo-feat-1.cfapps.io
: appNameFromManifest() // result: demo.cfapps.io
// specifying name overrides manifest.yml
cfPush([
appName: appName,
apiUrl: params.CF_API,
org: params.CF_ORG,
space: params.CF_SPACE,
credentialsId: 'pcf'
])
}
}
}
https://github.com/allianz-de/cidemo-pipeline-library

View Slide

CAN WE MATCH FEATURE DEPLOYMENTS?
CAN WE MATCH FEATURE DEPLOYMENTS?
Assume frontend is a Single Page
Application served by an
container…
Assume all /api/ URLs are handled by
a middleware application…
How do you correlate
frontend-feature-1 with
middleware-feature-1 ?
nginx

View Slide

HACK... CLOUD FOUNDRY BUILDPACKS
HACK... CLOUD FOUNDRY BUILDPACKS
Knowing Cloud Foundry , we can
1. add our own nginx.conf
2. include our own variables in the location blocks, for example:
3. and then use cf set-env and cf restage in our pipelines to correlate the deployments.
N.B. This demo was built before introduction of the nginx buildpack. If you are trying this new, do not
hack the staticfile buildpack. Use the nginx buildpack instead.
See for details.
evaluates embedded ruby before storing the nginx.conf
location /api/ {
proxy_pass <%= ENV["API_ENDPOINT"] %>/api/;
}
https://github.com/cloudfoundry/staticfile-buildpack/issues/103

View Slide

THAT'S NOT MICROSERVICES!
THAT'S NOT MICROSERVICES!
Correct. That is a distributed monolith. Why?!
Not all organizations are microservices-ready.
…
There's some justification for not starting with a microservices architecture until
the domain and the proper contexts are more clear.
—
“
Rebecca Parisons, CTO, ThoughtWorks

View Slide

PROTIP: CI FOR CI
PROTIP: CI FOR CI
How do you test a custom CI/CD stack?
With unit tests? No.
With the stack itself
Create new GitHub organization with repositories per stack or toolchain. Jenkins can autodiscover
repositories and branches within an organization.

View Slide

PROTIP: CI FOR CI
PROTIP: CI FOR CI
Leverage multibranch pipelines to test toolchains and versions.
This demo repository serves as integration testing for most common pipeline stages, e.g. production
builds, deployments, etc.
These integration tests are slower. But they are also easy, low maintenance and meaningful ✌

View Slide

Section Summary
Section Summary
CD & DISTRIBUTED MONOLITHS
CD & DISTRIBUTED MONOLITHS
It's acceptable to start with distributed monoliths and move to Microservices as a goal.
How do you automate CD of distributed monoliths?
1. Assume we correlate deployments via branch names
2. Leverage multibranch pipelines
3. Target branches with when directive and branch property
4. Overwrite app name when deploying
5. Configure communication endpoints dynamically in pipeline*
How do you automate testing of your CI stack?
With the stack itself
Leverage repository autodiscovery for GitHub organizations
Leverage autobuilds in multibranch pipelines
*This demo showed a workaround conceived mid-2017. Since then there is an official nginx buildpack that should be used instead. You can also use a Node.js Express app
to serve the frontend. The same principle of using endpoints set by environment variables applies.

View Slide

THANK YOU
THANK YOU
We're Hiring!
We're Hiring!
Julie Ng
Enterprise Architect
[email protected]

View Slide

EnterJS: DIY Full-Stack JavaScript CI CD

EnterJS: DIY Full-Stack JavaScript CI CD

jng

More Decks by jng

Other Decks in Technology

Featured

Transcript