The Evolution of a Node.js Service

Slide 1

Slide 1 text

The Evolution of a Node.js Service at GoDaddy Gergely Nemeth | nemethgergely.com | @nthgergo

Slide 2

Slide 2 text

Who am I? ● Gergely Nemeth ● Engineering Manager @ GoDaddy ● Software Developer ● Coffee Lover ☕

Slide 3

Slide 3 text

The Evolution of a Node.js Service at GoDaddy Gergely Nemeth | nemethgergely.com | @nthgergo

Slide 4

Slide 4 text

● This talk is not an in-depth Node.js case study ● You won't see a lot of barely readable code snippets ○ just a few :) What this talk is not about?

Slide 5

Slide 5 text

What this talk is about? ● The stories and lessons learned from building a greenfield project in an enterprise setting ● An overview of best practices and tools that we have adopted in the past years ● This talk is for you if: ○ you'd like to be more familiar with building and evolving Node.js services - both from a technical and organizational view, ○ you'd like to build greenfield projects in the near future using Node.js.

Slide 6

Slide 6 text

What this talk is about? A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system. John Gall

Slide 7

Slide 7 text

Background

Slide 8

Slide 8 text

What do we build? ● A personalization engine: ○ exposing a REST API, ○ shipping an admin interface for (mostly) POs, ○ all built on top of a machine learning stack.

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Legacy downstream services ● Running in OpenStack, ● In GoDaddy data centers, ● Some of them only accessible on the internal network.

Slide 11

Slide 11 text

Project goals

Slide 12

Slide 12 text

Goals ● Build a personalization platform that can be used by other teams ● For the initial implementation: ○ Increase user engagement ■ measured by click-through rate of call-to-actions April, 2017 April, 2018 April, 2016

Slide 13

Slide 13 text

Building a proof of concept

Slide 14

Slide 14 text

Proof of concept (1/2) ● disposable piece of code to prove a point ○ most probably won't take into account ■ scalability, ■ security, ■ error handling April, 2017 April, 2018 April, 2016

Slide 15

Slide 15 text

● it may easily get miscommunicated ○ it may set false expectations / tell the wrong story on the status of the project ● as a consequence, it may get incorporated into production systems ● providing the functionality is not enough - it must be a viable application Proof of concept (2/2) April, 2017 April, 2018 April, 2016

Slide 16

Slide 16 text

Building a proof of concept Building tracer code

Slide 17

Slide 17 text

Tracer ammunition (tracers) are bullets that are built with a small pyrotechnic charge in their base. Ignited by the burning powder, the pyrotechnic composition burns very brightly, making the projectile trajectory visible to the naked eye during daylight, and very bright during nighttime. This enables the shooter to make aiming corrections without observing the impact of the rounds fired and without using the sights of the weapon. Tracer code - where does the term come from? April, 2017 April, 2018 April, 2016

Slide 18

Slide 18 text

Tracer code - how does it apply to software? ● the main components built in the early stages of a project will be used in the production environment ○ gives you an opportunity to test architecture, and an idea how difficult adding new functionality will be ● later on, the architecture can be evolved, just like how the tracer bullet might change trajectory ● the tracer code approach requires user stories to be in place, it does not allow exploration April, 2017 April, 2018 April, 2016

Slide 19

Slide 19 text

First look at the service we built

Slide 20

Slide 20 text

First look at the service we built - with a single downstream dependency Card Catalog April, 2017 April, 2018 April, 2016 JSON {}

Slide 21

Slide 21 text

First look at the service we built - with a single downstream dependency Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Domains API

Slide 22

Slide 22 text

Cards

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

First look at the service we built - with a single downstream dependency Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Domains API Contains Card data

Slide 25

Slide 25 text

Graceful shutdowns ● The process of freeing up the resources used by the application before termination like ○ database connections, ○ or file descriptors ● Terminus ○ Adds graceful shutdown and Kubernetes readiness / liveness checks for any HTTP applications https://www.npmjs.com/package/@godaddy/terminus

Slide 26

Slide 26 text

Scaling: card creation

Slide 27

Slide 27 text

April, 2017 April, 2018 April, 2016

Slide 28

Slide 28 text

April, 2017 April, 2018 April, 2016

Slide 29

Slide 29 text

April, 2017 April, 2018 April, 2016

Slide 30

Slide 30 text

Service architecture with Card Inventory Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Domains API Card Schemas (in Ceph) Generated on-demand Card Inventory (GDocs)

Slide 31

Slide 31 text

● error prone ○ easy to edit the wrong row ● no visual feedback on the card edited ● card schemas are generated on demand, not on every change Better, but April, 2018 April, 2017 April, 2018 April, 2016

Slide 32

Slide 32 text

Scaling: card creation, the next iteration

Slide 33

Slide 33 text

Card Authoring Tool April, 2018 April, 2017 April, 2018 April, 2016 ● The need emerged for a CMS-like solution to manage cards ● Requirements ○ Version Card Schemas ○ Browse all the Card Definitions in the system ○ Track Card Definition state: has changes but not published; versus content is same as published content

Slide 34

Slide 34 text

What are PDRs? April, 2018 April, 2017 April, 2018 April, 2016 ● Stands for Pre-Development Review ● They are first and foremost discussions - pull requests where project members collaborate ● We use them to ○ investigate new technologies ○ compare solutions ○ in short: to learn.

Slide 35

Slide 35 text

April, 2018 April, 2017 April, 2018 April, 2016

Slide 36

Slide 36 text

April, 2018 April, 2017 April, 2018 April, 2016

Slide 37

Slide 37 text

April, 2018 April, 2017 April, 2018 April, 2016

Slide 38

Slide 38 text

Service architecture with Card Authoring Tool Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Domains API Card Schemas (in Ceph) Generated on each change Card Definitions (in GHE) Card Authoring Tool

Slide 39

Slide 39 text

Scaling: adding new downstream services

Slide 40

Slide 40 text

Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Domains API Card Schemas (in Ceph) Generated on each change Card Definitions (in GHE) Card Authoring Tool

Slide 41

Slide 41 text

Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Domains API Card Schemas (in Ceph) Generated on each change Card Definitions (in GHE) Card Authoring Tool Domains API Domains API Domains API Domains API

Slide 42

Slide 42 text

Introducing a unified API for partner teams April, 2018 April, 2017 April, 2018 April, 2016 ● Get the same type of data from different downstream services soon became an issue ○ We had to implement custom logic for each new partner team to collect information (sometimes calling multiple endpoints) ● Standardized Entity API ○ Each partner team has to expose an endpoint that returns data in the same format ○ Adding new partner teams became a single line of code change

Slide 43

Slide 43 text

Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Entity Service Card Schemas (in Ceph) Generated on each change Card Definitions (in GHE) Card Authoring Tool All partner APIs

Slide 44

Slide 44 text

Canary deployments ● Canary deployments is a technique to reduce the risk of introducing a new version in production by slowly rolling out the change ○ Capture & monitor metrics, and rollback if needed ○ Reduces user impact of changes https://martinfowler.com/bliki/CanaryRelease.html

Slide 45

Slide 45 text

Node.js 8 got released October 2017

Slide 46

Slide 46 text

Async-await in Node.js ● Since Node.js version 8, you can use async-await function logFetch(url) { return fetch(url) .then(response => response.text()) .then(text => { console.log(text); }).catch(err => { console.error('fetch failed', err); }); } async function logFetch(url) { try { const response = await fetch(url); console.log(await response.text()); } catch (err) { console.log('fetch failed', err); } } April, 2017 April, 2018 April, 2016

Slide 47

Slide 47 text

Async-await in Node.js with Express const boom = require('boom'); const asyncWrap = fn => (req, res, next) => { Promise.resolve(fn(req, res, next)).catch((err) => { if (!err.isBoom) { return next(boom.badImplementation(err)); } next(err); }); }; app.get('/', asyncWrap(async (request, response) => { const result = await getContent(); response.send(result); })); April, 2018 April, 2017 April, 2018 April, 2016

Slide 48

Slide 48 text

npm ci ● similar to npm install, except it's meant to be used in automated environments ○ npm ci can only install entire projects at a time: individual dependencies cannot be added with this command, ○ npm ci will never write to package.json or any of the package-locks: installs are essentially frozen ● as a result, performance is improved, can be 40% faster then npm install or yarn https://blog.npmjs.org/post/171556855892/introducing-npm-ci-for-faster-more-reliable

Slide 49

Slide 49 text

Moving to the public cloud

Slide 50

Slide 50 text

Challenges of moving to AWS April, 2018 April, 2017 April, 2018 April, 2016 ● Some partner APIs are not exposed on the public network ○ We need a way to "talk home" ● Dedicated infrastructure team vs. you build it, you run it

Slide 51

Slide 51 text

AWS Direct Connect or OpenVPN?

Slide 52

Slide 52 text

Talking home April, 2018 April, 2017 April, 2018 April, 2016 ● Two clients running in OpenStack ● Running the OpenVPN Servers in Amazon ECS ● CloudWatch is used to monitor if the OpenVPN server is up and running ○ If not, it triggers a Lambda function which replaces the OpenVPN master with the secondary ○ The old primary will be recovered by ECS and put back in rotation

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Card Catalog April, 2017 April, 2018 April, 2016 JSON {} Entity Service Card Schemas (in S3) Generated on each change Card Definitions (in GHE) Card Authoring Tool All partner APIs Through the OpenVPN bridge

Slide 55

Slide 55 text

You build it, you run it.

Slide 56

Slide 56 text

You build it, you run it April, 2018 April, 2017 April, 2018 April, 2016 ● Encourages ownership and accountability, which leads to more independent and responsible teammates ● Leads to operational excellence

Slide 57

Slide 57 text

Technology Stack April, 2018 April, 2017 April, 2018 April, 2016 ● Kubernetes ● Node.js / Python ● Postgresql ● GitHub Enterprise ● Jenkins

Slide 58

Slide 58 text

Yeah, that's kind of boring.

Slide 59

Slide 59 text

Boring is good for stability.

Slide 60

Slide 60 text

Ark by Heptio ● Ark gives you tools to backup and restore your Kubernetes cluster resources and persistent volumes: ○ Take backups of your cluster and restore in case of loss ○ Copy cluster resources across cloud providers ○ Replicate your production environment for development and testing environments https://github.com/heptio/ark

Slide 61

Slide 61 text

Skaffold https://github.com/GoogleContainerTools/skaffold ● Skaffold is a command line tool that facilitates continuous development for Kubernetes applications ○ Detect changes in your source code and automatically build/push/deploy ○ Support for multiple application components. Build and deploy only the pieces of your stack that have changed ○ Deploy regularly when saving files or run one off deployments using the same configuration April, 2018 April, 2017 April, 2018 April, 2016

Slide 62

Slide 62 text

The future

Slide 63

Slide 63 text

The future ● Moving to AWS Direct Connect ● Moving from a pull-based architecture to a push-based one for Entity Service ○ Partner teams need to push data whenever there is a change / new record ■ Improved latency ■ Simplified data flow

Slide 64

Slide 64 text

That's it folks. Gergely Nemeth | nemethgergely.com | @nthgergo

Slide 65

Slide 65 text

Further reading ● https://nemethgergely.com/using-kubernetes-for-local-development/ ● https://nemethgergely.com/skaffold-and-localstack-to-develop-aws-locally/ ● https://github.com/heptio/ark ● https://docs.npmjs.com/cli/ci