Slide 1

Slide 1 text

Modern Web APIs with Node.js A glance across the border Sven Kölpin [email protected] @dskgry Manuel Rauber [email protected] @manuelrauber

Slide 2

Slide 2 text

Your Speakers Sven Kölpin Does stuff @ OPEN KNOWLEDGE GmbH ! [email protected] " @dskgry Manuel Rauber Consultant @ Thinktecture AG ! [email protected] " @manuelrauber # https://manuel-rauber.com

Slide 3

Slide 3 text

Timetable When What 09:00 – 10:30 Working time 10:30 – 11:00 Break 11:00 – 12:30 Working time 12:30 – 13:30 Lunchbreak 13:30 – 15:00 Working time 15:00 – 15:30 ☕ & " 15:30 – 17:00 Working time

Slide 4

Slide 4 text

Talking points • Web APIs • Cross Platform • The web as a platform • Node.js • Web Sockets • RethinkDB • Testing

Slide 5

Slide 5 text

Web APIs Access all the data!

Slide 6

Slide 6 text

Web APIs

Slide 7

Slide 7 text

Web APIs • Lightweight service-based architecture • Functional services with dedicated interfaces • Use other services, like database or file system • REST does not model real world usecases • (JSON-based) Web APIs • Application push services via WebSocket • SignalR • Socket.io • Prefer HTTPS over HTTP HTTP HTTPS WebSocket Service A Service B Service C Web APIs (ASP.NET, Node.js, …) HTML5-Application (Single-Page Application) Web Cordova Electron

Slide 8

Slide 8 text

Cross-Platform Runs on all the devices!

Slide 9

Slide 9 text

Single- vs. Multi- vs. Cross-Platform macOS Linux Windows iOS Windows Phone Android BlackBerry 10 FireOS Browser TV … Refrigerator

Slide 10

Slide 10 text

Cross-Platform

Slide 11

Slide 11 text

But it doesn’t look like a native application! Exactly.

Slide 12

Slide 12 text

Angular 2: Komponentenbasierte HTML5-Anwendungen • BASTA! Spring 2017 https://spotifyblogcom.files.wordpress.com/2014/12/overview.png

Slide 13

Slide 13 text

Angular 2: Komponentenbasierte HTML5-Anwendungen • BASTA! Spring 2017 http://media.idownloadblog.com/wp-content/uploads/2014/06/Google-Docs-Sheets-Slides-teaser-001.jpg

Slide 14

Slide 14 text

The web as a platform It’s there. Use it!

Slide 15

Slide 15 text

The web as a platform

Slide 16

Slide 16 text

React Single-Page Applications

Slide 17

Slide 17 text

Single-Page Applications • Web application with ONE HTML page • App-Like Behavior (WebApps) • Fast & highly interactive • Real time • Offline capabilities • Mobile Web • Server: Data, not DOM

Slide 18

Slide 18 text

Lifecycle SPA Browser Server (e.g. node) Initial Request HTML / JavaScript /CSS XHR Request JSON Render Page Static files Page running Dynamic resources (HTTP API)

Slide 19

Slide 19 text

Component import React, {Component} from 'react'; class MyView extends Component { render() { return (

Hello World!

); } }

Slide 20

Slide 20 text

Facts • Est. 2013 • Version 16.x • Not a framework

Slide 21

Slide 21 text

Demo Show me, what you got!

Slide 22

Slide 22 text

Node.js Or: How you can execute jQuery on the server. ;-)

Slide 23

Slide 23 text

Intro • Server-side JavaScript powered by Chrome’s V8 JavaScript engine • Asynchronous, event-driven I/O API • Can access system resources (e.g. file system) • Package management via Node Package Manager (npm) oder yarn • Cross platform: Linux, macOS, Windows • Upcoming: Alternative runtime powered by Microsoft’s ChakraCore

Slide 24

Slide 24 text

Features • ECMAScript 2015 – 2017 (V8) • Classes • Typed arrays • Fat arrow/Lambda Expressions • Templated Strings • Promises • Async / Await • …

Slide 25

Slide 25 text

Server Architecture

Slide 26

Slide 26 text

Why does it scale? • Focus on requests • Delegate I/O (event loop) • No blocking threads in code • Less memory/cpu consumption • JSON • Avoid heavy serverside calculations • Clustering • Use the event loop • Native modules

Slide 27

Slide 27 text

Modules • JavaScript-File === Modul (CommonJS) • „Singleton“ • Can use other modules (require) • Exports functions or data (exports) • Node.js offers modules • FileSystem, HTTP, OS,… • 3rd party modules via npm const fs = require('fs'); const os = require('os'); fs.mkdir("/some/folder", (err, folder)=> {}); os.platform(); // win32

Slide 28

Slide 28 text

Custom modules • fileA.js • fileB.js module.exports = { sayHello(){ console.log("Hello IJS 2017"); } }; const moduleA = require("./fileA"); console.log(moduleA.sayHello()); // Hello IJS 2017 console.log(moduleA); // { sayHello: [Function: sayHello] }

Slide 29

Slide 29 text

How about some work… for you? ;-) Modern JavaScript: ECMAScript 2015-2017

Slide 30

Slide 30 text

Let & Const • var === function scope • Hoisting • Mother of all bugs • let & const === block Scope • Prefer const function fn() { if (true) { var x = 1; } console.log(x); // ? } function fn() { if (true) { const x = 1; // or let } console.log(x); // ? }

Slide 31

Slide 31 text

Destructuring: Objects const person = { name : "max", age : 53 }; const name = person.name; const age = person.age //... const {name, age} = person; console.log(name); // max console.log(age); // 53 const {name:firstName} = person; console.log(firstName); // max

Slide 32

Slide 32 text

Destructuring: Arrays var arr = ["a", "b", "c"]; const [d,e,f] = arr; console.log(d, e, f); // a b c const [,, third] = arr; console.log(third); // c const [first] = arr; console.log(first); // a

Slide 33

Slide 33 text

Classes class Person{ static myStaticMethod(){} constructor(fn,ln){ this.firstName = fn; this.lastName = ln; } toString() { return `${this.firstName} ${this.lastName}`; } }

Slide 34

Slide 34 text

Inheritance class Student extends Person { constructor(fn, ln, matrNo) { super(fn, ln); this.matrNo = matrNo; } } let student = new Student("Sven", "Koelpin", "123456");

Slide 35

Slide 35 text

Classes – Watch out • syntactic sugar • super calls • no privates (yet) • no class properties (yet)

Slide 36

Slide 36 text

Arrow functions function say(what) { return what; } const say = what => what; say('hello‘); function add(a, b) { return a + b; } const add = (a, b) => { return a + b; } function Person() { this.age = 0; setInterval(function() { this.age++; // BOOOM }, 1000); } function Person() { this.age = 0; setInterval(()=> { this.age++; // works }, 1000); }

Slide 37

Slide 37 text

Promises • JS === single-threaded à„callback hell“? • a.k.a. „pyramid of doom“ Server.get('tweets', tweets => { Server.get('tweets/1', tweet => { Server.get('tweets/1/likes', likes => { //... }); }); });

Slide 38

Slide 38 text

Promises • Make async operations „thenable“ • They still run within a single thread Server.get('tweets') .then(tweets => Server.get('tweets/1')) .then(tweet => Server.get('tweets/1/likes')) .then(/*...*/) .catch(e => /*...*/)

Slide 39

Slide 39 text

Promises - concurrent • Runs concurrently, but still on a single thread (I/O delegation) Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]) .then(result => { const [tweets, tweet, likes] = result; }) .catch(e = >{ // one request f*cked up });

Slide 40

Slide 40 text

Async + Await • Promises that look synchronous const receiveData = async () => { try { const tweets = await Server.get('tweets'); const tweet = await Server.get('tweets/1'); const likes = await Server.get('tweets/1/likes'); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/);

Slide 41

Slide 41 text

Async + Await - concurrent const receiveData = async () => { try { const [tweets, tweet, likes] = await Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/);

Slide 42

Slide 42 text

Installation

Slide 43

Slide 43 text

Node.js Installation • If not done yet: install Node.js • https://nodejs.org v8+ • IDE • VS Code: https://code.visualstudio.com/ • WebStorm: https://www.jetbrains.com/webstorm/

Slide 44

Slide 44 text

Git Basics • Clone Project • http://bit.ly/ijs-nodejs-workshop • git clone https://github.com/dskgry/nodejs-workshop.git • Switch branches • git checkout -f BRANCHNAME (e.g. git checkout -f 01_basics) • Alternative: Use GitHub Desktop J

Slide 45

Slide 45 text

Install dependencies • Go to ./server • Run from command line: npm install • Start the server: npm start • Go to ./app • Run from command line: npm install • Start the app: npm start

Slide 46

Slide 46 text

Exercise: JavaScript basics • Branch: 01_basics • Folder: basics/esnext • Files: destructuring.js, arrowFunction.js, promises.js, asyncAwait.js • Run a file by typing node ‘filename‘ in a console • E.g. node destructuring • Need help? https://javascript.info/

Slide 47

Slide 47 text

Exercise: Node modules • Branch: 01_basics • Folder: basics/modules • Files: main.js, osinfo.js • Run by typing node main in a console • You need to be in the basics/modules folder • Need help? https://nodejs.org/en/docs/

Slide 48

Slide 48 text

Alternatives • .NET & .NET Core • .NET Core is also platform independent • Entity Framework for database access • Java • JAX-RS / Spring Rest • JPA • Ruby • Python • …

Slide 49

Slide 49 text

Pros • One language to rule them all: Full stack JS‘ish development • Universal/Isomorphic JavaScript: Share code for client & server • Open Source loving community

Slide 50

Slide 50 text

Pros • It scales • Enterprise proven: Netflix, Paypal, Groupon, Walmart, … https://nodejs.org/static/documents/casestudies/Node_CaseStudy_Nasa_FNL.pdf

Slide 51

Slide 51 text

Watch out! • Single threaded Event Loop: Avoid heavy CPU usage • Utilizes one CPU only: Scale via clustering • Relational databases can be strange • Code is (often the only) documentation • Still JavaScript à Use TS or Flow in bigger projects

Slide 52

Slide 52 text

Restify Web APIs done right.

Slide 53

Slide 53 text

Restify • Node.js module to build Web APIs • Middleware support • Client & Server module • Routing • vs Express/Koa.js • No templating • No rendering • Used by Netflix

Slide 54

Slide 54 text

Restify const restify = require('restify'); const server = restify.createServer(); server.get('person', (req, res, next) => { const person = {}; res.send(person); next(); }); server.listen(3001, function () { console.log('server up <3'); });

Slide 55

Slide 55 text

Middleware • Chainable • Run in the order registered • Global Middlewares for every request à Plugins • Middlewares per route • Logging • Validation • Authentication • Caching (Etag-Handlers) Middleware 1 // Server logic next() // more logic Middleware 2 // Server logic next() // more logic Middleware 3 // Server logic // more logic Request Response Client Logging Authentication …

Slide 56

Slide 56 text

Global Middlewares, Plugins • Via .use (after routing) • Via .pre (before routing) const server = restify.createServer(); server.use(restify.plugins.gzipResponse()); server.use(restify.plugins.queryParser()); server.use(restify.plugins.bodyParser()); const server = restify.createServer(); server.pre(restify.plugins.throttle({burst: 10, rate: 10, ip: true}));

Slide 57

Slide 57 text

CORS Plugin • Combines .pre and .use • Supported via official 3rd party plugin by TabDigital const corsMiddleware = require('restify-cors-middleware'); const cors = corsMiddleware({ origin: '*' }); server.pre(cors.preflight); server.use(cors.actual);

Slide 58

Slide 58 text

Custom global Middlewares • Middlewares for all routes // MyMiddleware.js module.exports = (req, res, next) => { // do smart stufff next(); }; // MyServer.js server.pre(myMiddleware); server.use(myMiddleware);

Slide 59

Slide 59 text

Route-based Middlewares server.get( '/foo', myMiddleware, (req, res, next) => { console.log('Authenticate'); next(); }, (req, res, next) => { res.send(200); return next(); // return is optional } );

Slide 60

Slide 60 text

Exercise: Hello restify • Branch: 02_hello_restify • Folder: server/src • Files: index.js, Server.js • Run index.js by typing npm start (from folder server) • Changes are detected automatically • Check if it worked with postman or your browser • Need help? • https://nodejs.org/en/docs/ • http://restify.com/

Slide 61

Slide 61 text

Let’s get real! Building a Web API for a real use case: Your own real time Twitter

Slide 62

Slide 62 text

Project structure • Separation of concerns • Also in JavaScript ;) • Domain oriented „packages“ • Technical „packages“

Slide 63

Slide 63 text

CORS • Problem: Same origin restriction policy • Solution: Cross-origin resource sharing (CORS) • W3C Standard • Server describes (via HTTP-Header) • Which origins are allowed (e.g. http://google.com) • Which methods are allowed (e.g. GET, POST)

Slide 64

Slide 64 text

Exercise: Hello TweetResource • Goal: Receive all tweets via. HTTP-API (GET localhost:3001/tweets) • Use restify cors plugin • Branch: 03_tweetresource • Files: index.js, Server.js, TweetsResource.js • Test with postman • Test with our app (don‘t forget CORS) • npm start • Need help? http://restify.com/

Slide 65

Slide 65 text

Exercise: Hello TweetService • Goal: Create TweetService • Implement getTweets, getTweet, countTweets, createTweet • Let‘s do TDD: Use the TweetServiceTest (npm test) • Add pagination to resource (read query parameters) • Use TweetService in Resource • Use restify queryParser plugin • Branch: 04_tweetservice • Files: TweetService.js, TweetServiceTest.js , TweetsResource.js, Server.js • Test in Postman (e.g. localhost:3001/tweets?page=1&size=1)

Slide 66

Slide 66 text

Validation • Never ever ever trust the client • JS === dynamically typed • 3rd party libs to check object shapes • Yup: • “Dead simple Object schema validation” • Validate objects • Sanitize objects

Slide 67

Slide 67 text

Yup const yup = require('yup'); const schema = yup.object().shape({ name: yup.string().required(), age: yup.number().required().positive().integer(), email: yup.string().email(), website: yup.string().url(), //... }); try { const validated = await schema.isValid({name: 'jimmy', age: 24}); } catch (e) { //... }

Slide 68

Slide 68 text

Validation as middleware server.get('foo', validation.validateQueryParams({ times: yup.number().min(1).max(10).default(1) }), (req, res, next) => {/*...*/} ); Validate query params server.post('foo', validation.validatePostBody({ mail: yup.string().mail().required(), name: yup.string().min(3).max(10) }), (req, res, next) => {/*...*/} ); Validate request payload

Slide 69

Slide 69 text

HTTP POST: An inspirational reminder • Creates a new resource • Server decides where • Location header • Answers with what you‘ve created • Status-Code: 201 (Created) POST /tweets Content-Type: application/json { "tweet“ :“…“ } ------------------------------------------------------------------------- CREATED 201 Location: /tweets/123 { “id“ : 123 "tweet“ :“…“ “author“ : { … } }

Slide 70

Slide 70 text

Exercise: Validation + HTTP POST • Goal: Add POST + „single“-GET resource methods • Validate the queryparams • Validate the request body • Extract a path-param • Branch: 05_post_validation • Files: TweetsResource.js, Validation.js, Server.js • Test in our app + Postman • Need help? • https://github.com/jquense/yup • http://restify.com/

Slide 71

Slide 71 text

Token-based authentication • Exchange credentials for token • Token: • Random string • Self-contained • JWT • Typically send via Authorization header • Don‘t create tokens on your own • Use certified Security Token Services • https://identityserver.github.io • https://github.com/panva/node-oidc-provider

Slide 72

Slide 72 text

Token-based authentication Client Server Username & Password 1. check 2. generate token Token Store token Token Validate & execute

Slide 73

Slide 73 text

Exercise: Middlewares + Security • Goal: Add Logger-Middleware and Security-Middleware • Branch: 06_middleware_security • Files: Logger.js, Security.js, Server.js • Test in our app + Postman • Don‘t forget to set the Authorization header in Postman

Slide 74

Slide 74 text

Real time web • Server pushes data directly to the client • Use Cases • Chat • Stock info / Newsticker • Progress information • Collaboration tools • Legacy • Polling / Long polling

Slide 75

Slide 75 text

Web Sockets • Bi-Directional • Server pushes • Client pushes • Use Cases • Collaboration tools • Chat • New protocol • Persistent connections

Slide 76

Slide 76 text

Web Sockets

Slide 77

Slide 77 text

Server-sent events • Push technology only • HTTP-based • Auto reconnect • Use-Cases • News streaming • Progress bars • But • Less supported

Slide 78

Slide 78 text

Web Sockets & Node === Easy • Usage via libraries • ws • socket.io const webSocket = require('ws'); const wss = new webSocket.Server({server}); wss.on('connection', ws => { const interval = setInterval(() => { const stockData = getStockData(); ws.send(JSON.stringify(stockData)); }, 1000); ws.on('close', () => { clearInterval(interval); }); });

Slide 79

Slide 79 text

EventEmitter • Node === event driven • Sync & Async events • Custom events via EventEmitter-Class • Loose coupling of modules • Synchronous! • Don’t forget to “removeListener” const readable = getReadableStreamSomehow(); readable.on('data', (chunk) => {…}); readable.on('end', () => {…}); const events = require('events'); const eventEmitter = new events.EventEmitter(); eventEmitter.addListener('myEvent', data => console.log(data)); //... somewhere else eventEmitter.emit('myEvent', someData);

Slide 80

Slide 80 text

Exercise: Server push • Goal: Add websocket support • Use EventEmitter • emit event when tweet created • consume event and send via Web Socket • Branch: 07_sockets • Files: TweetService.js, Server.js • Test in our app • Open in two browser windows and tweet something J • Need help? https://github.com/websockets/ws

Slide 81

Slide 81 text

Databases Every business application needs a database, right?

Slide 82

Slide 82 text

Databases • Plays well with different kind of databases • NoSQL databases like MongoDB, CouchDB • Relational databases like PostgresSQL, MSSQL • Real time databases like RethinkDB • Access databases directly or via ORM • SequelizeJS • TypeORM

Slide 83

Slide 83 text

RethinkDB • Open Source, scalable JSON real time database • Can push updated query results directly to the application • Use Cases are all real time based applications • Multiplayer games • Streaming analytics • Collaborative tools • Don’t use it, if you need full ACID or strong schemas

Slide 84

Slide 84 text

RethinkDB – API • RethinkDB simple queries • All queries return promises! const r = require('rethinkdb'); await r.table('MY_TABLE').get('1').run(connection); // get by id await r.table('MY_TABLE').count().run(connection); // count stuff

Slide 85

Slide 85 text

Exercise: Async-Programing + RethinkDB • Goal: Add RethinkDB support • Do some async programming • Use RethinkDB-API • Branch: 08_database • Files: index.js, TweetServicejs, TweetResource.js • Test in our app • Data will survive server restart J • Automatic real time support • Need help? https://rethinkdb.com/docs/nodejs/

Slide 86

Slide 86 text

Testing Does it work? Really?

Slide 87

Slide 87 text

Testing APIs • E2E-Tests • Slow & too much effort • Integration tests • Test API only • Mock services / Database • Check: • Status codes • Response (JSON) • Validation

Slide 88

Slide 88 text

SuperTest describe(’MyResource', () => { beforeEach(() => { someService.doStuff.mockImplementation(() => ({ id: 1, name: 'test'})); }); it(’a get request', async done => { const response = await supertest(server.getServer()).get('/somepath'); expect(response.status).toBe(200); done(); // tell SuperTest we’re done }); });

Slide 89

Slide 89 text

Exercise: Test our API • Goal: Test our API • See some mocking • Test responses (status codes and body) • Branch: 09_apitest • Files: TweetResourceTest.js • npm test • Need help? https://github.com/visionmedia/supertest

Slide 90

Slide 90 text

But wait: There’s more!

Slide 91

Slide 91 text

A public API needs more… • Reduce network traffic • Caching • GZIP • HATEOAS • Location-Header for POST-Methods • Link-Header for pagination • Rate limiting, Throttling • Versioning

Slide 92

Slide 92 text

Reduce traffic • Tell client that data did not change • Code 304 • Two standardized options: • Last Modified-Header • Did data change since… • Etag-Header • Did checksum of data change? GET /tweets OK 200 Etag: 7982882299 GET /tweets If-None-Match: 7982882299 304: NOT MODIFIED following request initial request

Slide 93

Slide 93 text

Use GZIP • Supported by all modern browsers • Supported by most servers • Reduces traffic by up to 70% • But • more computation • is not always smaller

Slide 94

Slide 94 text

HATEOAS • Hypermedia as the engine of application state • Navigate from anywhere to everywhere • Usage of API without prior knowledge

Slide 95

Slide 95 text

HATEOAS • Example: POST • Example: Pagination POST /tweets {… ------------------------------------------------------------------------- CREATED 201 Location: http://twttr.de/tweets/123 {…} GET /tweets/?page=1&size=10 HEADER Link: ; rel="next”>, ; rel="last”> BODY [{…},{…}]

Slide 96

Slide 96 text

Rate limiting (Throttling) • Public APIs • DDoS protection • Pay per request APIs • Rates for „expensive“ resources • Rates per timeframe • Hour, Minute, Second • How? • IP-based • Account-based

Slide 97

Slide 97 text

Versioning • via URL (twttr.com/v1/tweets) • via Accept-Header • via Custom-Header • Restify (Accept-Version) GET http:twttr.com/tweets/ Accept-Version: 1.0.0 200 OK [{tweet:”…”},{tweet:”…”}] GET http:twttr.com/tweets/ Accept-Version: 3.0.0 406 NOT ACCEPTABLE const server = restify.createServer({ name: 'Twttr', version: '1.0.0' });

Slide 98

Slide 98 text

Exercise: API Restify • Optimize our HTTP API • Add Versioning, Throttling and GZIP support • Set the Accept-Version header in postman to test this • Add response headers (Location for POST, Link for pagination) • Implement conditional requests with ETAG (for single tweet retrieval) • Set the If-None-Match header in postman to test this • Branch: 10_api_restify • Files • Server.js, TweetResource.js, HttpHelper.js

Slide 99

Slide 99 text

Summary

Slide 100

Slide 100 text

Summary • Modern SPAs needs modern backends • A backend can be orchestrated with several technologies/languages • Node.js is one more possibility to write Web APIs • Web APIs should be use case centric – not pure REST • Leverage existing Node.js modules for kickstarting Web APIs • Restify sets up a base for writing Web APIs • Make usage of real time Web Sockets communication • RethinkDB is one possibility for databases in Node.js • Don’t forget gzip, versioning, rate limiting and security

Slide 101

Slide 101 text

Thank you! Questions? Repository https://github.com/dskgry/nodejs-workshop Sven Kölpin [email protected] @dskgry Manuel Rauber [email protected] @manuelrauber