Moderne Web APIs mit Node.js – volle Power im Backend

Moderne Web APIs mit Node.js
Volle Power im Backend
Sven Kölpin
[email protected]
@dskgry
Manuel Rauber
[email protected]
@manuelrauber

View Slide

Your Speakers
Sven Kölpin
Does stuff @ OPEN KNOWLEDGE GmbH
! [email protected]
" @dskgry
Manuel Rauber
Consultant @ Thinktecture AG
! [email protected]
" @manuelrauber
# https://manuel-rauber.com

View Slide

Timetable
When What
09:00 – 10:30 Working time
10:30 – 11:00 Break
12:30 – 13:30 Lunchbreak
15:00 – 15:30 ☕ & "

View Slide

Talking points
• Web APIs
• Cross Platform
• The web as a platform
• Node.js
• Web Sockets
• RethinkDB
• Testing

View Slide

Web APIs
Access all the data!

View Slide

Web APIs

View Slide

Web APIs
• Lightweight service-based architecture
• Functional services with dedicated interfaces
• Use other services, like database or file system
• REST does not model real world usecases
• (JSON-based) Web APIs
• Application push services via WebSocket
• SignalR
• Socket.io
• Prefer HTTPS over HTTP
HTTP HTTPS WebSocket
Service A Service B Service C
Web APIs
(ASP.NET, Node.js, …)
HTML5-Application
(Single-Page Application)
Web Cordova Electron

View Slide

Cross-Platform
Runs on all the devices!

View Slide

Single- vs. Multi- vs. Cross-Platform
macOS
Linux
Windows iOS
Windows Phone
Android
BlackBerry 10
FireOS
Browser
TV
…
Refrigerator

View Slide

Cross-Platform

View Slide

But it doesn’t look like a
native application!
Exactly.

View Slide

Angular 2: Komponentenbasierte HTML5-Anwendungen
• BASTA! Spring 2017
https://spotifyblogcom.files.wordpress.com/2014/12/overview.png

View Slide

Angular 2: Komponentenbasierte HTML5-Anwendungen
• BASTA! Spring 2017
http://media.idownloadblog.com/wp-content/uploads/2014/06/Google-Docs-Sheets-Slides-teaser-001.jpg

View Slide

The web as a platform
It’s there. Use it!

View Slide

The web as a platform

View Slide

React
Single-Page Applications

View Slide

Single-Page Applications
• Web application with ONE HTML page
• App-Like Behavior (WebApps)
• Fast & highly interactive
• Real time
• Offline capabilities
• Mobile Web
• Server: Data, not DOM

View Slide

Lifecycle SPA
Browser Server (e.g. node)
Initial Request
HTML / JavaScript /CSS
XHR Request
JSON
Render Page
Static files
Page running
Dynamic resources
(HTTP API)

View Slide

Component
import React, {Component} from 'react';
class MyView extends Component {
render() {
return (

Hello World! h1 >

);
}
}

View Slide

Facts
• Est. 2013
• Version 16.x
• Not a framework

View Slide

Demo
Show me, what you got!

View Slide

Node.js
Or: How you can execute jQuery on the server. ;-)

View Slide

Intro
• Server-side JavaScript powered by Chrome’s V8 JavaScript engine
• Asynchronous, event-driven I/O API
• Can access system resources (e.g. file system)
• Package management via Node Package Manager (npm) oder yarn
• Cross platform: Linux, macOS, Windows
• Upcoming: Alternative runtime powered by Microsoft’s ChakraCore

View Slide

Features
• ECMAScript 2015 – 2017 (V8)
• Classes
• Typed arrays
• Fat arrow/Lambda Expressions
• Templated Strings
• Promises
• Async / Await
• …

View Slide

Server Architecture

View Slide

Why does it scale?
• Focus on requests
• Delegate I/O (event loop)
• No blocking threads in code
• Less memory/cpu consumption
• JSON
• Avoid heavy serverside calculations
• Clustering
• Use the event loop
• Native modules

View Slide

Modules
• JavaScript-File === Modul (CommonJS)
• „Singleton“
• Can use other modules (require)
• Exports functions or data (exports)
• Node.js offers modules
• FileSystem, HTTP, OS,…
• 3rd party modules via npm
const fs = require('fs');
const os = require('os');
fs.mkdir("/some/folder", (err, folder)=> {});
os.platform(); // win32

View Slide

Custom modules
• fileA.js
• fileB.js
module.exports = {
sayHello(){
console.log("Hello IJS 2017");
}
};
const moduleA = require("./fileA");
console.log(moduleA.sayHello()); // Hello IJS 2017
console.log(moduleA); // { sayHello: [Function: sayHello] }

View Slide

How about some work… for you? ;-)
Modern JavaScript: ECMAScript 2015-2017

View Slide

Let & Const
• var === function scope
• Hoisting
• Mother of all bugs
• let & const === block Scope
• Prefer const
function fn() {
if (true) {
var x = 1;
}
console.log(x); // ?
}
function fn() {
if (true) {
const x = 1; // or let
}
console.log(x); // ?
}

View Slide

Destructuring: Objects
const person = {
name : "max",
age : 53
};
const name = person.name;
const age = person.age;
//...
const {name, age} = person;
console.log(name); // max
console.log(age); // 53
const {name:firstName} = person;
console.log(firstName); // max

View Slide

Destructuring: Arrays
const arr = ["a", "b", "c"];
const [d,e,f] = arr;
console.log(d, e, f); // a b c
const [,, third] = arr;
console.log(third); // c
const [first] = arr;
console.log(first); // a

View Slide

Classes
class Person{
static myStaticMethod(){}
constructor(fn,ln){
this.firstName = fn;
this.lastName = ln;
}
toString() {
return `${this.firstName} ${this.lastName}`;
}
}

View Slide

Inheritance
class Student extends Person {
constructor(fn, ln, matrNo) {
super(fn, ln);
this.matrNo = matrNo;
}
}
let student = new Student("Sven", "Koelpin", "123456");

View Slide

Classes – Watch out
• syntactic sugar
• super calls
• no privates (yet)
• no class properties (yet)

View Slide

Arrow functions
function say(what) {
return what;
}
const say = what => what;
say('hello‘);
function add(a, b) {
return a + b;
}
const add = (a, b) => {
return a + b;
}
function Person() {
this.age = 0;
setInterval(function() {
this.age++; // BOOOM
}, 1000);
}
function Person() {
this.age = 0;
setInterval(()=> {
this.age++; // works
}, 1000);
}

View Slide

Promises
• JS === single-threaded à„callback hell“?
• a.k.a. „pyramid of doom“
Server.get('tweets', tweets => {
Server.get('tweets/1', tweet => {
Server.get('tweets/1/likes', likes => {
//...
});
});
});

View Slide

Promises
• Make async operations „thenable“
• They still run within a single thread
Server.get('tweets')
.then(tweets => Server.get('tweets/1'))
.then(tweet => Server.get('tweets/1/likes'))
.then(/*...*/)
.catch(e => /*...*/)

View Slide

Promises - concurrent
• Runs concurrently, but still on a single thread (I/O delegation)
Promise.all([
Server.get('tweets'),
Server.get('tweets/1'),
Server.get('tweets/1/likes')
])
.then(result => {
const [tweets, tweet, likes] = result;
})
.catch(e = >{
// one request f*cked up
});

View Slide

Async + Await
• Promises that look synchronous
const receiveData = async () => {
try {
const tweets = await Server.get('tweets');
const tweet = await Server.get('tweets/1');
const likes = await Server.get('tweets/1/likes');
} catch (e) {
// ...
}
};
receiveData() // this is a promise
.then(/*...*/);

View Slide

Async + Await - concurrent
const receiveData = async () => {
try {
const [tweets, tweet, likes] = await Promise.all([
Server.get('tweets'),
Server.get('tweets/1'),
Server.get('tweets/1/likes')
]);
} catch (e) {
// ...
}
};
receiveData() // this is a promise
.then(/*...*/);

View Slide

Installation

View Slide

Node.js Installation
• If not done yet: install Node.js
• https://nodejs.org v8+
• IDE
• VS Code: https://code.visualstudio.com/
• WebStorm: https://www.jetbrains.com/webstorm/
• RethinkDB: https://www.rethinkdb.com
• Git-Client (bspw. SourceTree, GitHub Desktop, Command-line, ...)

View Slide

Git Basics
• Clone Project
• http://bit.ly/nodejs-repo
• git clone https://github.com/dskgry/nodejs-workshop.git
• Switch branches
• git checkout -f BRANCHNAME (e.g. git checkout -f 01_basics)
• Alternative: Use GitHub Desktop J

View Slide

Install dependencies
• Go to ./server
• Run from command line: npm install
• Start the server: npm start
• Go to ./app
• Run from command line: npm install
• Start the app: npm start

View Slide

Exercise: JavaScript basics
• Branch: 01_basics
• Folder: basics/esnext
• Files: destructuring.js, arrowFunction.js, promises.js, asyncAwait.js
• Run a file by typing node ‘filename‘ in a console
• E.g. node destructuring
• Need help? https://javascript.info/
• All exercices are within the provided JavaScript files

View Slide

Exercise: Node modules
• Branch: 01_basics
• Folder: basics/modules
• Files: main.js, osinfo.js
• Run by typing node main in a console
• You need to be in the basics/modules folder
• Need help? https://nodejs.org/en/docs/

View Slide

Alternatives
• .NET & .NET Core
• .NET Core is also platform independent
• Entity Framework for database access
• Java
• JAX-RS / Spring Rest
• JPA
• Ruby
• Python
• …

View Slide

Pros
• One language to rule them all: Full stack JS‘ish development
• Universal/Isomorphic JavaScript: Share code for client & server
• Open Source loving community

View Slide

Pros
• It scales
• Enterprise proven: Netflix, Paypal, Groupon, Walmart, …
https://nodejs.org/static/documents/casestudies/Node_CaseStudy_Nasa_FNL.pdf

View Slide

Watch out!
• Single threaded Event Loop: Avoid heavy CPU usage
• Utilizes one CPU only: Scale via clustering
• Relational databases can be strange
• Code is (often the only) documentation
• Still JavaScript à Use TS or Flow in bigger projects

View Slide

Restify
Web APIs done right.

View Slide

Restify
• Node.js module to build Web APIs
• Middleware support
• Client & Server module
• Routing
• vs Express/Koa.js
• No templating
• No rendering
• Used by Netflix

View Slide

Restify
const restify = require('restify');
const server = restify.createServer();
server.get('person', (req, res, next) => {
const person = {};
res.send(person);
next();
});
server.listen(3001, () => {
console.log('server up <3');
});

View Slide

Middleware
• Chainable
• Run in the order registered
• Global Middlewares for every
request à Plugins
• Middlewares per route
• Logging
• Validation
• Authentication
• Caching (Etag-Handlers)
Middleware 1
// Server logic
next()
// more logic
Middleware 2
// Server logic
next()
// more logic
Middleware 3
// Server logic
// more logic
Request
Response
Client
Logging Authentication …

View Slide

Global Middlewares, Plugins
• Via .use (after routing)
• Via .pre (before routing)
server.use(restify.plugins.gzipResponse());
server.use(restify.plugins.queryParser());
server.use(restify.plugins.bodyParser());
server.pre(restify.plugins.throttle({burst: 10, rate: 10, ip: true}));

View Slide

CORS Plugin
• Combines .pre and .use
• Supported via official 3rd party plugin by TabDigital
const corsMiddleware = require('restify-cors-middleware');
const cors = corsMiddleware({ origin: '*' });
server.pre(cors.preflight);
server.use(cors.actual);

View Slide

Custom global Middlewares
• Middlewares for all routes
// MyMiddleware.js
module.exports = (req, res, next) => {
// do smart stuff
next();
};
// MyServer.js
server.pre(myMiddleware);
server.use(myMiddleware);

View Slide

Route-based Middlewares
server.get(
'/foo',
myMiddleware,
(req, res, next) => {
console.log('Authenticate');
next();
},
(req, res, next) => {
res.send(200);
return next(); // return is optional
}
);

View Slide

Exercise: Hello restify
• Branch: 02_hello_restify (git checkout -f 02_hello_restify)
• Folder: server/src
• Files: index.js, Server.js
• Run index.js by typing npm start (from folder server)
• Changes are detected automatically
• Check if it worked with postman or your browser
• https://www.getpostman.com/
• Need help?
• https://nodejs.org/en/docs/
• http://restify.com/

View Slide

Let’s get real!
Building a Web API for a real use case: Your own real time Twitter

View Slide

Project structure
• Separation of concerns
• Also in JavaScript ;)
• Domain oriented „packages“
• Technical „packages“

View Slide

CORS
• Problem: Same origin restriction policy
• Solution: Cross-origin resource sharing (CORS)
• W3C Standard
• Server describes (via HTTP-Header)
• Which origins are allowed (e.g. http://google.com)
• Which methods are allowed (e.g. GET, POST)

View Slide

Exercise: Hello TweetResource
• Goal: Receive all tweets via. HTTP-API (GET localhost:3001/tweets)
• Use restify cors plugin
• Branch: 03_tweetresource
• Files: index.js, Server.js, TweetsResource.js
• Test with postman
• Test with our app (don‘t forget CORS)
• npm start
• Need help? http://restify.com/

View Slide

Exercise: Hello TweetService
• Goal: Create TweetService
• Implement getTweets, getTweet, countTweets, createTweet
• Let‘s do TDD: Use the TweetServiceTest (npm test)
• Add pagination to resource (read query parameters)
• Use TweetService in Resource
• Use restify queryParser plugin
• Branch: 04_tweetservice
• Files: TweetService.js, TweetServiceTest.js , TweetsResource.js,
Server.js
• Test in Postman (e.g. localhost:3001/tweets?page=1&size=1)

View Slide

Validation
• Never ever ever trust the client
• JS === dynamically typed
• 3rd party libs to check object shapes
• Yup:
• “Dead simple Object schema validation”
• Validate objects
• Sanitize objects

View Slide

Yup
const yup = require('yup');
const schema = yup.object().shape({
name: yup.string().required(),
age: yup.number().required().positive().integer(). strict(),
email: yup.string().email(),
website: yup.string().url(),
//...
});
try {
const validated = await schema.isValid({name: 'jimmy', age: 24});
} catch (e) {
//...
}

View Slide

Validation as middleware
server.get('foo',
validation.validateQueryParams({
times: yup.number().min(1).max(10).default(1)
}),
(req, res, next) => {/*...*/}
);
Validate query params
server.post('foo',
validation.validatePostBody({
mail: yup.string().mail().required(),
name: yup.string().min(3).max(10)
}),
(req, res, next) => {/*...*/}
);
Validate request payload

View Slide

HTTP POST: An inspirational reminder
• Creates a new resource
• Server decides where
• Location header
• Answers with what you‘ve created
• Status-Code: 201 (Created)
POST /tweets
Content-Type: application/json
{
"tweet“ :“…“
}
-------------------------------------------------------------------------
CREATED 201
Location: /tweets/123
{
“id“ : 123
"tweet“ :“…“
“author“ : {
…
}
}

View Slide

Exercise: Validation + HTTP POST
• Goal: Add POST + „single“-GET resource methods
• Validate the queryparams
• Validate the request body
• Extract a path-param
• Branch: 05_post_validation
• Files: TweetsResource.js, Validation.js, Server.js
• Test in our app + Postman
• Need help?
• https://github.com/jquense/yup
• http://restify.com/

View Slide

Token-based authentication
• Exchange credentials for token
• Token:
• Random string
• Self-contained
• JWT
• Typically send via Authorization header
• Don‘t create tokens on your own
• Use certified Security Token Services
• https://identityserver.github.io
• https://github.com/panva/node-oidc-provider

View Slide

Token-based authentication
Client Server
Username & Password
1. check
2. generate token
Token
Store token Token
Validate & execute

View Slide

Exercise: Middlewares + Security
• Goal: Add Logger-Middleware and Security-Middleware
• Branch: 06_middleware_security
• Files: Logger.js, Security.js, Server.js
• Test in our app + Postman
• Don‘t forget to set the Authorization header in Postman

View Slide

Real time web
• Server pushes data directly to the client
• Use Cases
• Chat
• Stock info / Newsticker
• Progress information
• Collaboration tools
• Legacy
• Polling / Long polling

View Slide

Web Sockets
• Bi-Directional
• Server pushes
• Client pushes
• Use Cases
• Collaboration tools
• Chat
• New protocol
• Persistent connections

View Slide

Web Sockets

View Slide

Web Sockets & Node === Easy
• Usage via libraries
• ws
• socket.io
const webSocket = require('ws');
const wss = new webSocket.Server({server});
wss.on('connection', ws => {
const interval = setInterval(() => {
const stockData = getStockData();
ws.send(JSON.stringify(stockData));
}, 1000);
ws.on('close', () => {
clearInterval(interval);
});
});

View Slide

EventEmitter
• Node === event driven
• Sync & Async events
• Custom events via EventEmitter-Class
• Loose coupling of modules
• Synchronous!
• Don’t forget to “removeListener”
const readable = getReadableStreamSomehow();
readable.on('data', (chunk) => {…});
readable.on('end', () => {…});
const events = require('events');
const eventEmitter = new events.EventEmitter();
eventEmitter.addListener('myEvent', data =>
console.log(data));
//... somewhere else
eventEmitter.emit('myEvent', someData);

View Slide

Exercise: Server push
• Goal: Add websocket support
• Use EventEmitter
• emit event when tweet created
• consume event and send via Web Socket
• Branch: 07_sockets
• Files: TweetService.js, Server.js
• Test in our app
• Open in two browser windows and tweet something J
• Need help? https://github.com/websockets/ws

View Slide

Databases
Every business application needs a database, right?

View Slide

Databases
• Plays well with different kind of databases
• NoSQL databases like MongoDB, CouchDB
• Relational databases like PostgresSQL, MSSQL
• Real time databases like RethinkDB
• Access databases directly or via ORM
• SequelizeJS
• TypeORM

View Slide

RethinkDB
• Open Source, scalable JSON real time database
• Can push updated query results directly to the application
• Use Cases are all real time based applications
• Multiplayer games
• Streaming analytics
• Collaborative tools
• Don’t use it, if you need full ACID or strong schemas

View Slide

RethinkDB – API
• RethinkDB simple queries
• All queries return promises!
const r = require('rethinkdb');
await r.table('MY_TABLE').get('1').run(connection); // get by id
await r.table('MY_TABLE').count().run(connection); // count stuff

View Slide

Exercise: Async-Programing + RethinkDB
• Goal: Add RethinkDB support
• Do some async programming
• Use RethinkDB-API
• Branch: 08_database
• Files: index.js, TweetServicejs, TweetResource.js
• Test in our app
• Data will survive server restart J
• Automatic real time support
• Need help? https://rethinkdb.com/docs/nodejs/

View Slide

Testing
Does it work? Really?

View Slide

Testing APIs
• E2E-Tests
• Slow & too much effort
• Integration tests
• Test API only
• Mock services / Database
• Check:
• Status codes
• Response (JSON)
• Validation

View Slide

SuperTest
describe(’MyResource', () => {
beforeEach(() => {
someService.doStuff.mockImplementation(() => ({ id: 1, name: 'test'}));
});
it(’a get request', async done => {
const response = await supertest(server.getServer()).get('/somepath');
expect(response.status).toBe(200);
done(); // tell SuperTest we’re done
});
});

View Slide

Exercise: Test our API
• Goal: Test our API
• See some mocking
• Test responses (status codes and body)
• Branch: 09_apitest
• Files: TweetResourceTest.js
• npm test
• Need help? https://github.com/visionmedia/supertest

View Slide

But wait: There’s more!

View Slide

A public API needs more…
• Reduce network traffic
• Caching
• GZIP
• HATEOAS
• Location-Header for POST-Methods
• Link-Header for pagination
• Rate limiting, Throttling
• Versioning

View Slide

Reduce traffic
• Tell client that data did not change
• Code 304
• Two standardized options:
• Last Modified-Header
• Did data change since…
• Etag-Header
• Did checksum of data change?
GET /tweets
OK 200
Etag: 7982882299
GET /tweets
If-None-Match: 7982882299
304: NOT MODIFIED
following request
initial request

View Slide

Use GZIP
• Supported by all modern browsers
• Supported by most servers
• Reduces traffic by up to 70%
• But
• more computation
• is not always smaller

View Slide

HATEOAS
• Hypermedia as the engine of application state
• Navigate from anywhere to everywhere
• Usage of API without prior knowledge

View Slide

HATEOAS
• Example: POST
• Example: Pagination
POST /tweets
{…
-------------------------------------------------------------------------
CREATED 201
Location: http://twttr.de/tweets/123
{…}
GET /tweets/?page=1&size=10
HEADER
Link:
; rel="next”>,
; rel="last”>
BODY
[{…},{…}]

View Slide

Rate limiting (Throttling)
• Public APIs
• DDoS protection
• Pay per request APIs
• Rates for „expensive“ resources
• Rates per timeframe
• Hour, Minute, Second
• How?
• IP-based
• Account-based

View Slide

Versioning
• via URL (twttr.com/v1/tweets)
• via Accept-Header
• via Custom-Header
• Restify (Accept-Version)
GET http:twttr.com/tweets/
Accept-Version: 1.0.0
200 OK
[{tweet:”…”},{tweet:”…”}]
GET http:twttr.com/tweets/
Accept-Version: 3.0.0
406 NOT ACCEPTABLE
const server = restify.createServer({
name: 'Twttr',
version: '1.0.0'
});

View Slide

Exercise: API Restify
• Optimize our HTTP API
• Add Versioning, Throttling and GZIP support
• Set the Accept-Version header in postman to test this
• Add response headers (Location for POST, Link for pagination)
• Implement conditional requests with ETAG (for single tweet retrieval)
• Set the If-None-Match header in postman to test this
• Branch: 10_api_restify
• Files
• Server.js, TweetResource.js, HttpHelper.js

View Slide

Summary

View Slide

Summary
• Modern SPAs needs modern backends
• A backend can be orchestrated with several technologies/languages
• Node.js is one more possibility to write Web APIs
• Web APIs should be use case centric – not pure REST
• Leverage existing Node.js modules for kickstarting Web APIs
• Restify sets up a base for writing Web APIs
• Make usage of real time Web Sockets communication
• RethinkDB is one possibility for databases in Node.js
• Don’t forget gzip, versioning, rate limiting and security

View Slide

Thank you! Questions?
Repository
https://github.com/dskgry/nodejs-workshop
Sven Kölpin
[email protected]
@dskgry
Manuel Rauber
[email protected]
@manuelrauber

View Slide

Moderne Web APIs mit Node.js – volle Power im Backend

Moderne Web APIs mit Node.js – volle Power im Backend

Manuel Rauber

More Decks by Manuel Rauber

Other Decks in Programming

Featured

Transcript