Share
Embed
Modern Web APIs with Node.js
Companies like Netflix, PayPal and even NASA have shown that with the help of Node.js, JavaScript can be used to implement scalable server-side solutions. The asynchronous and non-blocking nature of the language makes it easy to create modern and fast web applications with real-time features, while keeping a small overall footprint. Sven Kölpin and Manuel Rauber will introduce you to this brave new world. They will use a demo application to show you how to create modern web APIs with Node.js. Topics like API design, real-time communication, security, database access and single-page applications will be covered within the workshop.
GitHub: https://github.com/dskgry/nodejs-workshop
Manuel Rauber
October 23, 2017
Transcript
101 slides
-
Modern Web APIs with Node.js A glance across the border
Sven Kölpin [email protected] @dskgry Manuel Rauber [email protected] @manuelrauber -
Your Speakers Sven Kölpin Does stuff @ OPEN KNOWLEDGE GmbH
! [email protected] " @dskgry Manuel Rauber Consultant @ Thinktecture AG ! [email protected] " @manuelrauber # https://manuel-rauber.com -
Timetable When What 09:00 – 10:30 Working time 10:30 –
11:00 Break 11:00 – 12:30 Working time 12:30 – 13:30 Lunchbreak 13:30 – 15:00 Working time 15:00 – 15:30 ☕ & " 15:30 – 17:00 Working time -
Talking points • Web APIs • Cross Platform • The
web as a platform • Node.js • Web Sockets • RethinkDB • Testing -
Web APIs • Lightweight service-based architecture • Functional services with
dedicated interfaces • Use other services, like database or file system • REST does not model real world usecases • (JSON-based) Web APIs • Application push services via WebSocket • SignalR • Socket.io • Prefer HTTPS over HTTP HTTP HTTPS WebSocket Service A Service B Service C Web APIs (ASP.NET, Node.js, …) HTML5-Application (Single-Page Application) Web Cordova Electron -
Single- vs. Multi- vs. Cross-Platform macOS Linux Windows iOS Windows
Phone Android BlackBerry 10 FireOS Browser TV … Refrigerator -
Single-Page Applications • Web application with ONE HTML page •
App-Like Behavior (WebApps) • Fast & highly interactive • Real time • Offline capabilities • Mobile Web • Server: Data, not DOM -
Lifecycle SPA Browser Server (e.g. node) Initial Request HTML /
JavaScript /CSS XHR Request JSON Render Page Static files Page running Dynamic resources (HTTP API) -
Component import React, {Component} from 'react'; class MyView extends Component
{ render() { return ( <div className='container'> <h1>Hello World!</ h1 > </div> ); } } -
Facts • Est. 2013 • Version 16.x • Not a
framework -
Intro • Server-side JavaScript powered by Chrome’s V8 JavaScript engine
• Asynchronous, event-driven I/O API • Can access system resources (e.g. file system) • Package management via Node Package Manager (npm) oder yarn • Cross platform: Linux, macOS, Windows • Upcoming: Alternative runtime powered by Microsoft’s ChakraCore -
Features • ECMAScript 2015 – 2017 (V8) • Classes •
Typed arrays • Fat arrow/Lambda Expressions • Templated Strings • Promises • Async / Await • … -
Why does it scale? • Focus on requests • Delegate
I/O (event loop) • No blocking threads in code • Less memory/cpu consumption • JSON • Avoid heavy serverside calculations • Clustering • Use the event loop • Native modules -
Modules • JavaScript-File === Modul (CommonJS) • „Singleton“ • Can
use other modules (require) • Exports functions or data (exports) • Node.js offers modules • FileSystem, HTTP, OS,… • 3rd party modules via npm const fs = require('fs'); const os = require('os'); fs.mkdir("/some/folder", (err, folder)=> {}); os.platform(); // win32 -
Custom modules • fileA.js • fileB.js module.exports = { sayHello(){
console.log("Hello IJS 2017"); } }; const moduleA = require("./fileA"); console.log(moduleA.sayHello()); // Hello IJS 2017 console.log(moduleA); // { sayHello: [Function: sayHello] } -
Let & Const • var === function scope • Hoisting
• Mother of all bugs • let & const === block Scope • Prefer const function fn() { if (true) { var x = 1; } console.log(x); // ? } function fn() { if (true) { const x = 1; // or let } console.log(x); // ? } -
Destructuring: Objects const person = { name : "max", age
: 53 }; const name = person.name; const age = person.age //... const {name, age} = person; console.log(name); // max console.log(age); // 53 const {name:firstName} = person; console.log(firstName); // max -
Destructuring: Arrays var arr = ["a", "b", "c"]; const [d,e,f]
= arr; console.log(d, e, f); // a b c const [,, third] = arr; console.log(third); // c const [first] = arr; console.log(first); // a -
Classes class Person{ static myStaticMethod(){} constructor(fn,ln){ this.firstName = fn; this.lastName
= ln; } toString() { return `${this.firstName} ${this.lastName}`; } } -
Inheritance class Student extends Person { constructor(fn, ln, matrNo) {
super(fn, ln); this.matrNo = matrNo; } } let student = new Student("Sven", "Koelpin", "123456"); -
Classes – Watch out • syntactic sugar • super calls
• no privates (yet) • no class properties (yet) -
Arrow functions function say(what) { return what; } const say
= what => what; say('hello‘); function add(a, b) { return a + b; } const add = (a, b) => { return a + b; } function Person() { this.age = 0; setInterval(function() { this.age++; // BOOOM }, 1000); } function Person() { this.age = 0; setInterval(()=> { this.age++; // works }, 1000); } -
Promises • JS === single-threaded à„callback hell“? • a.k.a. „pyramid
of doom“ Server.get('tweets', tweets => { Server.get('tweets/1', tweet => { Server.get('tweets/1/likes', likes => { //... }); }); }); -
Promises • Make async operations „thenable“ • They still run
within a single thread Server.get('tweets') .then(tweets => Server.get('tweets/1')) .then(tweet => Server.get('tweets/1/likes')) .then(/*...*/) .catch(e => /*...*/) -
Promises - concurrent • Runs concurrently, but still on a
single thread (I/O delegation) Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]) .then(result => { const [tweets, tweet, likes] = result; }) .catch(e = >{ // one request f*cked up }); -
Async + Await • Promises that look synchronous const receiveData
= async () => { try { const tweets = await Server.get('tweets'); const tweet = await Server.get('tweets/1'); const likes = await Server.get('tweets/1/likes'); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/); -
Async + Await - concurrent const receiveData = async ()
=> { try { const [tweets, tweet, likes] = await Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/); -
Node.js Installation • If not done yet: install Node.js •
https://nodejs.org v8+ • IDE • VS Code: https://code.visualstudio.com/ • WebStorm: https://www.jetbrains.com/webstorm/ -
Git Basics • Clone Project • http://bit.ly/ijs-nodejs-workshop • git clone
https://github.com/dskgry/nodejs-workshop.git • Switch branches • git checkout -f BRANCHNAME (e.g. git checkout -f 01_basics) • Alternative: Use GitHub Desktop J -
Install dependencies • Go to ./server • Run from command
line: npm install • Start the server: npm start • Go to ./app • Run from command line: npm install • Start the app: npm start -
Exercise: JavaScript basics • Branch: 01_basics • Folder: basics/esnext •
Files: destructuring.js, arrowFunction.js, promises.js, asyncAwait.js • Run a file by typing node ‘filename‘ in a console • E.g. node destructuring • Need help? https://javascript.info/ -
Exercise: Node modules • Branch: 01_basics • Folder: basics/modules •
Files: main.js, osinfo.js • Run by typing node main in a console • You need to be in the basics/modules folder • Need help? https://nodejs.org/en/docs/ -
Alternatives • .NET & .NET Core • .NET Core is
also platform independent • Entity Framework for database access • Java • JAX-RS / Spring Rest • JPA • Ruby • Python • … -
Pros • One language to rule them all: Full stack
JS‘ish development • Universal/Isomorphic JavaScript: Share code for client & server • Open Source loving community -
Pros • It scales • Enterprise proven: Netflix, Paypal, Groupon,
Walmart, … https://nodejs.org/static/documents/casestudies/Node_CaseStudy_Nasa_FNL.pdf -
Watch out! • Single threaded Event Loop: Avoid heavy CPU
usage • Utilizes one CPU only: Scale via clustering • Relational databases can be strange • Code is (often the only) documentation • Still JavaScript à Use TS or Flow in bigger projects -
Restify • Node.js module to build Web APIs • Middleware
support • Client & Server module • Routing • vs Express/Koa.js • No templating • No rendering • Used by Netflix -
Restify const restify = require('restify'); const server = restify.createServer(); server.get('person',
(req, res, next) => { const person = {}; res.send(person); next(); }); server.listen(3001, function () { console.log('server up <3'); }); -
Middleware • Chainable • Run in the order registered •
Global Middlewares for every request à Plugins • Middlewares per route • Logging • Validation • Authentication • Caching (Etag-Handlers) Middleware 1 // Server logic next() // more logic Middleware 2 // Server logic next() // more logic Middleware 3 // Server logic // more logic Request Response Client Logging Authentication … -
Global Middlewares, Plugins • Via .use (after routing) • Via
.pre (before routing) const server = restify.createServer(); server.use(restify.plugins.gzipResponse()); server.use(restify.plugins.queryParser()); server.use(restify.plugins.bodyParser()); const server = restify.createServer(); server.pre(restify.plugins.throttle({burst: 10, rate: 10, ip: true})); -
CORS Plugin • Combines .pre and .use • Supported via
official 3rd party plugin by TabDigital const corsMiddleware = require('restify-cors-middleware'); const cors = corsMiddleware({ origin: '*' }); server.pre(cors.preflight); server.use(cors.actual); -
Custom global Middlewares • Middlewares for all routes // MyMiddleware.js
module.exports = (req, res, next) => { // do smart stufff next(); }; // MyServer.js server.pre(myMiddleware); server.use(myMiddleware); -
Route-based Middlewares server.get( '/foo', myMiddleware, (req, res, next) => {
console.log('Authenticate'); next(); }, (req, res, next) => { res.send(200); return next(); // return is optional } ); -
Exercise: Hello restify • Branch: 02_hello_restify • Folder: server/src •
Files: index.js, Server.js • Run index.js by typing npm start (from folder server) • Changes are detected automatically • Check if it worked with postman or your browser • Need help? • https://nodejs.org/en/docs/ • http://restify.com/ -
Let’s get real! Building a Web API for a real
use case: Your own real time Twitter -
Project structure • Separation of concerns • Also in JavaScript
;) • Domain oriented „packages“ • Technical „packages“ -
CORS • Problem: Same origin restriction policy • Solution: Cross-origin
resource sharing (CORS) • W3C Standard • Server describes (via HTTP-Header) • Which origins are allowed (e.g. http://google.com) • Which methods are allowed (e.g. GET, POST) -
Exercise: Hello TweetResource • Goal: Receive all tweets via. HTTP-API
(GET localhost:3001/tweets) • Use restify cors plugin • Branch: 03_tweetresource • Files: index.js, Server.js, TweetsResource.js • Test with postman • Test with our app (don‘t forget CORS) • npm start • Need help? http://restify.com/ -
Exercise: Hello TweetService • Goal: Create TweetService • Implement getTweets,
getTweet, countTweets, createTweet • Let‘s do TDD: Use the TweetServiceTest (npm test) • Add pagination to resource (read query parameters) • Use TweetService in Resource • Use restify queryParser plugin • Branch: 04_tweetservice • Files: TweetService.js, TweetServiceTest.js , TweetsResource.js, Server.js • Test in Postman (e.g. localhost:3001/tweets?page=1&size=1) -
Validation • Never ever ever trust the client • JS
=== dynamically typed • 3rd party libs to check object shapes • Yup: • “Dead simple Object schema validation” • Validate objects • Sanitize objects -
Yup const yup = require('yup'); const schema = yup.object().shape({ name:
yup.string().required(), age: yup.number().required().positive().integer(), email: yup.string().email(), website: yup.string().url(), //... }); try { const validated = await schema.isValid({name: 'jimmy', age: 24}); } catch (e) { //... } -
Validation as middleware server.get('foo', validation.validateQueryParams({ times: yup.number().min(1).max(10).default(1) }), (req, res,
next) => {/*...*/} ); Validate query params server.post('foo', validation.validatePostBody({ mail: yup.string().mail().required(), name: yup.string().min(3).max(10) }), (req, res, next) => {/*...*/} ); Validate request payload -
HTTP POST: An inspirational reminder • Creates a new resource
• Server decides where • Location header • Answers with what you‘ve created • Status-Code: 201 (Created) POST /tweets Content-Type: application/json { "tweet“ :“…“ } ------------------------------------------------------------------------- CREATED 201 Location: /tweets/123 { “id“ : 123 "tweet“ :“…“ “author“ : { … } } -
Exercise: Validation + HTTP POST • Goal: Add POST +
„single“-GET resource methods • Validate the queryparams • Validate the request body • Extract a path-param • Branch: 05_post_validation • Files: TweetsResource.js, Validation.js, Server.js • Test in our app + Postman • Need help? • https://github.com/jquense/yup • http://restify.com/ -
Token-based authentication • Exchange credentials for token • Token: •
Random string • Self-contained • JWT • Typically send via Authorization header • Don‘t create tokens on your own • Use certified Security Token Services • https://identityserver.github.io • https://github.com/panva/node-oidc-provider -
Token-based authentication Client Server Username & Password 1. check 2.
generate token Token Store token Token Validate & execute -
Exercise: Middlewares + Security • Goal: Add Logger-Middleware and Security-Middleware
• Branch: 06_middleware_security • Files: Logger.js, Security.js, Server.js • Test in our app + Postman • Don‘t forget to set the Authorization header in Postman -
Real time web • Server pushes data directly to the
client • Use Cases • Chat • Stock info / Newsticker • Progress information • Collaboration tools • Legacy • Polling / Long polling -
Web Sockets • Bi-Directional • Server pushes • Client pushes
• Use Cases • Collaboration tools • Chat • New protocol • Persistent connections -
Server-sent events • Push technology only • HTTP-based • Auto
reconnect • Use-Cases • News streaming • Progress bars • But • Less supported -
Web Sockets & Node === Easy • Usage via libraries
• ws • socket.io const webSocket = require('ws'); const wss = new webSocket.Server({server}); wss.on('connection', ws => { const interval = setInterval(() => { const stockData = getStockData(); ws.send(JSON.stringify(stockData)); }, 1000); ws.on('close', () => { clearInterval(interval); }); }); -
EventEmitter • Node === event driven • Sync & Async
events • Custom events via EventEmitter-Class • Loose coupling of modules • Synchronous! • Don’t forget to “removeListener” const readable = getReadableStreamSomehow(); readable.on('data', (chunk) => {…}); readable.on('end', () => {…}); const events = require('events'); const eventEmitter = new events.EventEmitter(); eventEmitter.addListener('myEvent', data => console.log(data)); //... somewhere else eventEmitter.emit('myEvent', someData); -
Exercise: Server push • Goal: Add websocket support • Use
EventEmitter • emit event when tweet created • consume event and send via Web Socket • Branch: 07_sockets • Files: TweetService.js, Server.js • Test in our app • Open in two browser windows and tweet something J • Need help? https://github.com/websockets/ws -
Databases • Plays well with different kind of databases •
NoSQL databases like MongoDB, CouchDB • Relational databases like PostgresSQL, MSSQL • Real time databases like RethinkDB • Access databases directly or via ORM • SequelizeJS • TypeORM -
RethinkDB • Open Source, scalable JSON real time database •
Can push updated query results directly to the application • Use Cases are all real time based applications • Multiplayer games • Streaming analytics • Collaborative tools • Don’t use it, if you need full ACID or strong schemas -
RethinkDB – API • RethinkDB simple queries • All queries
return promises! const r = require('rethinkdb'); await r.table('MY_TABLE').get('1').run(connection); // get by id await r.table('MY_TABLE').count().run(connection); // count stuff -
Exercise: Async-Programing + RethinkDB • Goal: Add RethinkDB support •
Do some async programming • Use RethinkDB-API • Branch: 08_database • Files: index.js, TweetServicejs, TweetResource.js • Test in our app • Data will survive server restart J • Automatic real time support • Need help? https://rethinkdb.com/docs/nodejs/ -
Testing APIs • E2E-Tests • Slow & too much effort
• Integration tests • Test API only • Mock services / Database • Check: • Status codes • Response (JSON) • Validation -
SuperTest describe(’MyResource', () => { beforeEach(() => { someService.doStuff.mockImplementation(() =>
({ id: 1, name: 'test'})); }); it(’a get request', async done => { const response = await supertest(server.getServer()).get('/somepath'); expect(response.status).toBe(200); done(); // tell SuperTest we’re done }); }); -
Exercise: Test our API • Goal: Test our API •
See some mocking • Test responses (status codes and body) • Branch: 09_apitest • Files: TweetResourceTest.js • npm test • Need help? https://github.com/visionmedia/supertest -
A public API needs more… • Reduce network traffic •
Caching • GZIP • HATEOAS • Location-Header for POST-Methods • Link-Header for pagination • Rate limiting, Throttling • Versioning -
Reduce traffic • Tell client that data did not change
• Code 304 • Two standardized options: • Last Modified-Header • Did data change since… • Etag-Header • Did checksum of data change? GET /tweets OK 200 Etag: 7982882299 GET /tweets If-None-Match: 7982882299 304: NOT MODIFIED following request initial request -
Use GZIP • Supported by all modern browsers • Supported
by most servers • Reduces traffic by up to 70% • But • more computation • is not always smaller -
HATEOAS • Hypermedia as the engine of application state •
Navigate from anywhere to everywhere • Usage of API without prior knowledge -
HATEOAS • Example: POST • Example: Pagination POST /tweets {…
------------------------------------------------------------------------- CREATED 201 Location: http://twttr.de/tweets/123 {…} GET /tweets/?page=1&size=10 HEADER Link: <http://twttr.de/tweets/?page=2&size=10>; rel="next”>, <http://twttr.de/tweets/?page=9&size=10 >; rel="last”> BODY [{…},{…}] -
Rate limiting (Throttling) • Public APIs • DDoS protection •
Pay per request APIs • Rates for „expensive“ resources • Rates per timeframe • Hour, Minute, Second • How? • IP-based • Account-based -
Versioning • via URL (twttr.com/v1/tweets) • via Accept-Header • via
Custom-Header • Restify (Accept-Version) GET http:twttr.com/tweets/ Accept-Version: 1.0.0 200 OK [{tweet:”…”},{tweet:”…”}] GET http:twttr.com/tweets/ Accept-Version: 3.0.0 406 NOT ACCEPTABLE const server = restify.createServer({ name: 'Twttr', version: '1.0.0' }); -
Exercise: API Restify • Optimize our HTTP API • Add
Versioning, Throttling and GZIP support • Set the Accept-Version header in postman to test this • Add response headers (Location for POST, Link for pagination) • Implement conditional requests with ETAG (for single tweet retrieval) • Set the If-None-Match header in postman to test this • Branch: 10_api_restify • Files • Server.js, TweetResource.js, HttpHelper.js -
Summary • Modern SPAs needs modern backends • A backend
can be orchestrated with several technologies/languages • Node.js is one more possibility to write Web APIs • Web APIs should be use case centric – not pure REST • Leverage existing Node.js modules for kickstarting Web APIs • Restify sets up a base for writing Web APIs • Make usage of real time Web Sockets communication • RethinkDB is one possibility for databases in Node.js • Don’t forget gzip, versioning, rate limiting and security