Modern Web APIs with Node.js

Companies like Netflix, PayPal and even NASA have shown that with the help of Node.js, JavaScript can be used to implement scalable server-side solutions. The asynchronous and non-blocking nature of the language makes it easy to create modern and fast web applications with real-time features, while keeping a small overall footprint. Sven Kölpin and Manuel Rauber will introduce you to this brave new world. They will use a demo application to show you how to create modern web APIs with Node.js. Topics like API design, real-time communication, security, database access and single-page applications will be covered within the workshop.

GitHub: https://github.com/dskgry/nodejs-workshop

Transcript

1. Modern Web APIs with Node.js A glance across the border Sven Kölpin sven.koelpin@openknowledge.de @dskgry Manuel Rauber manuel.rauber@thinktecture.com @manuelrauber
2. Your Speakers Sven Kölpin Does stuff @ OPEN KNOWLEDGE GmbH ! sven.koelpin@openknowledge.de " @dskgry Manuel Rauber Consultant @ Thinktecture AG ! manuel.rauber@thinktecture.com " @manuelrauber # https://manuel-rauber.com
3. Timetable When What 09:00 – 10:30 Working time 10:30 – 11:00 Break 11:00 – 12:30 Working time 12:30 – 13:30 Lunchbreak 13:30 – 15:00 Working time 15:00 – 15:30 ☕ & " 15:30 – 17:00 Working time
4. Talking points • Web APIs • Cross Platform • The web as a platform • Node.js • Web Sockets • RethinkDB • Testing
5. Web APIs Access all the data!
6. Web APIs
7. Web APIs • Lightweight service-based architecture • Functional services with dedicated interfaces • Use other services, like database or file system • REST does not model real world usecases • (JSON-based) Web APIs • Application push services via WebSocket • SignalR • Socket.io • Prefer HTTPS over HTTP HTTP HTTPS WebSocket Service A Service B Service C Web APIs (ASP.NET, Node.js, …) HTML5-Application (Single-Page Application) Web Cordova Electron
8. Cross-Platform Runs on all the devices!
9. Single- vs. Multi- vs. Cross-Platform macOS Linux Windows iOS Windows Phone Android BlackBerry 10 FireOS Browser TV … Refrigerator
10. Cross-Platform
11. But it doesn’t look like a native application! Exactly.
12. Angular 2: Komponentenbasierte HTML5-Anwendungen • BASTA! Spring 2017 https://spotifyblogcom.files.wordpress.com/2014/12/overview.png
13. Angular 2: Komponentenbasierte HTML5-Anwendungen • BASTA! Spring 2017 http://media.idownloadblog.com/wp-content/uploads/2014/06/Google-Docs-Sheets-Slides-teaser-001.jpg
14. The web as a platform It’s there. Use it!
15. The web as a platform
16. React Single-Page Applications
17. Single-Page Applications • Web application with ONE HTML page • App-Like Behavior (WebApps) • Fast & highly interactive • Real time • Offline capabilities • Mobile Web • Server: Data, not DOM
18. Lifecycle SPA Browser Server (e.g. node) Initial Request HTML / JavaScript /CSS XHR Request JSON Render Page Static files Page running Dynamic resources (HTTP API)
19. Component import React, {Component} from 'react'; class MyView extends Component { render() { return ( <div className='container'> <h1>Hello World!</ h1 > </div> ); } }
20. Facts • Est. 2013 • Version 16.x • Not a framework
21. Demo Show me, what you got!
22. Node.js Or: How you can execute jQuery on the server. ;-)
23. Intro • Server-side JavaScript powered by Chrome’s V8 JavaScript engine • Asynchronous, event-driven I/O API • Can access system resources (e.g. file system) • Package management via Node Package Manager (npm) oder yarn • Cross platform: Linux, macOS, Windows • Upcoming: Alternative runtime powered by Microsoft’s ChakraCore
24. Features • ECMAScript 2015 – 2017 (V8) • Classes • Typed arrays • Fat arrow/Lambda Expressions • Templated Strings • Promises • Async / Await • …
25. Server Architecture
26. Why does it scale? • Focus on requests • Delegate I/O (event loop) • No blocking threads in code • Less memory/cpu consumption • JSON • Avoid heavy serverside calculations • Clustering • Use the event loop • Native modules
27. Modules • JavaScript-File === Modul (CommonJS) • „Singleton“ • Can use other modules (require) • Exports functions or data (exports) • Node.js offers modules • FileSystem, HTTP, OS,… • 3rd party modules via npm const fs = require('fs'); const os = require('os'); fs.mkdir("/some/folder", (err, folder)=> {}); os.platform(); // win32
28. Custom modules • fileA.js • fileB.js module.exports = { sayHello(){ console.log("Hello IJS 2017"); } }; const moduleA = require("./fileA"); console.log(moduleA.sayHello()); // Hello IJS 2017 console.log(moduleA); // { sayHello: [Function: sayHello] }
29. How about some work… for you? ;-) Modern JavaScript: ECMAScript 2015-2017
30. Let & Const • var === function scope • Hoisting • Mother of all bugs • let & const === block Scope • Prefer const function fn() { if (true) { var x = 1; } console.log(x); // ? } function fn() { if (true) { const x = 1; // or let } console.log(x); // ? }
31. Destructuring: Objects const person = { name : "max", age : 53 }; const name = person.name; const age = person.age //... const {name, age} = person; console.log(name); // max console.log(age); // 53 const {name:firstName} = person; console.log(firstName); // max
32. Destructuring: Arrays var arr = ["a", "b", "c"]; const [d,e,f] = arr; console.log(d, e, f); // a b c const [,, third] = arr; console.log(third); // c const [first] = arr; console.log(first); // a
33. Classes class Person{ static myStaticMethod(){} constructor(fn,ln){ this.firstName = fn; this.lastName = ln; } toString() { return `${this.firstName} ${this.lastName}`; } }
34. Inheritance class Student extends Person { constructor(fn, ln, matrNo) { super(fn, ln); this.matrNo = matrNo; } } let student = new Student("Sven", "Koelpin", "123456");
35. Classes – Watch out • syntactic sugar • super calls • no privates (yet) • no class properties (yet)
36. Arrow functions function say(what) { return what; } const say = what => what; say('hello‘); function add(a, b) { return a + b; } const add = (a, b) => { return a + b; } function Person() { this.age = 0; setInterval(function() { this.age++; // BOOOM }, 1000); } function Person() { this.age = 0; setInterval(()=> { this.age++; // works }, 1000); }
37. Promises • JS === single-threaded à„callback hell“? • a.k.a. „pyramid of doom“ Server.get('tweets', tweets => { Server.get('tweets/1', tweet => { Server.get('tweets/1/likes', likes => { //... }); }); });
38. Promises • Make async operations „thenable“ • They still run within a single thread Server.get('tweets') .then(tweets => Server.get('tweets/1')) .then(tweet => Server.get('tweets/1/likes')) .then(/*...*/) .catch(e => /*...*/)
39. Promises - concurrent • Runs concurrently, but still on a single thread (I/O delegation) Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]) .then(result => { const [tweets, tweet, likes] = result; }) .catch(e = >{ // one request f*cked up });
40. Async + Await • Promises that look synchronous const receiveData = async () => { try { const tweets = await Server.get('tweets'); const tweet = await Server.get('tweets/1'); const likes = await Server.get('tweets/1/likes'); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/);
41. Async + Await - concurrent const receiveData = async () => { try { const [tweets, tweet, likes] = await Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/);
42. Installation
43. Node.js Installation • If not done yet: install Node.js • https://nodejs.org v8+ • IDE • VS Code: https://code.visualstudio.com/ • WebStorm: https://www.jetbrains.com/webstorm/
44. Git Basics • Clone Project • http://bit.ly/ijs-nodejs-workshop • git clone https://github.com/dskgry/nodejs-workshop.git • Switch branches • git checkout -f BRANCHNAME (e.g. git checkout -f 01_basics) • Alternative: Use GitHub Desktop J
45. Install dependencies • Go to ./server • Run from command line: npm install • Start the server: npm start • Go to ./app • Run from command line: npm install • Start the app: npm start
46. Exercise: JavaScript basics • Branch: 01_basics • Folder: basics/esnext • Files: destructuring.js, arrowFunction.js, promises.js, asyncAwait.js • Run a file by typing node ‘filename‘ in a console • E.g. node destructuring • Need help? https://javascript.info/
47. Exercise: Node modules • Branch: 01_basics • Folder: basics/modules • Files: main.js, osinfo.js • Run by typing node main in a console • You need to be in the basics/modules folder • Need help? https://nodejs.org/en/docs/
48. Alternatives • .NET & .NET Core • .NET Core is also platform independent • Entity Framework for database access • Java • JAX-RS / Spring Rest • JPA • Ruby • Python • …
49. Pros • One language to rule them all: Full stack JS‘ish development • Universal/Isomorphic JavaScript: Share code for client & server • Open Source loving community
50. Pros • It scales • Enterprise proven: Netflix, Paypal, Groupon, Walmart, … https://nodejs.org/static/documents/casestudies/Node_CaseStudy_Nasa_FNL.pdf
51. Watch out! • Single threaded Event Loop: Avoid heavy CPU usage • Utilizes one CPU only: Scale via clustering • Relational databases can be strange • Code is (often the only) documentation • Still JavaScript à Use TS or Flow in bigger projects
52. Restify Web APIs done right.
53. Restify • Node.js module to build Web APIs • Middleware support • Client & Server module • Routing • vs Express/Koa.js • No templating • No rendering • Used by Netflix
54. Restify const restify = require('restify'); const server = restify.createServer(); server.get('person', (req, res, next) => { const person = {}; res.send(person); next(); }); server.listen(3001, function () { console.log('server up <3'); });
55. Middleware • Chainable • Run in the order registered • Global Middlewares for every request à Plugins • Middlewares per route • Logging • Validation • Authentication • Caching (Etag-Handlers) Middleware 1 // Server logic next() // more logic Middleware 2 // Server logic next() // more logic Middleware 3 // Server logic // more logic Request Response Client Logging Authentication …
56. Global Middlewares, Plugins • Via .use (after routing) • Via .pre (before routing) const server = restify.createServer(); server.use(restify.plugins.gzipResponse()); server.use(restify.plugins.queryParser()); server.use(restify.plugins.bodyParser()); const server = restify.createServer(); server.pre(restify.plugins.throttle({burst: 10, rate: 10, ip: true}));
57. CORS Plugin • Combines .pre and .use • Supported via official 3rd party plugin by TabDigital const corsMiddleware = require('restify-cors-middleware'); const cors = corsMiddleware({ origin: '*' }); server.pre(cors.preflight); server.use(cors.actual);
58. Custom global Middlewares • Middlewares for all routes // MyMiddleware.js module.exports = (req, res, next) => { // do smart stufff next(); }; // MyServer.js server.pre(myMiddleware); server.use(myMiddleware);
59. Route-based Middlewares server.get( '/foo', myMiddleware, (req, res, next) => { console.log('Authenticate'); next(); }, (req, res, next) => { res.send(200); return next(); // return is optional } );
60. Exercise: Hello restify • Branch: 02_hello_restify • Folder: server/src • Files: index.js, Server.js • Run index.js by typing npm start (from folder server) • Changes are detected automatically • Check if it worked with postman or your browser • Need help? • https://nodejs.org/en/docs/ • http://restify.com/
61. Let’s get real! Building a Web API for a real use case: Your own real time Twitter
62. Project structure • Separation of concerns • Also in JavaScript ;) • Domain oriented „packages“ • Technical „packages“
63. CORS • Problem: Same origin restriction policy • Solution: Cross-origin resource sharing (CORS) • W3C Standard • Server describes (via HTTP-Header) • Which origins are allowed (e.g. http://google.com) • Which methods are allowed (e.g. GET, POST)
64. Exercise: Hello TweetResource • Goal: Receive all tweets via. HTTP-API (GET localhost:3001/tweets) • Use restify cors plugin • Branch: 03_tweetresource • Files: index.js, Server.js, TweetsResource.js • Test with postman • Test with our app (don‘t forget CORS) • npm start • Need help? http://restify.com/
65. Exercise: Hello TweetService • Goal: Create TweetService • Implement getTweets, getTweet, countTweets, createTweet • Let‘s do TDD: Use the TweetServiceTest (npm test) • Add pagination to resource (read query parameters) • Use TweetService in Resource • Use restify queryParser plugin • Branch: 04_tweetservice • Files: TweetService.js, TweetServiceTest.js , TweetsResource.js, Server.js • Test in Postman (e.g. localhost:3001/tweets?page=1&size=1)
66. Validation • Never ever ever trust the client • JS === dynamically typed • 3rd party libs to check object shapes • Yup: • “Dead simple Object schema validation” • Validate objects • Sanitize objects
67. Yup const yup = require('yup'); const schema = yup.object().shape({ name: yup.string().required(), age: yup.number().required().positive().integer(), email: yup.string().email(), website: yup.string().url(), //... }); try { const validated = await schema.isValid({name: 'jimmy', age: 24}); } catch (e) { //... }
68. Validation as middleware server.get('foo', validation.validateQueryParams({ times: yup.number().min(1).max(10).default(1) }), (req, res, next) => {/*...*/} ); Validate query params server.post('foo', validation.validatePostBody({ mail: yup.string().mail().required(), name: yup.string().min(3).max(10) }), (req, res, next) => {/*...*/} ); Validate request payload
69. HTTP POST: An inspirational reminder • Creates a new resource • Server decides where • Location header • Answers with what you‘ve created • Status-Code: 201 (Created) POST /tweets Content-Type: application/json { "tweet“ :“…“ } ------------------------------------------------------------------------- CREATED 201 Location: /tweets/123 { “id“ : 123 "tweet“ :“…“ “author“ : { … } }
70. Exercise: Validation + HTTP POST • Goal: Add POST + „single“-GET resource methods • Validate the queryparams • Validate the request body • Extract a path-param • Branch: 05_post_validation • Files: TweetsResource.js, Validation.js, Server.js • Test in our app + Postman • Need help? • https://github.com/jquense/yup • http://restify.com/
71. Token-based authentication • Exchange credentials for token • Token: • Random string • Self-contained • JWT • Typically send via Authorization header • Don‘t create tokens on your own • Use certified Security Token Services • https://identityserver.github.io • https://github.com/panva/node-oidc-provider
72. Token-based authentication Client Server Username & Password 1. check 2. generate token Token Store token Token Validate & execute
73. Exercise: Middlewares + Security • Goal: Add Logger-Middleware and Security-Middleware • Branch: 06_middleware_security • Files: Logger.js, Security.js, Server.js • Test in our app + Postman • Don‘t forget to set the Authorization header in Postman
74. Real time web • Server pushes data directly to the client • Use Cases • Chat • Stock info / Newsticker • Progress information • Collaboration tools • Legacy • Polling / Long polling
75. Web Sockets • Bi-Directional • Server pushes • Client pushes • Use Cases • Collaboration tools • Chat • New protocol • Persistent connections
76. Web Sockets
77. Server-sent events • Push technology only • HTTP-based • Auto reconnect • Use-Cases • News streaming • Progress bars • But • Less supported
78. Web Sockets & Node === Easy • Usage via libraries • ws • socket.io const webSocket = require('ws'); const wss = new webSocket.Server({server}); wss.on('connection', ws => { const interval = setInterval(() => { const stockData = getStockData(); ws.send(JSON.stringify(stockData)); }, 1000); ws.on('close', () => { clearInterval(interval); }); });
79. EventEmitter • Node === event driven • Sync & Async events • Custom events via EventEmitter-Class • Loose coupling of modules • Synchronous! • Don’t forget to “removeListener” const readable = getReadableStreamSomehow(); readable.on('data', (chunk) => {…}); readable.on('end', () => {…}); const events = require('events'); const eventEmitter = new events.EventEmitter(); eventEmitter.addListener('myEvent', data => console.log(data)); //... somewhere else eventEmitter.emit('myEvent', someData);
80. Exercise: Server push • Goal: Add websocket support • Use EventEmitter • emit event when tweet created • consume event and send via Web Socket • Branch: 07_sockets • Files: TweetService.js, Server.js • Test in our app • Open in two browser windows and tweet something J • Need help? https://github.com/websockets/ws
81. Databases Every business application needs a database, right?
82. Databases • Plays well with different kind of databases • NoSQL databases like MongoDB, CouchDB • Relational databases like PostgresSQL, MSSQL • Real time databases like RethinkDB • Access databases directly or via ORM • SequelizeJS • TypeORM
83. RethinkDB • Open Source, scalable JSON real time database • Can push updated query results directly to the application • Use Cases are all real time based applications • Multiplayer games • Streaming analytics • Collaborative tools • Don’t use it, if you need full ACID or strong schemas
84. RethinkDB – API • RethinkDB simple queries • All queries return promises! const r = require('rethinkdb'); await r.table('MY_TABLE').get('1').run(connection); // get by id await r.table('MY_TABLE').count().run(connection); // count stuff
85. Exercise: Async-Programing + RethinkDB • Goal: Add RethinkDB support • Do some async programming • Use RethinkDB-API • Branch: 08_database • Files: index.js, TweetServicejs, TweetResource.js • Test in our app • Data will survive server restart J • Automatic real time support • Need help? https://rethinkdb.com/docs/nodejs/
86. Testing Does it work? Really?
87. Testing APIs • E2E-Tests • Slow & too much effort • Integration tests • Test API only • Mock services / Database • Check: • Status codes • Response (JSON) • Validation
88. SuperTest describe(’MyResource', () => { beforeEach(() => { someService.doStuff.mockImplementation(() => ({ id: 1, name: 'test'})); }); it(’a get request', async done => { const response = await supertest(server.getServer()).get('/somepath'); expect(response.status).toBe(200); done(); // tell SuperTest we’re done }); });
89. Exercise: Test our API • Goal: Test our API • See some mocking • Test responses (status codes and body) • Branch: 09_apitest • Files: TweetResourceTest.js • npm test • Need help? https://github.com/visionmedia/supertest
90. But wait: There’s more!
91. A public API needs more… • Reduce network traffic • Caching • GZIP • HATEOAS • Location-Header for POST-Methods • Link-Header for pagination • Rate limiting, Throttling • Versioning
92. Reduce traffic • Tell client that data did not change • Code 304 • Two standardized options: • Last Modified-Header • Did data change since… • Etag-Header • Did checksum of data change? GET /tweets OK 200 Etag: 7982882299 GET /tweets If-None-Match: 7982882299 304: NOT MODIFIED following request initial request
93. Use GZIP • Supported by all modern browsers • Supported by most servers • Reduces traffic by up to 70% • But • more computation • is not always smaller
94. HATEOAS • Hypermedia as the engine of application state • Navigate from anywhere to everywhere • Usage of API without prior knowledge
95. HATEOAS • Example: POST • Example: Pagination POST /tweets {… ------------------------------------------------------------------------- CREATED 201 Location: http://twttr.de/tweets/123 {…} GET /tweets/?page=1&size=10 HEADER Link: <http://twttr.de/tweets/?page=2&size=10>; rel="next”>, <http://twttr.de/tweets/?page=9&size=10 >; rel="last”> BODY [{…},{…}]
96. Rate limiting (Throttling) • Public APIs • DDoS protection • Pay per request APIs • Rates for „expensive“ resources • Rates per timeframe • Hour, Minute, Second • How? • IP-based • Account-based
97. Versioning • via URL (twttr.com/v1/tweets) • via Accept-Header • via Custom-Header • Restify (Accept-Version) GET http:twttr.com/tweets/ Accept-Version: 1.0.0 200 OK [{tweet:”…”},{tweet:”…”}] GET http:twttr.com/tweets/ Accept-Version: 3.0.0 406 NOT ACCEPTABLE const server = restify.createServer({ name: 'Twttr', version: '1.0.0' });
98. Exercise: API Restify • Optimize our HTTP API • Add Versioning, Throttling and GZIP support • Set the Accept-Version header in postman to test this • Add response headers (Location for POST, Link for pagination) • Implement conditional requests with ETAG (for single tweet retrieval) • Set the If-None-Match header in postman to test this • Branch: 10_api_restify • Files • Server.js, TweetResource.js, HttpHelper.js
99. Summary
100. Summary • Modern SPAs needs modern backends • A backend can be orchestrated with several technologies/languages • Node.js is one more possibility to write Web APIs • Web APIs should be use case centric – not pure REST • Leverage existing Node.js modules for kickstarting Web APIs • Restify sets up a base for writing Web APIs • Make usage of real time Web Sockets communication • RethinkDB is one possibility for databases in Node.js • Don’t forget gzip, versioning, rate limiting and security
101. Thank you! Questions? Repository https://github.com/dskgry/nodejs-workshop Sven Kölpin sven.koelpin@openknowledge.de @dskgry Manuel Rauber manuel.rauber@thinktecture.com @manuelrauber