Moderne Web APIs mit Node.js – ein Blick hinter die Kulissen

Moderne Web APIs mit Node.js – ein Blick hinter die Kulissen

FullStack JavaScript: Vorbei sind die Zeiten, in denen JavaScript nur auf dem Client eingesetzt wurde! Große Firmen, wie Netflix, PayPal oder sogar die NASA machen es vor: JavaScript auf dem Server mit Node.js. Und dabei nutzen sie die Vorteile von Node.js: Asynchronität und Performance machen Echtzeitanforderungen zum Kinderspiel. Gepaart mit einem kleinen Server-Footprint und einer hohen Skalierbarkeit ist Node.js ideal für den Servereinsatz geeignet. Gemeinsam mit Ihnen werfen Sven Koelpin und Manuel Rauber einen Blick auf die neue Welt und zeigen, wie man mit Node.js moderne Web APIs entwickeln kann. Dabei werden Themen wie Absicherung von Web APIs, Datenbankanbindung und die Bereitstellung einer Single-Page Application anhand eines praktischen Beispiels erörtert – als Grundlage für Ihre „next generation“ Web API.

GitHub: https://github.com/thinktecture/api-summit-2017-nodejs-workshop

667fbca1f58bc0215c744b5ae8f8e5d2?s=128

Manuel Rauber

June 21, 2017
Tweet

Transcript

  1. Moderne Web APIs mit Node.js Ein Blick hinter die Kulissen

    Sven Kölpin sven.koelpin@openknowledge.de @dskgry Manuel Rauber manuel.rauber@thinktecture.com @manuelrauber
  2. Your Speakers Sven Kölpin Does stuff @ OPEN KNOWLEDGE GmbH

    ! sven.koelpin@openknowledge.de " @dskgry Manuel Rauber Consultant @ Thinktecture AG ! manuel.rauber@thinktecture.com " @manuelrauber # https://manuel-rauber.com
  3. Timetable When What 09:00 – 10:30 Working time 10:30 –

    11:00 Break 11:00 – 12:30 Working time 12:30 – 13:30 Lunchbreak 13:30 – 15:00 Working time 15:00 – 15:30 ☕ & " 15:30 – 17:00 Working time
  4. Talking points • Web APIs • Cross Platform • The

    web as a platform • Node.js • Web Sockets • RethinkDB • Testing
  5. Web APIs Access all the data!

  6. Web APIs

  7. Web APIs • Lightweight service-based architecture • Functional services with

    dedicated interfaces • Use other services, like database or file system • REST does not model real world usecases • (JSON-based) Web APIs • Application push services via WebSocket • SignalR • Socket.io • Prefer HTTPS over HTTP HTTP HTTPS WebSocket Service A Service B Service C Web APIs (ASP.NET, Node.js, …) HTML5-Application (Single-Page Application) Web Cordova Electron
  8. Cross-Platform Runs on all the devices!

  9. Single- vs. Multi- vs. Cross-Platform macOS Linux Windows iOS Windows

    Phone Android BlackBerry 10 FireOS Browser TV … Refrigerator
  10. Cross-Platform

  11. But it doesn’t look like a native application! Exactly.

  12. Angular 2: Komponentenbasierte HTML5-Anwendungen • BASTA! Spring 2017 https://spotifyblogcom.files.wordpress.com/2014/12/overview.png

  13. Angular 2: Komponentenbasierte HTML5-Anwendungen • BASTA! Spring 2017 http://media.idownloadblog.com/wp-content/uploads/2014/06/Google-Docs-Sheets-Slides-teaser-001.jpg

  14. The web as a platform It’s there. Use it!

  15. The web as a platform

  16. HTML5 • Markup language • Native features for the web!

    • Native input elements (like date, time, …) • Audio-/Video without plugins • Hardware accelerated 3D-content without plugins • Gamepad, Microphone without plugins • Local storage mechanisms (Key-Value-based, databases)
  17. CSS3 • Making the web a canvas for creativity •

    3D transformations • Animations • Responsive layouts
  18. JavaScript & TypeScript | Flow • The web’s world famous

    programming language • Dynamically typed • Current version: ECMAScript 2017 • Typings via TypeScript | Flow • Better static code analysis for IDE’s IntelliSense • Better refactoring capabilities
  19. React Single-Page Applications

  20. Single-Page Applications • Web application with ONE HTML page •

    App-Like Behavior (WebApps) • Fast & highly interactive • Real time • Offline capabilities • Mobile Web • Server: Data, not DOM
  21. Lifecycle SPA Browser Server (e.g. node) Initial Request HTML /

    JavaScript /CSS XHR Request JSON Render Page Static files Page running Dynamic resources (HTTP API)
  22. Component import React, {Component} from 'react'; class MyView extends Component

    { render() { return ( <div className='container'> <h1>Hello World!</ h1 > </div> ); } }
  23. Facts • Est. 2013 • Version 15.x • Not a

    framework
  24. Demo Show me, what you got!

  25. Node.js Or: How you can execute jQuery on the server.

    ;-)
  26. Intro • Server-side JavaScript powered by Chrome’s V8 JavaScript engine

    • Asynchronous, event-driven I/O API • Can access system resources (e.g. file system) • Package management via Node Package Manager (npm) oder yarn • Cross platform: Linux, macOS, Windows • Upcoming: Alternative runtime powered by Microsoft’s ChakraCore
  27. Features • ECMAScript 2015 – 2017 (V8) • Classes •

    Typed arrays • Fat arrow/Lambda Expressions • Templated Strings • Promises • Async / Await • …
  28. Server Architecture

  29. Why does it scale? • Focus on requests • Delegate

    I/O (event loop) • „Native“ threading • No blocking threads in code • Less memory/cpu consumption • JSON • Avoid heavy serverside calculations • Clustering • Use the event loop • Native modules
  30. Modules • JavaScript-File === Modul (CommonJS) • „Singleton“ • Can

    use other modules (require) • Exports functions or data (exports) • Node.js offers modules • FileSystem, HTTP, OS,… • 3rd party modules via npm const fs = require('fs'); const os = require('os'); fs.mkdir("/some/folder", (err, folder)=> {}); os.platform(); // win32
  31. Custom modules • fileA.js • fileB.js module.exports = { sayHello(){

    console.log("Hello Summit"); } }; const moduleA = require("./fileA"); console.log(moduleA.sayHello()); // hello summit console.log(moduleA); // { sayHello: [Function: sayHello] }
  32. How about some work… for you? ;-) Modern JavaScript: ECMAScript

    2015-2017
  33. Let & Const • var === function scope • Hoisting

    • Mother of all bugs • let & const === block Scope • Prefer const function fn() { if (true) { var x = 1; } console.log(x); // ? } function fn() { if (true) { const x = 1; // or let } console.log(x); // ? }
  34. Destructuring: Objects const person = { name : "max", age

    : 53 }; const name = person.name; const age = person.age //... const {name, age} = person; console.log(name); // max console.log(age); // 53 const {name:firstName} = person; console.log(firstName); // max
  35. Destructuring: Arrays var arr = ["a", "b", "c"]; const [d,e,f]

    = arr; console.log(d, e, f); // a b c const [,, third] = arr; console.log(third); // c const [first] = arr; console.log(first); // a
  36. Classes class Person{ static myStaticMethod(){} constructor(fn, ln){ this.firstName = fn;

    this.lastName = ln; } toString() { return `${this.firstName} ${this.lastName}`; } }
  37. Inheritance class Student extends Person { constructor(fn, ln, matrNo) {

    super(fn, ln); this.matrNo = matrNo; } } let student = new Student("Sven", "Koelpin", "123456");
  38. Classes – Watch out • syntactic sugar • super calls

    • no privates (yet) • no class properties (yet)
  39. Arrow functions function say(what) { return what; } const say

    = what => what; say('hello‘); function add(a, b) { return a + b; } const add = (a, b) => { return a + b; } function Person() { this.age = 0; setInterval(function() { this.age++; // BOOOM }, 1000); } function Person() { this.age = 0; setInterval(()=> { this.age++; // works }, 1000); }
  40. Promises • JS === single-threaded à„callback hell“? • a.k.a. „pyramid

    of doom“ Server.get('tweets', tweets => { Server.get('tweets/1', tweet => { Server.get('tweets/1/likes', likes => { //... }); }); });
  41. Promises • Make async operations „thenable“ • They still run

    within a single thread Server.get('tweets') .then(tweets => Server.get('tweets/1')) .then(tweet => Server.get('tweets/1/likes')) .then(/*...*/) .catch(e => /*...*/)
  42. Promises - concurrent • Runs concurrently, but still on a

    single thread (I/O delegation) Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]) .then(result => { const [tweets, tweet, likes] = result; }) .catch(e = >{ // one request f*cked up });
  43. Async + Await • Promises that look synchronous const receiveData

    = async () => { try { const tweets = await Server.get('tweets'); const tweet = await Server.get('tweets/1'); const likes = await Server.get('tweets/1/likes'); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/);
  44. Async + Await - concurrent const receiveData = async ()

    => { try { const [tweets, tweet, likes] = await Promise.all([ Server.get('tweets'), Server.get('tweets/1'), Server.get('tweets/1/likes') ]); } catch (e) { // ... } }; receiveData() // this is a promise .then(/*...*/);
  45. Installation

  46. Node.js Installation • If not done yet: install Node.js •

    https://nodejs.org v8+ • IDE • VS Code: https://code.visualstudio.com/ • WebStorm: https://www.jetbrains.com/webstorm/
  47. Git Basics • Clone Project • git clone http://bit.ly/api-summit-2017-nodejs •

    Switch branches • git checkout -f BRANCHNAME (e.g. git checkout -f 01_basics) • Alternative: Use GitHub Desktop J
  48. Install dependencies • Go to ./server • Run from command

    line: npm install • Start the server: npm start • Go to ./app • Run from command line: npm install • Start the app: npm start
  49. Exercise: JavaScript basics • Branch: 01_basics • Folder: basics/esnext •

    Files: destructuring.js, arrowFunction.js, promises.js, asyncAwait.js • Run a file by typing node ‘filename‘ in a console • E.g. node destructuring • Need help? https://javascript.info/
  50. Exercise: Node modules • Branch: 01_basics • Folder: basics/modules •

    Files: main.js, osinfo.js • Run by typing node main in a console • You need to be in the basics/modules folder • Need help? https://nodejs.org/en/docs/
  51. Alternatives • .NET & .NET Core • .NET Core is

    also platform independent • Entity Framework for database access • Java • JAX-RS / Spring Rest • JPA • Ruby • Python • …
  52. Pros • One language to rule them all: Full stack

    JS‘ish development • Universal/Isomorphic JavaScript: Share code for client & server • Open Source loving community
  53. Pros • It scales • Enterprise proven: Netflix, Paypal, Groupon,

    Walmart, … https://nodejs.org/static/documents/casestudies/Node_CaseStudy_Nasa_FNL.pdf
  54. Watch out! • Single threaded Event Loop: Avoid heavy CPU

    usage • Utilizes one CPU only: Scale via clustering • Relational databases can be strange • Code is (often the only) documentation • Still JavaScript à Use TS or Flow in bigger projects
  55. Restify Web APIs done right.

  56. Restify • Node.js module to build Web APIs • Middleware

    support • Client & Server module • Routing • vs Express/Koa.js • No templating • No rendering • Used by Netflix
  57. Restify const restify = require('restify'); const server = restify.createServer(); server.get('person',

    (req, res, next) => { const person = {}; res.send(person); next(); }); server.listen(3001, function () { console.log('server up <3'); });
  58. Middleware • Chainable • Run in the order registered •

    Global Middlewares for every request à Plugins • Middlewares per route • Logging • Validation • Authentication • Caching (Etag-Handlers) Middleware 1 // Server logic next() // more logic Middleware 2 // Server logic next() // more logic Middleware 3 // Server logic // more logic Request Response Client Logging Authentication …
  59. Global Middlewares, Plugins • Via .use (after routing) • Via

    .pre (before routing) const server = restify.createServer(); server.use(restify.CORS()); server.use(restify.acceptParser(['application/json'])); server.use(restify.gzipResponse()); server.use(restify.queryParser()); server.use(restify.bodyParser()); const server = restify.createServer(); server.pre(restify.throttle({burst: 10, rate: 10, ip: true})); server.pre(restify.pre.sanitizePath());
  60. Custom global Middlewares • Middlewares for all routes // MyMiddleware.js

    module.exports = (req, res, next) => { // do smart stufff next(); }; // MyServer.js server.pre(myMiddleware); server.use(myMiddleware);
  61. Route-based Middlewares server.get( '/foo', myMiddleware, (req, res, next) => {

    console.log('Authenticate'); next(); }, (req, res, next) => { res.send(200); return next(); // return is optional } );
  62. Exercise: Hello restify • Branch: 02_hello_restify • git checkout 02_hello_restify

    -f • Folder: server/src • Files: index.js, Server.js • Run index.js by typing npm start • Changes are detected automatically • Check if it worked with postman or your browser • Need help? • https://nodejs.org/en/docs/ • http://restify.com/
  63. Let’s get real! Building a Web API for a real

    use case: Your own real time Twitter
  64. Project structure • Separation of concerns • Also in JavaScript

    ;) • Domain oriented „packages“ • Technical „packages“
  65. CORS • Problem: Same origin restriction policy • Solution: Cross-origin

    resource sharing (CORS) • W3C Standard • Server describes (via HTTP-Header) • Which origins are allowed (e.g. http://google.com) • Which methods are allowed (e.g. GET, POST)
  66. Exercise: Hello TweetResource • Goal: Receive all tweets via HTTP-API

    (GET localhost:3001/tweets) • Use restify cors plugin • Branch: 03_tweetresource • Files: index.js, Server.js, TweetsResource.js • Test with postman • Test with our app (don‘t forget CORS) • npm start • Need help? http://restify.com/
  67. Exercise: Hello TweetService • Goal: Create TweetService • Implement getTweets,

    getTweet, countTweets, createTweet • Let‘s do TDD: Use the TweetServiceTest (npm test) • Add pagination to resource (read query parameters) • Use TweetService in Resource • Use restify queryParser plugin • Branch: 04_tweetservice • Files: TweetService.js, TweetServiceTest.js , TweetsResource.js, Server.js • Test in Postman (e.g. localhost:3001/tweets?page=1&size=1)
  68. Validation • Never ever ever trust the client • JS

    === dynamically typed • 3rd party libs to check object shapes • Yup: • “Dead simple Object schema validation” • Validate objects • Sanitize objects
  69. Yup const yup = require('yup'); const schema = yup.object().shape({ name:

    yup.string().required(), age: yup.number().required().positive().integer(), email: yup.string().email(), website: yup.string().url(), //... }); try { const validated = await schema.isValid({name: 'jimmy', age: 24}); } catch (e) { //... }
  70. Validation as middleware server.get('foo', validation.validateQueryParams({ times: yup.number().min(1).max(10).default(1) }), (req, res,

    next) => {/*...*/} ); Validate query params server.post('foo', validation.validatePostBody({ mail: yup.string().mail().required(), name: yup.string().min(3).max(10) }), (req, res, next) => {/*...*/} ); Validate request payload
  71. HTTP POST: An inspirational reminder • Creates a new resource

    • Server decides where • Location header • Answers with what you‘ve created • Status-Code: 201 (Created) POST /tweets Content-Type: application/json { "tweet“ :“…“ } ------------------------------------------------------------------------- CREATED 201 Location: /tweets/123 { “id“ : 123 "tweet“ :“…“ “author“ : { … } }
  72. Exercise: Validation + HTTP POST • Goal: Add POST +

    „single“-GET resource methods • Validate the queryparams • Validate the request body • Extract a path-param • Branch: 05_post_validation • Files: TweetsResource.js, Validation.js, Server.js • Test in our app + Postman • Need help? • https://github.com/jquense/yup • http://restify.com/
  73. Token-based authentication • Exchange credentials for token • Token: •

    Random string • Self-contained • JWT • Typically send via Authorization header • Don‘t create tokens on your own • Use certified Security Token Services • https://identityserver.github.io • https://github.com/panva/node-oidc-provider
  74. Token-based authentication Client Server Username & Password 1. check 2.

    generate token Token Store token Token Validate & execute
  75. Exercise: Middlewares + Security • Goal: Add Logger-Middleware and Security-Middleware

    • Branch: 06_middlware_security • Files: Logger.js, Security.js, Server.js • Test in our app + Postman • Don‘t forget to set the Authorization header in Postman
  76. Real time web • Server pushes data directly to the

    client • Use Cases • Chat • Stock info / Newsticker • Progress information • Collaboration tools • Legacy • Polling / Long polling
  77. Web Sockets • Bi-Directional • Server pushes • Client pushes

    • Use Cases • Collaboration tools • Chat • New protocol • Persistent connections
  78. Web Sockets

  79. Web Sockets & Node === Easy • Usage via libraries

    • ws • socket.io const webSocket = require('ws'); const wss = new webSocket.Server({server}); wss.on('connection', ws => { const interval = setInterval(() => { const stockData = getStockData(); ws.send(JSON.stringify(stockData)); }, 1000); ws.on('close', () => { clearInterval(interval); }); });
  80. EventEmitter • Node === event driven • Sync & Async

    events • Custom events via EventEmitter-Class • Loose coupling of modules • Synchronous! • Don’t forget to “removeListener” const readable = getReadableStreamSomehow(); readable.on('data', (chunk) => {…}); readable.on('end', () => {…}); const events = require('events'); const eventEmitter = new events.EventEmitter(); eventEmitter.addListener('myEvent', data => console.log(data)); //... somewhere else eventEmitter.emit('myEvent', someData);
  81. Exercise: Server push • Goal: Add websocket support • Use

    EventEmitter • emit event when tweet created • consume event and send via Web Socket • Branch: 07_sockets • Files: TweetService.js, Server.js • Test in our app • Open in two browser windows and tweet something J • Need help? https://github.com/websockets/ws
  82. Databases Every business application needs a database, right?

  83. Databases • Plays well with different kind of databases •

    NoSQL databases like MongoDB, CouchDB • Relational databases like PostgresSQL, MSSQL • Real time databases like RethinkDB • Access databases directly or via ORM • SequelizeJS • TypeORM
  84. RethinkDB • Open Source, scalable JSON real time database •

    Can push updated query results directly to the application • Use Cases are all real time based applications • Multiplayer games • Streaming analytics • Collaborative tools • Don’t use it, if you need full ACID or strong schemas
  85. RethinkDB – API • RethinkDB simple queries • All queries

    return promises! const r = require('rethinkdb'); await r.table('MY_TABLE').get('1').run(connection); // get by id await r.table('MY_TABLE').count().run(connection); // count stuff
  86. Exercise: Async-Programing + RethinkDB • Goal: Add RethinkDB support •

    Do some async programming • Use RethinkDB-API • Branch: 08_database • Files: index.js, TweetServicejs, TweetResource.js • Test in our app • Data will survive server restart J • Automatic real time support • Need help? https://rethinkdb.com/docs/nodejs/
  87. Testing Does it work? Really?

  88. Testing APIs • E2E-Tests • Slow & too much effort

    • Integration tests • Test API only • Mock services / Database • Check: • Status codes • Response (JSON) • Validation
  89. SuperTest describe(’MyResource', () => { beforeEach(() => { someService.doStuff.mockImplementation(() =>

    ({ id: 1, name: 'test'})); }); it(’a get request', async done => { const response = await supertest(server.getServer()).get('/somepath'); expect(response.status).toBe(200); done(); // tell SuperTest we’re done }); });
  90. Exercise: Test our API • Goal: Test our API •

    See some mocking • Test responses (status codes and body) • Branch: 09_apitest • Files: TweetResourceTest.js • npm test • Need help? https://github.com/visionmedia/supertest
  91. But wait: There’s more!

  92. A public API needs more… • Reduce network traffic •

    Caching • GZIP • HATEOAS • Location-Header for POST-Methods • Link-Header for pagination • Rate limiting, Throttling • Versioning
  93. Reduce traffic • Tell client that data did not change

    • Code 304 • Two standardized options: • Last Modified-Header • Did data change since… • Etag-Header • Did checksum of data change? GET /tweets OK 200 Etag: 7982882299 GET /tweets If-None-Match: 7982882299 304: NOT MODIFIED following request initial request
  94. Use GZIP • Supported by all modern browsers • Supported

    by most servers • Reduces traffic by up to 70% • But • more computation • is not always smaller
  95. HATEOAS • Hypermedia as the engine of application state •

    Navigate from anywhere to everywhere • Usage of API without prior knowledge
  96. HATEOAS • Example: POST • Example: Pagination POST /tweets {…

    ------------------------------------------------------------------------- CREATED 201 Location: http://twttr.de/tweets/123 {…} GET /tweets/?page=1&size=10 HEADER Link: <http://twttr.de/tweets/?page=2&size=10>; rel="next”>, <http://twttr.de/tweets/?page=9&size=10 >; rel="last”> BODY [{…},{…}]
  97. Rate limiting (Throttling) • Public APIs • DDoS protection •

    Pay per request APIs • Rates for „expensive“ resources • Rates per timeframe • Hour, Minute, Second • How? • IP-based • Account-based
  98. Versioning • via URL (twttr.com/v1/tweets) • via Accept-Header • via

    Custom-Header • Restify (Accept-Version) GET http:twttr.com/tweets/ Accept-Version: 1.0.0 200 OK [{tweet:”…”},{tweet:”…”}] GET http:twttr.com/tweets/ Accept-Version: 3.0.0 406 NOT ACCEPTABLE const server = restify.createServer({ name: 'Twttr', version: '1.0.0' });
  99. Exercise: API Restify • Optimize our HTTP API • Add

    Versioning, Throttling and GZIP support • Set the Accept-Version header in postman to test this • Add response headers (Location for POST, Link for pagination) • Implement conditional requests with ETAG (for single tweet retrieval) • Set the If-None-Match header in postman to test this • Branch: 10_api_restify • Files • Server.js, TweetResource.js, HttpHelper.js
  100. Summary

  101. Summary • Modern SPAs needs modern backends • A backend

    can be orchestrated with several technologies/languages • Node.js is one more possibility to write Web APIs • Web APIs should be use case centric – not pure REST • Leverage existing Node.js modules for kickstarting Web APIs • Restify sets up a base for writing Web APIs • Make usage of real time Web Sockets communication • RethinkDB is one possibility for databases in Node.js • Don’t forget gzip, versioning, rate limiting and security
  102. Product Placement https://entwickler.de/press/shortcuts/die-serverwelt-von-node-js-2-262553.html

  103. Thank you! Questions? Repository https://github.com/dskgry/api-summit-2017-nodejs-workshop Sven Kölpin sven.koelpin@openknowledge.de @dskgry Manuel

    Rauber manuel.rauber@thinktecture.com @manuelrauber