Tweet
Tweet

Slide 1

Slide 1 text

Scalable REST API in Node.js

Slide 2

Slide 2 text

Who am I - Hi, I’m Mehdi Hasan Khan ! - Currently Software Architect @ ShopUp 
 We’re a JavaScript shop and we are hiring - Writing JavaScript professionally for 9 years - Developer of Avro Keyboard - Dad of a wonderful kid
 
 Twitter: @MehdiHK GitHub: https://github.com/mugli LinkedIn: https://www.linkedin.com/in/mehdihk/

Slide 3

Slide 3 text

What I DON’T mean when I talk about scaling

Slide 4

Slide 4 text

Lies, Damned Lies and Benchmarks

Slide 5

Slide 5 text

In real world your database and other backend services are probably the bottleneck. Not the HTTP Framework.

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Non-specific scalability promise is a dangerous myth

Slide 8

Slide 8 text

What do I mean when I talk about scaling

Slide 9

Slide 9 text

Growth Team size Code Resource requirement

Slide 10

Slide 10 text

Common API Life Cycle • Build • Design • Develop • Document • Test • Use • Access Control • Data Access • Run • Scale with traffic • Availability • Monitoring

Slide 11

Slide 11 text

This talk focuses on the 1st step

Slide 12

Slide 12 text

Build

Slide 13

Slide 13 text

Design Develop Document Test

Slide 14

Slide 14 text

Design & Document Build

Slide 15

Slide 15 text

Why design before develop?

Slide 16

Slide 16 text

Most badly designed REST APIs aren't bad because they were developed by bad coders. It's because they were not designed in the first place, they were just developed.

Slide 17

Slide 17 text

JUST WRITE IT. require('express')

Slide 18

Slide 18 text

As the flexibility of a system increases, its usability decreases Flexibility-Usability Tradeoff Universal Principles of Design

Slide 19

Slide 19 text

Pragmatic REST 101 https://www.slideshare.net/apigee/rest-design-webinar

Slide 20

Slide 20 text

https://pages.apigee.com/rs/apigee/images/api-design-ebook-2012-03.pdf

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

• Include .json extension or not in endpoints? • CamelCase or snake _case? • Version in the url or header or param? • Verbs are bad? How do I make search endpoints? • Should GET requests have body? • Do you even paginate bro? Cursor or offset? • Formatted output or minified? • How do I avoid over-fetching?

Slide 24

Slide 24 text

If you read enough advice, everything will cancel each other out

Slide 25

Slide 25 text

(╯°□°)╯︵ ┻━┻

Slide 26

Slide 26 text

What everyone is asking for is Consistency

Slide 27

Slide 27 text

Consistency reduces cognitive load

Slide 28

Slide 28 text

“The competent programmer is fully aware of the limited size of his own skull” Edsger Dijkstra

Slide 29

Slide 29 text

Think of API as UI for other devs

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Think of API as a contract

Slide 32

Slide 32 text

Tell me exactly what’s accepted

Slide 33

Slide 33 text

Tell me exactly what I can expect

Slide 34

Slide 34 text

Promise me you won’t change them without notice

Slide 35

Slide 35 text

The case for schema first API

Slide 36

Slide 36 text

Don’t make me think (╯°□°)╯︵ ┻━┻

Slide 37

Slide 37 text

Tech Debt That Hurts

Slide 38

Slide 38 text

Don’t make me think (╯°□°)╯︵ ┻━┻

Slide 39

Slide 39 text

Better: hapi and joi

Slide 40

Slide 40 text

Better: hapi and joi • Correct status code • Auto-generated error message • Consistent error object • Declarative syntax, no ugly regex hacks, type checks • What’s expected is immediately visible • Makes refactoring easy for other API developers • Can be used to check response object too before they are send to the client! 
 (Not shown in the example)

Slide 41

Slide 41 text

Constraints set us free

Slide 42

Slide 42 text

Bonus: Zero effort living documentation!

Slide 43

Slide 43 text

Documentation by hand gets outdated, API should be the source of truth

Slide 44

Slide 44 text

Living documentation

Slide 45

Slide 45 text

For schema definition, configuration is better than code

Slide 46

Slide 46 text

Using Joi with Express npm install celebrate

Slide 47

Slide 47 text

Beyond Joi

Slide 48

Slide 48 text

https://json-schema.org/

Slide 49

Slide 49 text

Beyond Joi and JavaScript Ecosystem

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Fastify with JSON Schema

Slide 52

Slide 52 text

Same benefit: Zero effort living documentation

Slide 53

Slide 53 text

..and, wait for it…

Slide 54

Slide 54 text

You can generate code from the spec too!

Slide 55

Slide 55 text

Reverse: You can generate code from the spec too!

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

Can’t we have both for API?

Slide 59

Slide 59 text

Enter OpenAPI

Slide 60

Slide 60 text

The OpenAPI Specification defines a standard, programming language-agnostic interface description for REST APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic.

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

Sample Spec

Slide 63

Slide 63 text

Quick start: Express & OpenAPI 3 https://developer.ibm.com/recipes/tutorials/builds-apis-with-node-js-express-and- openapi-3/ npm i express-openapi-validator

Slide 64

Slide 64 text

https://openapi.tools/ Benefit of having spec that targets both humans and machines

Slide 65

Slide 65 text

No content

Slide 66

Slide 66 text

Nope

Slide 67

Slide 67 text

Don’t get overwhelmed by tools

Slide 68

Slide 68 text

Think from the first principles

Slide 69

Slide 69 text

Think of API as UI for other devs

Slide 70

Slide 70 text

Think of API as a contract

Slide 71

Slide 71 text

And then

Slide 72

Slide 72 text

Develop Build

Slide 73

Slide 73 text

Express: Order based routing

Slide 74

Slide 74 text

⚠ Be aware of route orders

Slide 75

Slide 75 text

It gets worse with middleware ordering

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

Compare that to hapi

Slide 78

Slide 78 text

• hapi searches paths in order from most specific to least specific. • If you have two routes, /filename.jpg and /filename.{ext} a request to /filename.jpg will match the first route, and not the second, whatever their order is in code. • That also mean, a route with the path /{files*} will be the last route tested, and will only match if all other routes fail.

Slide 79

Slide 79 text

• hapi has deterministic routing. Each request can only map to one route, and its routing table will be the same every time you start the server. • As the application size and teams grow, routing conflicts become more of a concern. You want a banana but you get a gorilla holding the banana and the entire jungle. • If you have two routes that conflict, hapi will show an error on startup, providing details on the routes that conflict, making it much easier to debug and fix. This is much better than spending hours debugging this at runtime.

Slide 80

Slide 80 text

Caching

Slide 81

Slide 81 text

hapi provides powerful caching capabilities

Slide 82

Slide 82 text

Test Build

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

Express and supertest

Slide 85

Slide 85 text

You want to test the route • You don’t need to spawn the whole server • You don’t need to listen to a socket • You don’t even need to send a real http call and process it • Because you are not testing if the underlying http stack is working in Node.js, you are testing if you pass some values to your endpoint, if it returns correct result

Slide 86

Slide 86 text

Both fastify and hapi support http injection

Slide 87

Slide 87 text

fastify.inject

Slide 88

Slide 88 text

server.inject in hapi

Slide 89

Slide 89 text

Final thoughts • Express is minimal and optimized for quick learning and prototyping. Unfortunately it became the most popular framework in Node.js ecosystem for the same reason. There are better options though. • Hapi brought a lot of battle tested and production ready practices to Node.js • Fastify had the luxury of coming later and learnt from both of them. • If you are stuck with existing express based api, there are middlewares that tries to make it better.

Slide 90

Slide 90 text

Thank you and keep building awesome stuff!