From Node.js to Design Patterns - BuildPiper

A design pattern provides a general reusable solution for the common problems that occur in software design. The pattern typically shows relationships and interactions between classes or objects. The idea is to speed up the development process by providing well-tested, proven development/design paradigms. Design patterns come in three different categories. Creational patterns include the generation of object instances. Structure refers to how an object is made and how things behave and interact In this Webinar(Live Meetup) we will be covering - What is node js - When to use node js - Async I/O operations in node js - Advantages of Async/Await - Some interesting - async patterns - Performance comparison

Transcript

2. $ ~ whoami 👋 I'm Luciano ( 🍕🍝) Senior Architect @ fourTheorem (Dublin ) nodejsdp.link 📔 Co-Author of Node.js Design Patterns 👉 Let's connect! (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 2
3. Always re-imagining We are a pioneering technology consultancy focused on AWS and serverless | | Accelerated Serverless AI as a Service Platform Modernisation loige ✉ Reach out to us at 😇 We are always looking for talent: hello@fourTheorem.com fth.link/careers 3
4. We host a weekly podcast about AWS awsbites.com loige 4
5. Fact: Async JavaScript is tricky! callbacks promises Async/Await async generators streams event emitters util.promisify() Promise.all() Promise.allSettled() 😱 loige 5
6. Agenda Async WUT?! Callbacks Promises Async / Await async Patterns Mixed style async A performance trick! loige 6
7. What does async even mean? In JavaScript and in Node.js, input/output operations are non- blocking. Classic examples: reading the content of a file, making an HTTP request, loading data from a database, etc. loige 7
8. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 8
9. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 9
10. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 10
11. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 11
12. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout (done) loige 12
13. Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout (done) (done) loige 13
14. Non-blocking I/O is convenient: you can do work while waiting for I/O! But, what if we need to do something when the I/O operation completes? loige 14
15. Once upon a time there were... Callbacks loige 15
16. Anatomy of callback-based non-blocking code doSomethingAsync(arg1, arg2, cb) This is a callback loige 16
17. doSomethingAsync(arg1, arg2, (err, data) => { // ... do something with data }) You are defining what happens when the I/O operations completes (or fails) with a function. doSomethingAsync will call that function for you! loige Anatomy of callback-based non-blocking code 17
18. doSomethingAsync(arg1, arg2, (err, data) => { if (err) { // ... handle error return } // ... do something with data }) Always handle errors first! loige Anatomy of callback-based non-blocking code 18
19. An example Fetch the latest booking for a given user If it exists print it loige 19
20. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 An example loige 20
21. A more realistic example Fetch the latest booking for a given user If it exists, cancel it If it was already paid for, refund the user loige 21
22. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 cancelBooking(booking.id, (err) => { 9 if (err) { 10 console.error(err) 11 return 12 } 13 14 15 16 17 18 19 20 21 22 23 24 25 }) 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 cancelBooking(booking.id, (err) => { 9 if (err) { 10 console.error(err) 11 return 12 } 13 14 if (booking.paid) { 15 console.log('Booking was paid, refunding the user') 16 17 18 19 20 21 22 23 24 } 25 }) 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 loige 22
23. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 loige 23
24. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 loige THE PIRAMID OF DOOM (or callback hell 🔥) 24
25. Some times, just refactoring the code can help... loige 25
26. function cancelAndRefundBooking(booking, cb) { cancelBooking(booking.id, (err) => { if (err) { return cb(err) } if (!booking.paid) { return cb(null, {refundedAmount: 0}) } refundUser(booking.userId, booking.paidAmount, (err) => { if (err) { return cb(err) } return cb(null, {refundedAmount: booking.paidAmount}) }) }) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 loige 26
27. getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { cancelAndRefundBooking(booking, (err, result) => { if (err) { console.error(err) return } console.log(`Booking cancelled (${result.refundedAmount} refunded)`) }) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 27
28. 😟 Is this the best we can do? loige 28
29. Let's talk about Promise loige 29
30. With callbacks we are not in charge! We need to trust that the async function will call our callbacks when the async work is completed! loige 30
31. Promise help us to be more in control! const promiseObj = doSomethingAsync(arg1, arg2) An object that represents the status of the async operation loige 31
32. const promiseObj = doSomethingAsync(arg1, arg2) A promise object is a tiny state machine with 2 possible states pending (still performing the async operation) settled (completed) ✅ fullfilled (witha value) 🔥 rejected (with an error) loige Promise help us to be more in control! 32
33. const promiseObj = doSomethingAsync(arg1, arg2) promiseObj.then((data) => { // ... do something with data }) loige Promise help us to be more in control! 33
34. const promiseObj = doSomethingAsync(arg1, arg2) promiseObj.then((data) => { // ... do something with data }) promiseObj.catch((err) => { // ... handle errors } loige Promise help us to be more in control! 34
35. Promises can be chained ⛓ This solves the pyramid of doom problem! doSomethingAsync(arg1, arg2) .then((result) => doSomethingElseAsync(result)) .then((result) => doEvenMoreAsync(result) .then((result) => keepDoingStuffAsync(result)) .catch((err) => { /* ... */ }) 35 loige
36. Promises can be chained ⛓ This solves the pyramid of doom problem! doSomethingAsync(arg1, arg2) .then((result) => doSomethingElseAsync(result)) // ... .catch((err) => { /* ... */ }) .finally(() => { /* ... */ }) loige 36
37. How to create a promise new Promise ((resolve, reject) => { // ... }) loige 37
38. How to create a promise new Promise ((resolve, reject) => { // ... do something async // reject(someError) // resolve(someValue) }) loige 38
39. How to create a promise Promise.resolve('SomeValue') Promise.reject(new Error('SomeError')) loige 39
40. How to create a promise (example) function queryDB(client, query) { return new Promise((resolve, reject) => { client.executeQuery(query, (err, data) => { if (err) { return reject(err) } resolve(data) }) }) } 1 2 3 4 5 6 7 8 9 10 11 loige 40
41. How to create a promise (example) queryDB(dbClient, 'SELECT * FROM bookings') .then((data) => { // ... do something with data }) .catch((err) => { console.error('Failed to run query', err) }) .finally(() => { dbClient.disconnect() }) 1 2 3 4 5 6 7 8 9 10 queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .then((data) => { // ... do something with data }) queryDB(dbClient, 'SELECT * FROM bookings') 1 2 3 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .catch((err) => { console.error('Failed to run query', err) }) queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 5 6 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .finally(() => { dbClient.disconnect() }) queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 8 9 10 queryDB(dbClient, 'SELECT * FROM bookings') .then((data) => { // ... do something with data }) .catch((err) => { console.error('Failed to run query', err) }) .finally(() => { dbClient.disconnect() }) 1 2 3 4 5 6 7 8 9 10 loige 41
42. Let's re-write our example with Promise Fetch the latest booking for a given user If it exists, cancel it If it was already paid for, refund the user loige 42
43. getLatestBooking(userId) .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) .then((refund) => { if (refund) { console.log('User refunded') } }) .catch((err) => { console.error(err) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) getLatestBooking(userId) 1 2 3 4 5 6 7 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 9 10 11 12 13 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((refund) => { if (refund) { console.log('User refunded') } }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 15 16 17 18 19 .catch((err) => { 20 console.error(err) 21 }) 22 .catch((err) => { console.error(err) }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 20 21 22 getLatestBooking(userId) .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) .then((refund) => { if (refund) { console.log('User refunded') } }) .catch((err) => { console.error(err) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 loige 43
44. enters... Async/Await loige 44
45. Sometimes, we just want to wait for a promise to resolve before executing the next line... const promiseObj = doSomethingAsync(arg1, arg2) const data = await promiseObj // ... process the data await allows us to do exactly that loige 45
46. const data = await doSomethingAsync(arg1, arg2) // ... process the data We don't have to assign the promise to a variable to use await Sometimes, we just want to wait for a promise to resolve before executing the next line... loige 46
47. try { const data = await doSomethingAsync(arg1, arg2) // ... process the data } catch (err) { // ... handle error } Unified error handling If we await a promise that eventually rejects we can capture the error with a regular try/catch block loige 47
48. Async functions async function doSomethingAsync(arg1, arg2) { // ... } special keyword that marks a function as async loige 48
49. Async functions async function doSomethingAsync(arg1, arg2) { return 'SomeValue' } function doSomethingAsync(arg1, arg2) { return Promise.resolve('SomeValue') } loige 49
50. Async functions async function doSomethingAsync(arg1, arg2) { throw new Error('SomeError') } function doSomethingAsync(arg1, arg2) { return Promise.reject(new Error('SomeError')) } loige 50
51. Async functions async function doSomethingAsync(arg1, arg2) { const res1 = await doSomethingElseAsync() const res2 = await doEvenMoreAsync(res1) const res3 = await keepDoingStuffAsync(res2) // ... } inside an async function you can use await to suspend the execution until the awaited promise resolves loige 51
52. Async functions async function doSomethingAsync(arg1, arg2) { const res = await doSomethingElseAsync() if (res) { for (const record of res1.records) { await updateRecord(record) } } } Async functions make it very easy to write code that manages asynchronous control flow loige 52
53. Let's re-write our example with async/await Fetch the latest booking for a given user If it exists, cancel it If it was already paid for, refund the user loige 53
54. async function cancelLatestBooking(userId) { const booking = await getLatestBooking(userId) if (!booking) { console.log(`No booking found for user ${userId}`) return } console.log(`Found booking for user ${userId}`, booking) await cancelBooking(booking.id) if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 async function cancelLatestBooking(userId) { } 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 18 const booking = await getLatestBooking(userId) async function cancelLatestBooking(userId) { 1 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 if (!booking) { console.log(`No booking found for user ${userId}`) return } async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 4 5 6 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 console.log(`Found booking for user ${userId}`, booking) async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 await cancelBooking(booking.id) async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 13 14 15 16 17 } 18 async function cancelLatestBooking(userId) { const booking = await getLatestBooking(userId) if (!booking) { console.log(`No booking found for user ${userId}`) return } console.log(`Found booking for user ${userId}`, booking) await cancelBooking(booking.id) if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 loige 54
55. Mini summary Async/Await generally helps to keep the code simple & readable To use Async/Await you need to understand Promise To use Promise you need to understand callbacks callbacks → Promise → async/await Don't skip any step of the async journey! loige 55
56. Async Patterns ❇ loige 56
57. Sequential execution const users = ['Peach', 'Toad', 'Mario', 'Luigi'] for (const userId of users) { await cancelLatestBooking(userId) } 1 2 3 4 5 const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 for (const userId of users) { 3 await cancelLatestBooking(userId) 4 } 5 for (const userId of users) { } const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 3 await cancelLatestBooking(userId) 4 5 await cancelLatestBooking(userId) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 for (const userId of users) { 3 4 } 5 const users = ['Peach', 'Toad', 'Mario', 'Luigi'] for (const userId of users) { await cancelLatestBooking(userId) } 1 2 3 4 5 loige 57
58. Sequential execution (gotcha!) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] users.forEach(async (userId) => { await cancelLatestBooking(userId) }) 1 2 3 4 5 loige ⚠ Don't do this with Array.map() or Array.forEach() Array.forEach() will run the provided function without awaiting for the returned promise, so all the invocation will actually happen concurrently! 58
59. Concurrent execution (Promise.all) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] await Promise.all( users.map( userId => cancelLatestBooking(userId) ) ) 1 2 3 4 5 6 7 loige Promise.all() receives a list of promises and it returns a new Promise. This promise will resolve once all the original promises resolve, but it will reject as soon as ONE promise rejects 59
60. Concurrent execution (Promise.allSettled) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] const results = await Promise.allSettled( users.map( userId => cancelLatestBooking(userId) ) ) 1 2 3 4 5 6 7 loige [ { status: 'fulfilled', value: true }, { status: 'fulfilled', value: true }, { status: 'rejected', reason: Error }, { status: 'fulfilled', value: true } ] 60
61. Mixing async styles loige 61
62. You want to use async/await but... you have a callback-based API! 😣 loige 62
63. Node.js offers promise-based alternative APIs Callback-based Promise-based setTimeout, setImmediate, setInterval import timers from 'timers/promises' import fs from 'fs' import fs from 'fs/promises' import stream from 'stream' import stream from 'stream/promises' import dns from 'dns' import dns from 'dns/promises' loige 63
64. util.promisify() import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) import { promisify } from 'util' const gzipPromise = promisify(gzip) const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 const gzipPromise = promisify(gzip) 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 7 import { promisify } from 'util' import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 3 const gzipPromise = promisify(gzip) 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 7 const gzipPromise = promisify(gzip) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 7 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 const gzipPromise = promisify(gzip) 4 5 6 7 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) import { promisify } from 'util' const gzipPromise = promisify(gzip) const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 loige 64
65. Promisify by hand 🖐 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) function gzipPromise (buffer, options) { return new Promise((resolve, reject) => { gzip(buffer, options, (err, gzippedData) => { if (err) { return reject(err) } resolve(gzippedData) }) }) } const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 function gzipPromise (buffer, options) { } import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 }) 12 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 return new Promise((resolve, reject) => { }) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 gzip(buffer, options, (err, gzippedData) => { }) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 if (err) { return reject(err) } import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 6 7 8 9 resolve(gzippedData) 10 }) 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 resolve(gzippedData) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 10 }) 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 16 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 }) 12 } 13 14 15 16 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) function gzipPromise (buffer, options) { return new Promise((resolve, reject) => { gzip(buffer, options, (err, gzippedData) => { if (err) { return reject(err) } resolve(gzippedData) }) }) } const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // <Buffer 1f 8b 08 00 00 00 00 ... 00 00 00> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 65
66. What if we we want to do the opposite? 🤷 Convert a promise-based function to a callback-based one loige 66
67. var env = nunjucks.configure('views') env.addFilter('videoTitle', function(videoId, cb) { // ... fetch the title through youtube APIs // ... extract the video title // ... and call the callback with the title }, true) 1 2 3 4 5 6 7 OK, this is not a common use case, so let me give you a real example! Nunjucks async filters {{ data | myCustomFilter }} We are forced to pass a callback-based function here! Ex: {{ youtubeId | videoTitle }} loige 67
68. util.callbackify() import { callbackify } from 'util' import Innertube from 'youtubei.js' // from npm async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } const videoTitleFilterCb = callbackify(videoTitleFilter) videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 import Innertube from 'youtubei.js' // from npm import { callbackify } from 'util' 1 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 4 5 6 7 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 const videoTitleFilterCb = callbackify(videoTitleFilter) import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 12 13 14 15 16 17 18 19 import { callbackify } from 'util' import Innertube from 'youtubei.js' // from npm async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } const videoTitleFilterCb = callbackify(videoTitleFilter) videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 loige 68
69. Callbackify by hand ✋ import Innertube from 'youtubei.js' // from npm async function videoTitleFilter (videoId) { // ... } function videoTitleFilterCb (videoId, cb) { videoTitleFilter(videoId) .then((videoTitle) => cb(null, videoTitle)) .catch(cb) } videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { // ... }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 function videoTitleFilterCb (videoId, cb) { videoTitleFilter(videoId) .then((videoTitle) => cb(null, videoTitle)) .catch(cb) } import Innertube from 'youtubei.js' // from npm 1 2 async function videoTitleFilter (videoId) { 3 // ... 4 } 5 6 7 8 9 10 11 12 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 13 // ... 14 }) 15 loige 69
70. OK, Cool! But is this stuff worth it? 🧐 loige 70
71. Let me show you a cool performance trick for Web Servers! 😎 loige 71
72. The request batching pattern one user /api/hotels/rome Web server DB loige 72
73. The request batching pattern multiple users (no batching) Web server DB /api/hotels/rome /api/hotels/rome /api/hotels/rome 73 loige
74. The request batching pattern multiple users (with batching!) Web server DB /api/hotels/rome /api/hotels/rome /api/hotels/rome 74 📘 Requests in-flight /api/hotels/rome ✅ loige
75. The web server import { createServer } from 'http' const urlRegex = /^\/api\/hotels\/([\w-]+)$/ createServer(async (req, res) => { const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) if (!matches) { res.writeHead(404, 'Not found') return res.end() } const [_, city] = matches const hotels = await getHotelsForCity(city) res.writeHead(200) res.end(JSON.stringify({ hotels })) }).listen(8000) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ import { createServer } from 'http' 1 2 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 createServer(async (req, res) => { }).listen(8000) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 19 const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 6 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 if (!matches) { res.writeHead(404, 'Not found') return res.end() } import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 9 10 11 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const [_, city] = matches import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const hotels = await getHotelsForCity(city) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 res.writeHead(200) res.end(JSON.stringify({ hotels })) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 17 18 }).listen(8000) 19 import { createServer } from 'http' const urlRegex = /^\/api\/hotels\/([\w-]+)$/ createServer(async (req, res) => { const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) if (!matches) { res.writeHead(404, 'Not found') return res.end() } const [_, city] = matches const hotels = await getHotelsForCity(city) res.writeHead(200) res.end(JSON.stringify({ hotels })) }).listen(8000) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 loige 75
76. The data fetching function (with batching) let pendingRequests = new Map() function getHotelsForCity (cityId) { if (pendingRequests.has(cityId)) { return pendingRequests.get(cityId) } const asyncOperation = db.query({ text: 'SELECT * FROM hotels WHERE cityid = $1', values: [cityId], }) pendingRequests.set(cityId, asyncOperation) asyncOperation.finally(() => { pendingRequests.delete(cityId) }) return asyncOperation } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 loige 76
77. Benchmarks loige.link/req-batch-bench Without request batching With request batching (+90% avg req/sec)* * This is an artificial benchmark and results might vary significantly in real-life scenarios. Always run your own benchmarks before deciding whether this optimization can have a positive effect for you. loige 77
78. Closing Notes JavaScript can be a very powerful and convenient language when we have to deal with a lot of I/O (e.g. web servers) The async story has evolved a lot in the last 10-15 years: new patterns and language constructs have emerged Async/Await is probably the best way to write async code today To use Async/Await correctly you need to understand Promise and callbacks Take your time and invest in learning the fundamentals loige 78
79. Cover Picture by on Marc-Olivier Jodoin Unsplash THANKS! 🙌 ❤ nodejsdp.link loige Grab these slides! 😍 Grab the book 79