1

$ ~ whoami 👋 I'm Luciano ( 🍕🍝) Senior Architect @ fourTheorem (Dublin ) nodejsdp.link 📔 Co-Author of Node.js Design Patterns 👉 Let's connect! (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 2

Always re-imagining We are a pioneering technology consultancy focused on AWS and serverless | | Accelerated Serverless AI as a Service Platform Modernisation loige ✉ Reach out to us at 😇 We are always looking for talent: [email protected] fth.link/careers 3

We host a weekly podcast about AWS awsbites.com loige 4

Fact: Async JavaScript is tricky! callbacks promises Async/Await async generators streams event emitters util.promisify() Promise.all() Promise.allSettled() 😱 loige 5

Agenda Async WUT?! Callbacks Promises Async / Await async Patterns Mixed style async A performance trick! loige 6

What does async even mean? In JavaScript and in Node.js, input/output operations are non- blocking. Classic examples: reading the content of a file, making an HTTP request, loading data from a database, etc. loige 7

Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 8

Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 9

Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 10

Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout loige 11

Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout (done) loige 12

Blocking style vs JavaScript Blocking style JavaScript 1. Assign a variable 2. Read data from a file 3. Print to stdout 1. Assign a variable 2. Read data from a file 3. Print to stdout (done) (done) loige 13

Non-blocking I/O is convenient: you can do work while waiting for I/O! But, what if we need to do something when the I/O operation completes? loige 14

Once upon a time there were... Callbacks loige 15

Anatomy of callback-based non-blocking code doSomethingAsync(arg1, arg2, cb) This is a callback loige 16

doSomethingAsync(arg1, arg2, (err, data) => { // ... do something with data }) You are defining what happens when the I/O operations completes (or fails) with a function. doSomethingAsync will call that function for you! loige Anatomy of callback-based non-blocking code 17

doSomethingAsync(arg1, arg2, (err, data) => { if (err) { // ... handle error return } // ... do something with data }) Always handle errors first! loige Anatomy of callback-based non-blocking code 18

An example Fetch the latest booking for a given user If it exists print it loige 19

getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 An example loige 20

A more realistic example Fetch the latest booking for a given user If it exists, cancel it If it was already paid for, refund the user loige 21

getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 cancelBooking(booking.id, (err) => { 9 if (err) { 10 console.error(err) 11 return 12 } 13 14 15 16 17 18 19 20 21 22 23 24 25 }) 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) getLatestBooking(userId, (err, booking) => { 1 if (err) { 2 console.error(err) 3 return 4 } 5 6 if (booking) { 7 console.log(`Found booking for user ${userId}`, booking) 8 cancelBooking(booking.id, (err) => { 9 if (err) { 10 console.error(err) 11 return 12 } 13 14 if (booking.paid) { 15 console.log('Booking was paid, refunding the user') 16 17 18 19 20 21 22 23 24 } 25 }) 26 } else { 27 console.log(`No booking found for user ${userId}`) 28 } 29 }) 30 loige 22

getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 loige 23

getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { console.log(`Found booking for user ${userId}`, booking) cancelBooking(booking.id, (err) => { if (err) { console.error(err) return } if (booking.paid) { console.log('Booking was paid, refunding the user') refundUser(userId, booking.paidAmount, (err) => { if (err) { console.error(err) return } console.log('User refunded') }) } }) } else { console.log(`No booking found for user ${userId}`) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 loige THE PIRAMID OF DOOM (or callback hell 🔥) 24

Some times, just refactoring the code can help... loige 25

function cancelAndRefundBooking(booking, cb) { cancelBooking(booking.id, (err) => { if (err) { return cb(err) } if (!booking.paid) { return cb(null, {refundedAmount: 0}) } refundUser(booking.userId, booking.paidAmount, (err) => { if (err) { return cb(err) } return cb(null, {refundedAmount: booking.paidAmount}) }) }) } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 loige 26

getLatestBooking(userId, (err, booking) => { if (err) { console.error(err) return } if (booking) { cancelAndRefundBooking(booking, (err, result) => { if (err) { console.error(err) return } console.log(`Booking cancelled (${result.refundedAmount} refunded)`) }) } }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 loige 27

😟 Is this the best we can do? loige 28

Let's talk about Promise loige 29

With callbacks we are not in charge! We need to trust that the async function will call our callbacks when the async work is completed! loige 30

Promise help us to be more in control! const promiseObj = doSomethingAsync(arg1, arg2) An object that represents the status of the async operation loige 31

const promiseObj = doSomethingAsync(arg1, arg2) A promise object is a tiny state machine with 2 possible states pending (still performing the async operation) settled (completed) ✅ fullfilled (witha value) 🔥 rejected (with an error) loige Promise help us to be more in control! 32

const promiseObj = doSomethingAsync(arg1, arg2) promiseObj.then((data) => { // ... do something with data }) loige Promise help us to be more in control! 33

const promiseObj = doSomethingAsync(arg1, arg2) promiseObj.then((data) => { // ... do something with data }) promiseObj.catch((err) => { // ... handle errors } loige Promise help us to be more in control! 34

Promises can be chained ⛓ This solves the pyramid of doom problem! doSomethingAsync(arg1, arg2) .then((result) => doSomethingElseAsync(result)) .then((result) => doEvenMoreAsync(result) .then((result) => keepDoingStuffAsync(result)) .catch((err) => { /* ... */ }) 35 loige

Promises can be chained ⛓ This solves the pyramid of doom problem! doSomethingAsync(arg1, arg2) .then((result) => doSomethingElseAsync(result)) // ... .catch((err) => { /* ... */ }) .finally(() => { /* ... */ }) loige 36

How to create a promise new Promise ((resolve, reject) => { // ... }) loige 37

How to create a promise new Promise ((resolve, reject) => { // ... do something async // reject(someError) // resolve(someValue) }) loige 38

How to create a promise Promise.resolve('SomeValue') Promise.reject(new Error('SomeError')) loige 39

How to create a promise (example) function queryDB(client, query) { return new Promise((resolve, reject) => { client.executeQuery(query, (err, data) => { if (err) { return reject(err) } resolve(data) }) }) } 1 2 3 4 5 6 7 8 9 10 11 loige 40

How to create a promise (example) queryDB(dbClient, 'SELECT * FROM bookings') .then((data) => { // ... do something with data }) .catch((err) => { console.error('Failed to run query', err) }) .finally(() => { dbClient.disconnect() }) 1 2 3 4 5 6 7 8 9 10 queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .then((data) => { // ... do something with data }) queryDB(dbClient, 'SELECT * FROM bookings') 1 2 3 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .catch((err) => { console.error('Failed to run query', err) }) queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 5 6 7 .finally(() => { 8 dbClient.disconnect() 9 }) 10 .finally(() => { dbClient.disconnect() }) queryDB(dbClient, 'SELECT * FROM bookings') 1 .then((data) => { 2 // ... do something with data 3 }) 4 .catch((err) => { 5 console.error('Failed to run query', err) 6 }) 7 8 9 10 queryDB(dbClient, 'SELECT * FROM bookings') .then((data) => { // ... do something with data }) .catch((err) => { console.error('Failed to run query', err) }) .finally(() => { dbClient.disconnect() }) 1 2 3 4 5 6 7 8 9 10 loige 41

Let's re-write our example with Promise Fetch the latest booking for a given user If it exists, cancel it If it was already paid for, refund the user loige 42

getLatestBooking(userId) .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) .then((refund) => { if (refund) { console.log('User refunded') } }) .catch((err) => { console.error(err) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) getLatestBooking(userId) 1 2 3 4 5 6 7 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 9 10 11 12 13 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 .catch((err) => { 20 console.error(err) 21 }) 22 .then((refund) => { if (refund) { console.log('User refunded') } }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 15 16 17 18 19 .catch((err) => { 20 console.error(err) 21 }) 22 .catch((err) => { console.error(err) }) getLatestBooking(userId) 1 .then((booking) => { 2 if (booking) { 3 console.log(`Found booking for user ${userId}`, booking) 4 return cancelBooking(booking.id) 5 } 6 console.log(`No booking found for user ${userId}`) 7 }) 8 .then((cancelledBooking) => { 9 if (cancelledBooking && cancelledBooking.paid) { 10 console.log('Booking was paid, refunding the user') 11 return refundUser(userId, cancelledBooking.paidAmount) 12 } 13 }) 14 .then((refund) => { 15 if (refund) { 16 console.log('User refunded') 17 } 18 }) 19 20 21 22 getLatestBooking(userId) .then((booking) => { if (booking) { console.log(`Found booking for user ${userId}`, booking) return cancelBooking(booking.id) } console.log(`No booking found for user ${userId}`) }) .then((cancelledBooking) => { if (cancelledBooking && cancelledBooking.paid) { console.log('Booking was paid, refunding the user') return refundUser(userId, cancelledBooking.paidAmount) } }) .then((refund) => { if (refund) { console.log('User refunded') } }) .catch((err) => { console.error(err) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 loige 43

enters... Async/Await loige 44

Sometimes, we just want to wait for a promise to resolve before executing the next line... const promiseObj = doSomethingAsync(arg1, arg2) const data = await promiseObj // ... process the data await allows us to do exactly that loige 45

const data = await doSomethingAsync(arg1, arg2) // ... process the data We don't have to assign the promise to a variable to use await Sometimes, we just want to wait for a promise to resolve before executing the next line... loige 46

try { const data = await doSomethingAsync(arg1, arg2) // ... process the data } catch (err) { // ... handle error } Unified error handling If we await a promise that eventually rejects we can capture the error with a regular try/catch block loige 47

Async functions async function doSomethingAsync(arg1, arg2) { // ... } special keyword that marks a function as async loige 48

Async functions async function doSomethingAsync(arg1, arg2) { return 'SomeValue' } function doSomethingAsync(arg1, arg2) { return Promise.resolve('SomeValue') } loige 49

Async functions async function doSomethingAsync(arg1, arg2) { throw new Error('SomeError') } function doSomethingAsync(arg1, arg2) { return Promise.reject(new Error('SomeError')) } loige 50

Async functions async function doSomethingAsync(arg1, arg2) { const res1 = await doSomethingElseAsync() const res2 = await doEvenMoreAsync(res1) const res3 = await keepDoingStuffAsync(res2) // ... } inside an async function you can use await to suspend the execution until the awaited promise resolves loige 51

Async functions async function doSomethingAsync(arg1, arg2) { const res = await doSomethingElseAsync() if (res) { for (const record of res1.records) { await updateRecord(record) } } } Async functions make it very easy to write code that manages asynchronous control flow loige 52

Let's re-write our example with async/await Fetch the latest booking for a given user If it exists, cancel it If it was already paid for, refund the user loige 53

async function cancelLatestBooking(userId) { const booking = await getLatestBooking(userId) if (!booking) { console.log(`No booking found for user ${userId}`) return } console.log(`Found booking for user ${userId}`, booking) await cancelBooking(booking.id) if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 async function cancelLatestBooking(userId) { } 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 18 const booking = await getLatestBooking(userId) async function cancelLatestBooking(userId) { 1 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 if (!booking) { console.log(`No booking found for user ${userId}`) return } async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 4 5 6 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 console.log(`Found booking for user ${userId}`, booking) async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 9 10 await cancelBooking(booking.id) 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 await cancelBooking(booking.id) async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 11 12 if (booking.paid) { 13 console.log('Booking was paid, refunding the user') 14 await refundUser(userId, booking.paidAmount) 15 console.log('User refunded') 16 } 17 } 18 if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } async function cancelLatestBooking(userId) { 1 const booking = await getLatestBooking(userId) 2 3 if (!booking) { 4 console.log(`No booking found for user ${userId}`) 5 return 6 } 7 8 console.log(`Found booking for user ${userId}`, booking) 9 10 await cancelBooking(booking.id) 11 12 13 14 15 16 17 } 18 async function cancelLatestBooking(userId) { const booking = await getLatestBooking(userId) if (!booking) { console.log(`No booking found for user ${userId}`) return } console.log(`Found booking for user ${userId}`, booking) await cancelBooking(booking.id) if (booking.paid) { console.log('Booking was paid, refunding the user') await refundUser(userId, booking.paidAmount) console.log('User refunded') } } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 loige 54

Mini summary Async/Await generally helps to keep the code simple & readable To use Async/Await you need to understand Promise To use Promise you need to understand callbacks callbacks → Promise → async/await Don't skip any step of the async journey! loige 55

Async Patterns ❇ loige 56

Sequential execution const users = ['Peach', 'Toad', 'Mario', 'Luigi'] for (const userId of users) { await cancelLatestBooking(userId) } 1 2 3 4 5 const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 for (const userId of users) { 3 await cancelLatestBooking(userId) 4 } 5 for (const userId of users) { } const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 3 await cancelLatestBooking(userId) 4 5 await cancelLatestBooking(userId) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] 1 2 for (const userId of users) { 3 4 } 5 const users = ['Peach', 'Toad', 'Mario', 'Luigi'] for (const userId of users) { await cancelLatestBooking(userId) } 1 2 3 4 5 loige 57

Sequential execution (gotcha!) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] users.forEach(async (userId) => { await cancelLatestBooking(userId) }) 1 2 3 4 5 loige ⚠ Don't do this with Array.map() or Array.forEach() Array.forEach() will run the provided function without awaiting for the returned promise, so all the invocation will actually happen concurrently! 58

Concurrent execution (Promise.all) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] await Promise.all( users.map( userId => cancelLatestBooking(userId) ) ) 1 2 3 4 5 6 7 loige Promise.all() receives a list of promises and it returns a new Promise. This promise will resolve once all the original promises resolve, but it will reject as soon as ONE promise rejects 59

Concurrent execution (Promise.allSettled) const users = ['Peach', 'Toad', 'Mario', 'Luigi'] const results = await Promise.allSettled( users.map( userId => cancelLatestBooking(userId) ) ) 1 2 3 4 5 6 7 loige [ { status: 'fulfilled', value: true }, { status: 'fulfilled', value: true }, { status: 'rejected', reason: Error }, { status: 'fulfilled', value: true } ] 60

Mixing async styles loige 61

You want to use async/await but... you have a callback-based API! 😣 loige 62

Node.js offers promise-based alternative APIs Callback-based Promise-based setTimeout, setImmediate, setInterval import timers from 'timers/promises' import fs from 'fs' import fs from 'fs/promises' import stream from 'stream' import stream from 'stream/promises' import dns from 'dns' import dns from 'dns/promises' loige 63

util.promisify() import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) import { promisify } from 'util' const gzipPromise = promisify(gzip) const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // 1 2 3 4 5 6 7 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 const gzipPromise = promisify(gzip) 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // 7 import { promisify } from 'util' import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 3 const gzipPromise = promisify(gzip) 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // 7 const gzipPromise = promisify(gzip) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 4 5 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 6 console.log(compressed) // 7 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 import { promisify } from 'util' 2 3 const gzipPromise = promisify(gzip) 4 5 6 7 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) import { promisify } from 'util' const gzipPromise = promisify(gzip) const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // 1 2 3 4 5 6 7 loige 64

Promisify by hand 🖐 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) function gzipPromise (buffer, options) { return new Promise((resolve, reject) => { gzip(buffer, options, (err, gzippedData) => { if (err) { return reject(err) } resolve(gzippedData) }) }) } const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 function gzipPromise (buffer, options) { } import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 }) 12 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // 16 return new Promise((resolve, reject) => { }) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // 16 gzip(buffer, options, (err, gzippedData) => { }) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // 16 if (err) { return reject(err) } import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 6 7 8 9 resolve(gzippedData) 10 }) 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // 16 resolve(gzippedData) import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 10 }) 11 }) 12 } 13 14 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) 15 console.log(compressed) // 16 const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) 1 2 function gzipPromise (buffer, options) { 3 return new Promise((resolve, reject) => { 4 gzip(buffer, options, (err, gzippedData) => { 5 if (err) { 6 return reject(err) 7 } 8 9 resolve(gzippedData) 10 }) 11 }) 12 } 13 14 15 16 import { gzip } from 'zlib' // zlib.gzip(buffer[, options], callback) function gzipPromise (buffer, options) { return new Promise((resolve, reject) => { gzip(buffer, options, (err, gzippedData) => { if (err) { return reject(err) } resolve(gzippedData) }) }) } const compressed = await gzipPromise(Buffer.from('Hello from Node.js')) console.log(compressed) // 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 loige 65

What if we we want to do the opposite? 🤷 Convert a promise-based function to a callback-based one loige 66

var env = nunjucks.configure('views') env.addFilter('videoTitle', function(videoId, cb) { // ... fetch the title through youtube APIs // ... extract the video title // ... and call the callback with the title }, true) 1 2 3 4 5 6 7 OK, this is not a common use case, so let me give you a real example! Nunjucks async filters {{ data | myCustomFilter }} We are forced to pass a callback-based function here! Ex: {{ youtubeId | videoTitle }} loige 67

util.callbackify() import { callbackify } from 'util' import Innertube from 'youtubei.js' // from npm async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } const videoTitleFilterCb = callbackify(videoTitleFilter) videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 import Innertube from 'youtubei.js' // from npm import { callbackify } from 'util' 1 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 4 5 6 7 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 const videoTitleFilterCb = callbackify(videoTitleFilter) import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 10 11 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 12 if (err) { 13 console.error(err) 14 return 15 } 16 17 console.log(videoTitle) 18 }) 19 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) import { callbackify } from 'util' 1 import Innertube from 'youtubei.js' // from npm 2 3 async function videoTitleFilter (videoId) { 4 const youtube = await new Innertube({ gl: 'US' }) 5 const details = await youtube.getDetails(videoId) 6 return details.title 7 } 8 9 const videoTitleFilterCb = callbackify(videoTitleFilter) 10 11 12 13 14 15 16 17 18 19 import { callbackify } from 'util' import Innertube from 'youtubei.js' // from npm async function videoTitleFilter (videoId) { const youtube = await new Innertube({ gl: 'US' }) const details = await youtube.getDetails(videoId) return details.title } const videoTitleFilterCb = callbackify(videoTitleFilter) videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { if (err) { console.error(err) return } console.log(videoTitle) }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 loige 68

Callbackify by hand ✋ import Innertube from 'youtubei.js' // from npm async function videoTitleFilter (videoId) { // ... } function videoTitleFilterCb (videoId, cb) { videoTitleFilter(videoId) .then((videoTitle) => cb(null, videoTitle)) .catch(cb) } videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { // ... }) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 function videoTitleFilterCb (videoId, cb) { videoTitleFilter(videoId) .then((videoTitle) => cb(null, videoTitle)) .catch(cb) } import Innertube from 'youtubei.js' // from npm 1 2 async function videoTitleFilter (videoId) { 3 // ... 4 } 5 6 7 8 9 10 11 12 videoTitleFilterCb('18y6OjdeR6o', (err, videoTitle) => { 13 // ... 14 }) 15 loige 69

OK, Cool! But is this stuff worth it? 🧐 loige 70

Let me show you a cool performance trick for Web Servers! 😎 loige 71

The request batching pattern one user /api/hotels/rome Web server DB loige 72

The request batching pattern multiple users (no batching) Web server DB /api/hotels/rome /api/hotels/rome /api/hotels/rome 73 loige

The request batching pattern multiple users (with batching!) Web server DB /api/hotels/rome /api/hotels/rome /api/hotels/rome 74 📘 Requests in-flight /api/hotels/rome ✅ loige

The web server import { createServer } from 'http' const urlRegex = /^\/api\/hotels\/([\w-]+)$/ createServer(async (req, res) => { const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) if (!matches) { res.writeHead(404, 'Not found') return res.end() } const [_, city] = matches const hotels = await getHotelsForCity(city) res.writeHead(200) res.end(JSON.stringify({ hotels })) }).listen(8000) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ import { createServer } from 'http' 1 2 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 createServer(async (req, res) => { }).listen(8000) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 19 const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 6 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 if (!matches) { res.writeHead(404, 'Not found') return res.end() } import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 9 10 11 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const [_, city] = matches import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 14 const hotels = await getHotelsForCity(city) 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 const hotels = await getHotelsForCity(city) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 15 16 res.writeHead(200) 17 res.end(JSON.stringify({ hotels })) 18 }).listen(8000) 19 res.writeHead(200) res.end(JSON.stringify({ hotels })) import { createServer } from 'http' 1 2 const urlRegex = /^\/api\/hotels\/([\w-]+)$/ 3 4 createServer(async (req, res) => { 5 const url = new URL(req.url, 'http://localhost') 6 const matches = urlRegex.exec(url.pathname) 7 8 if (!matches) { 9 res.writeHead(404, 'Not found') 10 return res.end() 11 } 12 13 const [_, city] = matches 14 const hotels = await getHotelsForCity(city) 15 16 17 18 }).listen(8000) 19 import { createServer } from 'http' const urlRegex = /^\/api\/hotels\/([\w-]+)$/ createServer(async (req, res) => { const url = new URL(req.url, 'http://localhost') const matches = urlRegex.exec(url.pathname) if (!matches) { res.writeHead(404, 'Not found') return res.end() } const [_, city] = matches const hotels = await getHotelsForCity(city) res.writeHead(200) res.end(JSON.stringify({ hotels })) }).listen(8000) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 loige 75

The data fetching function (with batching) let pendingRequests = new Map() function getHotelsForCity (cityId) { if (pendingRequests.has(cityId)) { return pendingRequests.get(cityId) } const asyncOperation = db.query({ text: 'SELECT * FROM hotels WHERE cityid = $1', values: [cityId], }) pendingRequests.set(cityId, asyncOperation) asyncOperation.finally(() => { pendingRequests.delete(cityId) }) return asyncOperation } 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 loige 76

Benchmarks loige.link/req-batch-bench Without request batching With request batching (+90% avg req/sec)* * This is an artificial benchmark and results might vary significantly in real-life scenarios. Always run your own benchmarks before deciding whether this optimization can have a positive effect for you. loige 77

Closing Notes JavaScript can be a very powerful and convenient language when we have to deal with a lot of I/O (e.g. web servers) The async story has evolved a lot in the last 10-15 years: new patterns and language constructs have emerged Async/Await is probably the best way to write async code today To use Async/Await correctly you need to understand Promise and callbacks Take your time and invest in learning the fundamentals loige 78

Cover Picture by on Marc-Olivier Jodoin Unsplash THANKS! 🙌 ❤ nodejsdp.link loige Grab these slides! 😍 Grab the book 79