Finding a lost song with Node.js and async iterators - NodeConf Remote 2021

Senior Architect @ fourTheorem (Dublin ) nodejsdp.link 📔 Co-Author of Node.js Design Patterns 👉 Let's connect! (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 9

We are business focused technologists that deliver. | | Accelerated

Serverless AI as a Service Platform Modernisation We are hiring: do you want to ? work with us loige 10

So, there was this song in my mind... 🐛 loige

11

I could only remember some random parts and the word

"dark" (probably in the title) loige 12

Luciano - scrobbling since 12 Feb 2007 loige 15

Luciano - scrobbling since 12 Feb 2007 loige ~250k scrobbles...

that song must be there! 15

loige There's an API! https://www.last.fm/api 16

loige Let's give it a shot curl "http://ws.audioscrobbler.com/2.0/? method=user.getrecenttracks&user=loige&api_key =${API_KEY}&format=json"

| jq . 18

It works! 🥳 Now let's do this with JavaScript loige

20

import { request } from 'undici' const query = new

URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json' }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) loige 21

import { request } from 'undici' const query = new

loige 22

loige We are getting a "paginated" response with 50 tracks

per page 22

loige We are getting a "paginated" response with 50 tracks

per page but there are 51 here! 🤔 22

loige We are getting a "paginated" response with 50 tracks

per page but there are 51 here! 🤔 (let's ignore this for now...) 22

loige We are getting a "paginated" response with 50 tracks

per page but there are 51 here! 🤔 How do we fetch the next pages? (let's ignore this for now...) 22

loige 23

let page = 1 while (true) { const query =

new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) if (page === Number(data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 24

let page = 1 while (true) { const query =

loige 25

Seems good! 👌 Let's look at the tracks... loige 26

// ... for (const track of data.recenttracks.track) { console.log( track.date?.['#text'],

`${track.artist['#text']} - ${track.name}` ) } console.log('--- end page ---') // ... loige 27

loige 28

loige * Note that page size here is 10 tracks

per page 28

loige * Note that page size here is 10 tracks

per page Every page has a song with undeﬁned time... This is the song I am currently listening to! It appears at the top of every page. 28

loige * Note that page size here is 10 tracks

per page Sometimes there are duplicated tracks between pages... 😨 28

The "sliding windows" problem 😩 loige 29

loige ... tracks (newest to oldest) 30

loige ... tracks (newest to oldest) 30 Page1 Page2

loige ... tracks (newest to oldest) 30 Page1 Page2 ...

new track

loige ... tracks (newest to oldest) 30 Page1 Page2 ...

Page1 Page2 new track

loige ... tracks (newest to oldest) 30 Page1 Page2 ...

Page1 Page2 new track moved from page 1 to page 2

loige 31

Time based windows 😎 loige 32

loige ...* tracks (newest to oldest) 33 * we are

done when we get an empty page (or num pages is 1) to ... from

loige ...* tracks (newest to oldest) 33 Page1 * we

are done when we get an empty page (or num pages is 1) to ... from

loige ...* tracks (newest to oldest) 33 Page1 t1 *

we are done when we get an empty page (or num pages is 1) to ... from

loige ...* tracks (newest to oldest) 33 Page1 before t1

(page 1 "to" t1) t1 * we are done when we get an empty page (or num pages is 1) to ... from

loige ...* tracks (newest to oldest) 33 Page1 before t1

(page 1 "to" t1) t1 t2 * we are done when we get an empty page (or num pages is 1) to ... from

loige ...* tracks (newest to oldest) 33 Page1 before t1

(page 1 "to" t1) t1 t2 before t2 (page 1 "to" t2) * we are done when we get an empty page (or num pages is 1) to ... from

let to = '' while (true) { const query =

new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 34

let to = '' while (true) { const query =

loige 35

loige The track of the last timestamp becomes the boundary

for the next page 35

We have a working solution! 🎉 Can we generalise it?

loige 36

We know how to iterate over every page/track. How do

we expose this information? loige const reader = LastFmRecentTracks({ apikey: process.env.API_KEY, user: 'loige' }) 37

// callbacks reader.readPages( (page) => { /* ... */ },

// on page (err) => { /* ... */} // on completion (or error) ) loige 38

// event emitter reader.read() reader.on('page', (page) => { /* ...

*/ }) reader.on('completed', (err) => { /* ... */ }) loige 39

// streams ❤ reader.pipe(/* transform or writable stream here */)

reader.on('end', () => { /* ... */ }) reader.on('error', () => { /* ... */ }) loige 40

// streams pipeline ❤ ❤ pipeline( reader, yourProcessingStream, (err) =>

{ // handle completion or err } ) loige 41

// ASYNC ITERATORS! 😵 for await (const page of reader)

{ /* ... */ } // ... do more stuff when all the // data is consumed loige 42

// ASYNC ITERATORS WITH ERROR HANDLING! 🤯 try { for

await (const page of reader) { /* ... */ } } catch (err) { // handle errors } // ... do more stuff when all the // data is consumed loige 43

How can we build an async iterator? 🧐 loige 44

Meet the iteration protocols! loige loige.co/javascript-iterator-patterns 45

Iteration concepts loige Iterator An object that acts as a

cursor to iterate over blocks of data sequentially Iterable An object that contains data that can be iterated over sequentially ... ... 46

The iterator protocol An object is an iterator if it

has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value. loige 47

function createCountdown (from) { let nextVal = from return {

next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48

function createCountdown (from) { let nextVal = from return {

const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:

3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true } loige 49

Generator functions "produce" iterators! loige 50

function * createCountdown (from) { for (let i = from;

i >= 0; i--) { yield i } } loige 51

function * createCountdown (from) { for (let i = from;

i >= 0; i--) { yield i } } loige 51

function * createCountdown (from) { for (let i = from;

i >= 0; i--) { yield i } } loige 51

const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:

3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true, value: undefined } loige 52

The iterable protocol An object is iterable if it implements

the Symbol.iterator method, a zero-argument function that returns an iterator. loige 53

function createCountdown (from) { let nextVal = from return {

[Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 54

function createCountdown (from) { let nextVal = from return {

function createCountdown (from) { return { [Symbol.iterator]: function * ()

{ for (let i = from; i >= 0; i--) { yield i } } } } loige 55

const countdown = createCountdown(3) for (const value of countdown) {

console.log(value) } // 3 // 2 // 1 // 0 loige 56

OK. So far this is all synchronous iteration. What about

async? 🙄 loige 57

The async iterator protocol An object is an async iterator

if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value. loige 58

import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

= 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 59

import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false,

value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 60

const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false,

value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 60

loige 61

import { setTimeout } from 'timers/promises' // async generators "produce"

async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 62

import { setTimeout } from 'timers/promises' // async generators "produce"

The async iterable protocol An object is an async iterable

if it implements the Symbol.asyncIterator method, a zero- argument function that returns an async iterator. loige 63

import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

= 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 64

import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay

= 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 64

HOT TIP 🔥 With async generators we can create objects

that are both async iterators and async iterables! loige (We don't need to specify Symbol.asyncIterator explicitly!) 65

import { setTimeout } from 'timers/promises' // async generators "produce"

async iterators // (and iterables!) async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 66

const countdown = createAsyncCountdown(3) for await (const value of countdown)

{ console.log(value) } loige 67

const countdown = createAsyncCountdown(3) for await (const value of countdown)

{ console.log(value) } loige 67

Now we know how to make our LastFmRecentTracks an Async

Iterable 🤩 loige 68

import { request } from 'undici' async function* createLastFmRecentTracks (apiKey,

user) { let to = '' while (true) { const query = new URLSearchParams({ method: 'user.getrecenttracks', user, api_key: apiKey, format: 'json', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track yield tracks if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } } loige 69

import { request } from 'undici' async function* createLastFmRecentTracks (apiKey,

const recentTracks = createLastFmRecentTracks( process.env.API_KEY, 'loige' ) for await (const

page of recentTracks) { console.log(page) } loige 70

Let's search for all the songs that contain the word

"dark" in their title! 🧐 loige 71

const recentTracks = createLastFmRecentTracks( process.env.API_KEY, 'loige' ) for await (const

page of recentTracks) { for (const track of page) { if (track.name.toLowerCase().includes('dark')) { console.log(`${track.artist['#text']} - ${track.name}`) } } } loige 72

loige 73

loige OMG! This is the song! 😱 ...from 8 years

ago! 73

For a more serious package that allows you to fetch

data from Last.fm: loige npm install scrobbles 74

Cover Picture by on ❤ Thanks to Jacek Spera, ,

, , , for reviews and suggestions. FPVmat A Unsplash @eoins @pelger @gbinside @ManuEomm @simonplend - loige.link/nodeconf-iter loige.link/async-it-code for await (const _ of createAsyncCountdown(1_000_000)) { console.log("THANK YOU! 😍 ") } loige nodejsdp.link 75

Finding a lost song with Node.js and async iter...

Finding a lost song with Node.js and async iterators - NodeConf Remote 2021

More Decks by Luciano Mammino

Other Decks in Technology

Featured

Transcript