Share
Embed
Finding a lost song with Node.js and async iterators - NodeConf Remote 2021
Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!
Luciano Mammino
October 18, 2021
Transcript
131 slides
-
Finding a lost song with Node.js & async iterators Luciano
Mammino ( ) @loige 2021-10-18 loige.link/nodeconf-iter 1 -
Photo by on Darius Bashar Unsplash A random song you
haven't listened to in years pops into your head... 👂🐛 3 -
It doesn't matter what you do all day... It keeps
coming back to you! 🐛 Photo by on Attentie Attentie Unsplash 4 -
And now you want to listen to it! Photo by
on Volodymyr Hryshchenko Unsplash 5 -
But, what if you can't remember the title or the
author?! Photo by on Tachina Lee Unsplash 6 -
THERE MUST BE A WAY TO REMEMBER! Photo by on
Marius Niveri Unsplash 7 -
Today, I'll tell you how I solved this problem using
- Last.fm API - Node.js - Async Iterators Photo by on Quinton Coetzee Unsplash 8 -
Let me introduce myself first... 👋 I'm Luciano ( 🍕🍝)
Senior Architect @ fourTheorem (Dublin ) 9 -
Let me introduce myself first... 👋 I'm Luciano ( 🍕🍝)
Senior Architect @ fourTheorem (Dublin ) nodejsdp.link 📔 Co-Author of Node.js Design Patterns 👉 9 -
Let me introduce myself first... 👋 I'm Luciano ( 🍕🍝)
Senior Architect @ fourTheorem (Dublin ) nodejsdp.link 📔 Co-Author of Node.js Design Patterns 👉 Let's connect! (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 9 -
We are business focused technologists that deliver. | | Accelerated
Serverless AI as a Service Platform Modernisation We are hiring: do you want to ? work with us loige 10 -
I could only remember some random parts and the word
"dark" (probably in the title) loige 12 -
Luciano - scrobbling since 12 Feb 2007 loige ~250k scrobbles...
that song must be there! 15 -
import { request } from 'undici' const query = new
URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json' }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) loige 21 -
import { request } from 'undici' const query = new
URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json' }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) loige 21 -
import { request } from 'undici' const query = new
URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json' }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) loige 21 -
import { request } from 'undici' const query = new
URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json' }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) loige 21 -
import { request } from 'undici' const query = new
URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json' }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) loige 21 -
loige We are getting a "paginated" response with 50 tracks
per page but there are 51 here! 🤔 22 -
loige We are getting a "paginated" response with 50 tracks
per page but there are 51 here! 🤔 (let's ignore this for now...) 22 -
loige We are getting a "paginated" response with 50 tracks
per page but there are 51 here! 🤔 How do we fetch the next pages? (let's ignore this for now...) 22 -
let page = 1 while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) if (page === Number(data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 24 -
let page = 1 while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) if (page === Number(data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 24 -
let page = 1 while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) if (page === Number(data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 24 -
let page = 1 while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) if (page === Number(data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 24 -
let page = 1 while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) if (page === Number(data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 24 -
let page = 1 while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', page }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() console.log(data) if (page === Number(data.recenttracks['@attr'].totalPages)) { break // it's the last page! } loige 24 -
// ... for (const track of data.recenttracks.track) { console.log( track.date?.['#text'],
`${track.artist['#text']} - ${track.name}` ) } console.log('--- end page ---') // ... loige 27 -
loige * Note that page size here is 10 tracks
per page 28 -
loige * Note that page size here is 10 tracks
per page Every page has a song with undefined time... This is the song I am currently listening to! It appears at the top of every page. 28 -
loige * Note that page size here is 10 tracks
per page Sometimes there are duplicated tracks between pages... 😨 28 -
loige ... tracks (newest to oldest) 30 Page1 Page2 ...
Page1 Page2 new track -
loige ... tracks (newest to oldest) 30 Page1 Page2 ...
Page1 Page2 new track moved from page 1 to page 2 -
loige ...* tracks (newest to oldest) 33 * we are
done when we get an empty page (or num pages is 1) to ... from -
loige ...* tracks (newest to oldest) 33 Page1 * we
are done when we get an empty page (or num pages is 1) to ... from -
loige ...* tracks (newest to oldest) 33 Page1 t1 *
we are done when we get an empty page (or num pages is 1) to ... from -
loige ...* tracks (newest to oldest) 33 Page1 before t1
(page 1 "to" t1) t1 * we are done when we get an empty page (or num pages is 1) to ... from -
loige ...* tracks (newest to oldest) 33 Page1 before t1
(page 1 "to" t1) t1 t2 * we are done when we get an empty page (or num pages is 1) to ... from -
loige ...* tracks (newest to oldest) 33 Page1 before t1
(page 1 "to" t1) t1 t2 before t2 (page 1 "to" t2) * we are done when we get an empty page (or num pages is 1) to ... from -
let to = '' while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 34 -
let to = '' while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 34 -
let to = '' while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 34 -
let to = '' while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 34 -
let to = '' while (true) { const query =
new URLSearchParams({ method: 'user.getrecenttracks', user: 'loige', api_key: process.env.API_KEY, format: 'json', limit: '10', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track console.log( `--- ↓ page to ${to}`, `remaining pages: ${data.recenttracks['@attr'].totalPages} ---` ) for (const track of tracks) { console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`) } if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } loige 34 -
loige The track of the last timestamp becomes the boundary
for the next page 35 -
We know how to iterate over every page/track. How do
we expose this information? loige const reader = LastFmRecentTracks({ apikey: process.env.API_KEY, user: 'loige' }) 37 -
// callbacks reader.readPages( (page) => { /* ... */ },
// on page (err) => { /* ... */} // on completion (or error) ) loige 38 -
// event emitter reader.read() reader.on('page', (page) => { /* ...
*/ }) reader.on('completed', (err) => { /* ... */ }) loige 39 -
// streams ❤ reader.pipe(/* transform or writable stream here */)
reader.on('end', () => { /* ... */ }) reader.on('error', () => { /* ... */ }) loige 40 -
// streams pipeline ❤ ❤ pipeline( reader, yourProcessingStream, (err) =>
{ // handle completion or err } ) loige 41 -
// ASYNC ITERATORS! 😵 for await (const page of reader)
{ /* ... */ } // ... do more stuff when all the // data is consumed loige 42 -
// ASYNC ITERATORS WITH ERROR HANDLING! 🤯 try { for
await (const page of reader) { /* ... */ } } catch (err) { // handle errors } // ... do more stuff when all the // data is consumed loige 43 -
Iteration concepts loige Iterator An object that acts as a
cursor to iterate over blocks of data sequentially Iterable An object that contains data that can be iterated over sequentially ... ... 46 -
The iterator protocol An object is an iterator if it
has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value. loige 47 -
function createCountdown (from) { let nextVal = from return {
next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48 -
function createCountdown (from) { let nextVal = from return {
next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48 -
function createCountdown (from) { let nextVal = from return {
next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48 -
function createCountdown (from) { let nextVal = from return {
next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48 -
function createCountdown (from) { let nextVal = from return {
next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 48 -
const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:
3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true } loige 49 -
function * createCountdown (from) { for (let i = from;
i >= 0; i--) { yield i } } loige 51 -
function * createCountdown (from) { for (let i = from;
i >= 0; i--) { yield i } } loige 51 -
function * createCountdown (from) { for (let i = from;
i >= 0; i--) { yield i } } loige 51 -
const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:
3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true, value: undefined } loige 52 -
The iterable protocol An object is iterable if it implements
the Symbol.iterator method, a zero-argument function that returns an iterator. loige 53 -
function createCountdown (from) { let nextVal = from return {
[Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 54 -
function createCountdown (from) { let nextVal = from return {
[Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 54 -
function createCountdown (from) { let nextVal = from return {
[Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 54 -
function createCountdown (from) { return { [Symbol.iterator]: function * ()
{ for (let i = from; i >= 0; i--) { yield i } } } } loige 55 -
const countdown = createCountdown(3) for (const value of countdown) {
console.log(value) } // 3 // 2 // 1 // 0 loige 56 -
OK. So far this is all synchronous iteration. What about
async? 🙄 loige 57 -
The async iterator protocol An object is an async iterator
if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value. loige 58 -
import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay
= 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 59 -
import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay
= 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 59 -
import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay
= 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 59 -
const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false,
value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 60 -
const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false,
value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 60 -
import { setTimeout } from 'timers/promises' // async generators "produce"
async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 62 -
import { setTimeout } from 'timers/promises' // async generators "produce"
async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 62 -
import { setTimeout } from 'timers/promises' // async generators "produce"
async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 62 -
The async iterable protocol An object is an async iterable
if it implements the Symbol.asyncIterator method, a zero- argument function that returns an async iterator. loige 63 -
import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay
= 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 64 -
import { setTimeout } from 'timers/promises' function createAsyncCountdown (from, delay
= 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 64 -
HOT TIP 🔥 With async generators we can create objects
that are both async iterators and async iterables! loige (We don't need to specify Symbol.asyncIterator explicitly!) 65 -
import { setTimeout } from 'timers/promises' // async generators "produce"
async iterators // (and iterables!) async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 66 -
const countdown = createAsyncCountdown(3) for await (const value of countdown)
{ console.log(value) } loige 67 -
const countdown = createAsyncCountdown(3) for await (const value of countdown)
{ console.log(value) } loige 67 -
Now we know how to make our LastFmRecentTracks an Async
Iterable 🤩 loige 68 -
import { request } from 'undici' async function* createLastFmRecentTracks (apiKey,
user) { let to = '' while (true) { const query = new URLSearchParams({ method: 'user.getrecenttracks', user, api_key: apiKey, format: 'json', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track yield tracks if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } } loige 69 -
import { request } from 'undici' async function* createLastFmRecentTracks (apiKey,
user) { let to = '' while (true) { const query = new URLSearchParams({ method: 'user.getrecenttracks', user, api_key: apiKey, format: 'json', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track yield tracks if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } } loige 69 -
import { request } from 'undici' async function* createLastFmRecentTracks (apiKey,
user) { let to = '' while (true) { const query = new URLSearchParams({ method: 'user.getrecenttracks', user, api_key: apiKey, format: 'json', to }) const url = `https://ws.audioscrobbler.com/2.0/?${query}` const { body } = await request(url) const data = await body.json() const tracks = data.recenttracks.track yield tracks if (data.recenttracks['@attr'].totalPages <= 1) { break // it's the last page! } const lastTrackInPage = tracks[tracks.length - 1] to = lastTrackInPage.date.uts } } loige 69 -
const recentTracks = createLastFmRecentTracks( process.env.API_KEY, 'loige' ) for await (const
page of recentTracks) { console.log(page) } loige 70 -
Let's search for all the songs that contain the word
"dark" in their title! 🧐 loige 71 -
const recentTracks = createLastFmRecentTracks( process.env.API_KEY, 'loige' ) for await (const
page of recentTracks) { for (const track of page) { if (track.name.toLowerCase().includes('dark')) { console.log(`${track.artist['#text']} - ${track.name}`) } } } loige 72 -
For a more serious package that allows you to fetch
data from Last.fm: loige npm install scrobbles 74 -
Cover Picture by on ❤ Thanks to Jacek Spera, ,
, , , for reviews and suggestions. FPVmat A Unsplash @eoins @pelger @gbinside @ManuEomm @simonplend - loige.link/nodeconf-iter loige.link/async-it-code for await (const _ of createAsyncCountdown(1_000_000)) { console.log("THANK YOU! 😍 ") } loige nodejsdp.link 75