Slide 1

Slide 1 text

1

Slide 2

Slide 2 text

META_SLIDE! loige loige.link/lets-iter 2

Slide 3

Slide 3 text

const array = ['foo', 'bar', 'baz'] for (const item of array) { console.log(item) } loige Output: foo bar baz 3

Slide 4

Slide 4 text

const str = 'foo' for (const item of str) { console.log(item) } loige Output: f o o 4

Slide 5

Slide 5 text

const set = new Set(['foo', 'bar', 'baz']) for (const item of set) { console.log(item) } loige Output: foo bar baz 5

Slide 6

Slide 6 text

const map = new Map([ ['foo', 'bar'], ['baz', 'qux'] ]) for (const item of map) { console.log(item) } loige Output: [ 'foo', 'bar' ] [ 'baz', 'qux' ] 6

Slide 7

Slide 7 text

const obj = { foo: 'bar', baz: 'qux' } for (const item of obj) { console.log(item) } loige Output: β›” Uncaught TypeError: obj is not iterable OMG `for ... of` does not work with plain objects! 😱 7

Slide 8

Slide 8 text

const obj = { foo: 'bar', baz: 'qux' } for (const item of Object.entries(obj)) { console.log(item) } loige Output: [ 'foo', 'bar' ] [ 'baz', 'qux' ] 8

Slide 9

Slide 9 text

loige 9

Slide 10

Slide 10 text

const array = ['foo', 'bar', 'baz'] console.log(...array) loige Output: foo bar baz spread syntax! 10

Slide 11

Slide 11 text

loige 11

Slide 12

Slide 12 text

loige 12

Slide 13

Slide 13 text

loige Iterators Iterables Async Iterators Async Iterables Generator functions Async Generators πŸ“Έ THE BIGGER PICTURE 13

Slide 14

Slide 14 text

loige Knowing iteration protocols allows us: Understand JavaScript better Write more modern, interoperable and idiomatic code Be able to write our own custom iterables (even async) πŸ˜₯ WHY SHOULD WE CARE? 14

Slide 15

Slide 15 text

WHO IS THIS GUY !? πŸ‘‹ I'm Luciano ( πŸ•πŸ) Senior Architect @ fourTheorem (Dublin ) nodejsdp.link πŸ“” Co-Author of Node.js Design Patterns πŸ‘‰ Let's connect! (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 15

Slide 16

Slide 16 text

ALWAYS RE-IMAGINING WE ARE A PIONEERING TECHNOLOGY CONSULTANCY FOCUSED ON AWS AND SERVERLESS | | Accelerated Serverless AI as a Service Platform Modernisation loige βœ‰ Reach out to us at πŸ˜‡ We are always looking for talent: [email protected] fth.link/careers 16

Slide 17

Slide 17 text

We host a weekly podcast about AWS loige awsbites.com loige 17

Slide 18

Slide 18 text

πŸ“’ AGENDA Generators Iterator protocol Iterable protocol Async Iterator protocol Async Iterable protcol Real-lifeβ„’ examples loige 18

Slide 19

Slide 19 text

LET'S CLONE πŸ‘πŸ loige loige.link/lets-iter-repo git clone https://github.com/lmammino/iteration-protocols-workshop.git cd iteration-protocols-workshop npm i 19

Slide 20

Slide 20 text

➑ GENERATORS loige 20

Slide 21

Slide 21 text

GENERATOR FN & OBJ loige function * myGenerator () { // generator body yield 'someValue' // ... do more stuff } const genObj = myGenerator() genObj.next() // -> { done: false, value: 'someValue' } 21

Slide 22

Slide 22 text

function * fruitGen () { yield ' πŸ‘ ' yield ' πŸ‰ ' yield ' πŸ‹ ' yield ' πŸ₯­ ' } const fruitGenObj = fruitGen() console.log(fruitGenObj.next()) // { value: ' πŸ‘ ', done: false } console.log(fruitGenObj.next()) // { value: ' πŸ‰ ', done: false } console.log(fruitGenObj.next()) // { value: ' πŸ‹ ', done: false } console.log(fruitGenObj.next()) // { value: ' πŸ₯­ ', done: false } console.log(fruitGenObj.next()) // { value: undefined, done: true } loige 22

Slide 23

Slide 23 text

function * fruitGen () { yield ' πŸ‘ ' yield ' πŸ‰ ' yield ' πŸ‹ ' yield ' πŸ₯­ ' } const fruitGenObj = fruitGen() // generator objects are iterable! for (const fruit of fruitGenObj) { console.log(fruit) } // πŸ‘ // πŸ‰ // πŸ‹ // πŸ₯­ loige 23

Slide 24

Slide 24 text

function * range (start, end) { for (let i = start; i < end; i++) { yield i } } // generators are lazy! for (const i of range(0, Number.MAX_VALUE)) { console.log(i) } const zeroToTen = [...range(0, 11)] loige 24

Slide 25

Slide 25 text

// generators can be "endless" function * cycle (values) { let current = 0 while (true) { yield values[current] current = (current + 1) % values.length } } for (const value of cycle(['even', 'odd'])) { console.log(value) } // even // odd // even // ... loige 25

Slide 26

Slide 26 text

πŸ“ MINI-SUMMARY loige A generator function returns a generator object which is both an iterator and an iterable. A generator function uses `yield` to yield a value and pause its execution. The generator object is used to make progress on an instance of the generator (by calling `next()`). Generator functions are a great way to create custom iterable objects. Generator objects are lazy and they can be endless. 26

Slide 27

Slide 27 text

πŸ“ EXERCISE(S) loige 02-generators/exercises/zip.js function * take (n, iterable) { // take at most n items from iterable and // yield them one by one (lazily) } 02-generators/exercises/zip.js function * zip (iterable1, iterable2) { // consume the two iterables until any of the 2 completes // yield a pair taking 1 item from the first and one from // the second at every step } 27

Slide 28

Slide 28 text

➑ ITERATORS & ITERABLES loige 28

Slide 29

Slide 29 text

ITERATOR OBJ loige An object that acts like a cursor to iterate over blocks of data sequentially 29

Slide 30

Slide 30 text

ITERABLE OBJ loige An object that contains data that can be iterated over sequentially 30

Slide 31

Slide 31 text

THE ITERATOR PROTOCOL An object is an iterator if it has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value. loige 31

Slide 32

Slide 32 text

function createCountdown (from) { let nextVal = from return { next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 32

Slide 33

Slide 33 text

const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value: 3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true } loige 33

Slide 34

Slide 34 text

πŸ”₯ Generator Objects are iterators (and iterables)! loige 34

Slide 35

Slide 35 text

function * createCountdown (from) { for (let i = from; i >= 0; i--) { yield i } } loige 35

Slide 36

Slide 36 text

const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value: 3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true, value: undefined } loige 36

Slide 37

Slide 37 text

THE ITERABLE PROTOCOL An object is iterable if it implements the Symbol.iterator method, a zero-argument function that returns an iterator. loige 37

Slide 38

Slide 38 text

function createCountdown (from) { let nextVal = from return { [Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 38

Slide 39

Slide 39 text

function createCountdown (from) { return { [Symbol.iterator]: function * () { for (let i = from; i >= 0; i--) { yield i } } } } loige 39

Slide 40

Slide 40 text

function * createCountdown () { for (let i = from; i >= 0; i--) { yield i } } loige πŸ”₯ or just use generators! 40

Slide 41

Slide 41 text

const countdown = createCountdown(3) for (const value of countdown) { console.log(value) } // 3 // 2 // 1 // 0 loige 41

Slide 42

Slide 42 text

const iterableIterator = { next () { return { done: false, value: 'hello' } }, [Symbol.iterator] () { return this } } An object can be an iterable and an iterator at the same time! loige 42

Slide 43

Slide 43 text

πŸ“ MINI-SUMMARY 1/2 loige An iterator is an object that allows us to traverse a collection The iterator protocol specifies that an object is an iterator if it has a `next()` method that returns an object with the shape `{done, value}`. `done` (a boolean) tells us if the iteration is completed `value` represents the value from the current iteration. You can write an iterator as an anonymous object (e.g. returned by a factory function), using classes or using generators. 43

Slide 44

Slide 44 text

πŸ“ MINI-SUMMARY 2/2 loige The iterable protocol defines what's expected for a JavaScript object to be considered iterable. That is an object that holds a collection of data on which you can iterate on sequentially. An object is iterable if it implements a special method called `Symbol.iterator` which returns an iterator. (An object is iterable if you can get an iterator from it!) Generator functions produce objects that are iterable. We saw that generators produce objects that are also iterators. It is possible to have objects that are both iterator and iterable. The trick is to create the object as an iterator and to implement a `Symbol.iterator` that returns the object itself (`this`). 44

Slide 45

Slide 45 text

πŸ“ EXERCISE(S) loige 04-iterable-protocol/exercises/binarytree.js class BinaryTree { // // ... // make this iterable! } 04-iterable-protocol/exercises/entries.js function entriesIterable (obj) { // Return an iterable that produce key/value pairs // from obj } 45

Slide 46

Slide 46 text

OK, very cool! But, so far this is all synchronous iteration. What about async? πŸ™„ loige 46

Slide 47

Slide 47 text

➑ ASYNC ITERATORS & ITERABLES loige 47

Slide 48

Slide 48 text

THE ASYNC ITERATOR PROTOCOL An object is an async iterator if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value. loige 48

Slide 49

Slide 49 text

import { setTimeout } from 'node:timers/promises' function createAsyncCountdown (from, delay = 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 49

Slide 50

Slide 50 text

const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false, value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 50

Slide 51

Slide 51 text

loige 51

Slide 52

Slide 52 text

import { setTimeout } from 'node:timers/promises' // async generators "produce" async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 52

Slide 53

Slide 53 text

THE ASYNC ITERABLE PROTOCOL An object is an async iterable if it implements the `Symbol.asyncIterator` method, a zero-argument function that returns an async iterator. loige 53

Slide 54

Slide 54 text

import { setTimeout } from 'node:timers/promises' function createAsyncCountdown (from, delay = 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 54

Slide 55

Slide 55 text

HOT TIP πŸ”₯ With async generators we can create objects that are both async iterators and async iterables! (We don't need to specify Symbol.asyncIterator explicitly!) loige 55

Slide 56

Slide 56 text

import { setTimeout } from 'node:timers/promises' // async generators "produce" async iterators // (and iterables!) async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 56

Slide 57

Slide 57 text

const countdown = createAsyncCountdown(3) for await (const value of countdown) { console.log(value) } loige 57

Slide 58

Slide 58 text

πŸ“ MINI-SUMMARY 1/2 loige Async iterators are the asynchronous counterpart of iterators. They are useful to iterate over data that becomes available asynchronously (e.g. coming from a database or a REST API). A good example is a paginated API, we could build an async iterator that gives a new page for every iteration. An object is an async iterator if it has a `next()` method which returns a `Promise` that resolves to an object with the shape: `{done, value}`. The main difference with the iterator protocol is that this time `next()` returns a promise. When we call next we need to make sure we `await` the returned promise. 58

Slide 59

Slide 59 text

πŸ“ MINI-SUMMARY 2/2 loige The async iterable protocol defines what it means for an object to be an async iterable. Once you have an async iterable you can use the `for await ... of` syntax on it. An object is an async iterable if it has a special method called `Symbol.asyncIterator` that returns an async iterator. Async iterables are a great way to abstract paginated data that is available asynchronously or similar operations like pulling jobs from a remote queue. A small spoiler, async iterables can also be used with Node.js streams... 59

Slide 60

Slide 60 text

πŸ“ EXERCISE loige 06-async-iterable-protocol/exercises/rickmorty.js function createCharactersPaginator () { // return an iterator that produces pages // with name of Rick and Morty characters // taken from the API // https://rickandmortyapi.com/api/character } // This is what we want to support πŸ‘‡ const paginator = createCharactersPaginator() for await (const page of paginator) { console.log(page) } 60

Slide 61

Slide 61 text

➑ TIPS & ASYNC ITERATORS IN NODE.JS loige 61

Slide 62

Slide 62 text

function isIterable (obj) { return typeof obj[Symbol.iterator] === 'function' } const array = [1, 2, 3] console.log(array, isIterable(array)) // true const genericObj = { foo: 'bar' } console.log(isIterable(genericObj)) // false console.log(isIterable(Object.entries(genericObj))) // true const fakeIterable = { [Symbol.iterator] () { return 'notAnIterator' } } console.log(fakeIterable, isIterable(fakeIterable)) // true 😑 IS THIS OBJECT AN ITERABLE? loige 62

Slide 63

Slide 63 text

function isAsyncIterable (obj) { return typeof obj[Symbol.asyncIterator] === 'function' } IS THIS OBJECT AN ASYNC ITERABLE? loige Are there async iterable objects in Node.js core? 63

Slide 64

Slide 64 text

console.log( typeof process.stdin[Symbol.asyncIterator] === 'function' ) // true! let bytes = 0 for await (const chunk of process.stdin) { bytes += chunk.length } console.log(`${bytes} bytes read from stdin`) READABLE STREAMS! loige 64

Slide 65

Slide 65 text

import { createWriteStream } from 'node:fs' const dest = createWriteStream('data.bin') let bytes = 0 for await (const chunk of process.stdin) { // what if we are writing too much too fast?!! 😱 dest.write(chunk) bytes += chunk.length } dest.end() console.log(`${bytes} written into data.bin`) ⚠ BACKPRESSURE WARNING loige 65

Slide 66

Slide 66 text

import { createWriteStream } from 'node:fs' import { once } from 'node:events' const dest = createWriteStream('data.bin') let bytes = 0 for await (const chunk of process.stdin) { const canContinue = dest.write(chunk) bytes += chunk.length if (!canContinue) { // backpressure, now we stop and we need to wait for drain await once(dest, 'drain') // ok now it's safe to resume writing } } dest.end() console.log(`${bytes} written into data.bin`) loige βœ… handling backpressure like a pro! ... or you can use pipeline() 66

Slide 67

Slide 67 text

import { pipeline } from 'node:stream/promises' import {createReadStream, createWriteStream} from 'node:fs' await pipeline( createReadStream('lowercase.txt'), async function* (source) { for await (const chunk of source) { yield await processChunk(chunk) } }, createWriteStream('uppercase.txt') ) console.log('Pipeline succeeded.') loige pipeline() supports async iterables! 😱 67

Slide 68

Slide 68 text

import { on } from 'node:events' import glob from 'glob' // from npm const matcher = glob('**/*.js') for await (const [filePath] of on(matcher, 'match')) { console.log(filePath) } loige creates an async iterable that will yield every time the `match` event happens 68

Slide 69

Slide 69 text

import { on } from 'node:events' import glob from 'glob' // from npm const matcher = glob('**/*.js') for await (const [filePath] of on(matcher, 'match')) { console.log(filePath) } // ⚠ WE WILL NEVER GET HERE πŸ‘‡ console.log('Done') loige This loop doesn't know when to stop! You could pass an AbortController here for more fine-grained control 69

Slide 70

Slide 70 text

πŸ“ FINAL SUMMARY loige You can check if an object is an iterable by checking `typeof obj[Symbol.iterator] === 'function'` You can check if an object is an async iterable with `typeof obj[Symbol.asyncIterator] === 'function'` In both cases, there's no guarantee that the iterable protocol is implemented correctly (the function might not return an iterator πŸ˜₯) Node.js Readable streams are also async iterable objects, so you could use `for await ... of` to consume the data in chunks If you do that and you end up writing data somewhere else, you'll need to handle backpressure yourself. It might be better to use `pipeline()` instead. You can convert Node.js event emitters to async iterable objects by using the `on` function from the module `events`. 70

Slide 71

Slide 71 text

Cover picture by on Back cover picture by on Camille Minouflet Unsplash Antonino Cicero Unsplash fourtheorem.com THANKS! πŸ™Œ loige.link/lets-iter loige nodejsdp.link 71