Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JavaScript Iteration Protocols - Workshop NodeConf EU 2022

Luciano Mammino
October 03, 2022
2

JavaScript Iteration Protocols - Workshop NodeConf EUΒ 2022

How many ways do you know to do iteration with JavaScript and Node.js? While, for loop, for...in, for..of, .map(), .forEach(), streams, iterators, etc! Yes, there are a lot of ways! But did you know that JavaScript has iteration protocols to standardise synchronous and even asynchronous iteration? In this workshop we will learn about these protocols and discover how to build iterators and iterable objects, both synchronous and asynchronous. We will learn about some common use cases for these protocols, explore generators and async generators (great tools for iteration) and finally discuss some hot tips, common pitfalls, and some (more or less successful) wild ideas!

Luciano Mammino

October 03, 2022
Tweet

Transcript

  1. 1

  2. META_SLIDE! loige loige.link/lets-iter 2

  3. const array = ['foo', 'bar', 'baz'] for (const item of

    array) { console.log(item) } loige Output: foo bar baz 3
  4. const str = 'foo' for (const item of str) {

    console.log(item) } loige Output: f o o 4
  5. const set = new Set(['foo', 'bar', 'baz']) for (const item

    of set) { console.log(item) } loige Output: foo bar baz 5
  6. const map = new Map([ ['foo', 'bar'], ['baz', 'qux'] ])

    for (const item of map) { console.log(item) } loige Output: [ 'foo', 'bar' ] [ 'baz', 'qux' ] 6
  7. const obj = { foo: 'bar', baz: 'qux' } for

    (const item of obj) { console.log(item) } loige Output: β›” Uncaught TypeError: obj is not iterable OMG `for ... of` does not work with plain objects! 😱 7
  8. const obj = { foo: 'bar', baz: 'qux' } for

    (const item of Object.entries(obj)) { console.log(item) } loige Output: [ 'foo', 'bar' ] [ 'baz', 'qux' ] 8
  9. loige 9

  10. const array = ['foo', 'bar', 'baz'] console.log(...array) loige Output: foo

    bar baz spread syntax! 10
  11. loige 11

  12. loige 12

  13. loige Iterators Iterables Async Iterators Async Iterables Generator functions Async

    Generators πŸ“Έ THE BIGGER PICTURE 13
  14. loige Knowing iteration protocols allows us: Understand JavaScript better Write

    more modern, interoperable and idiomatic code Be able to write our own custom iterables (even async) πŸ˜₯ WHY SHOULD WE CARE? 14
  15. WHO IS THIS GUY !? πŸ‘‹ I'm Luciano ( πŸ•πŸ)

    Senior Architect @ fourTheorem (Dublin ) nodejsdp.link πŸ“” Co-Author of Node.js Design Patterns πŸ‘‰ Let's connect! (blog) (twitter) (twitch) (github) loige.co @loige loige lmammino 15
  16. ALWAYS RE-IMAGINING WE ARE A PIONEERING TECHNOLOGY CONSULTANCY FOCUSED ON

    AWS AND SERVERLESS | | Accelerated Serverless AI as a Service Platform Modernisation loige βœ‰ Reach out to us at πŸ˜‡ We are always looking for talent: [email protected] fth.link/careers 16
  17. We host a weekly podcast about AWS loige awsbites.com loige

    17
  18. πŸ“’ AGENDA Generators Iterator protocol Iterable protocol Async Iterator protocol

    Async Iterable protcol Real-lifeβ„’ examples loige 18
  19. LET'S CLONE πŸ‘πŸ loige loige.link/lets-iter-repo git clone https://github.com/lmammino/iteration-protocols-workshop.git cd iteration-protocols-workshop

    npm i 19
  20. ➑ GENERATORS loige 20

  21. GENERATOR FN & OBJ loige function * myGenerator () {

    // generator body yield 'someValue' // ... do more stuff } const genObj = myGenerator() genObj.next() // -> { done: false, value: 'someValue' } 21
  22. function * fruitGen () { yield ' πŸ‘ ' yield

    ' πŸ‰ ' yield ' πŸ‹ ' yield ' πŸ₯­ ' } const fruitGenObj = fruitGen() console.log(fruitGenObj.next()) // { value: ' πŸ‘ ', done: false } console.log(fruitGenObj.next()) // { value: ' πŸ‰ ', done: false } console.log(fruitGenObj.next()) // { value: ' πŸ‹ ', done: false } console.log(fruitGenObj.next()) // { value: ' πŸ₯­ ', done: false } console.log(fruitGenObj.next()) // { value: undefined, done: true } loige 22
  23. function * fruitGen () { yield ' πŸ‘ ' yield

    ' πŸ‰ ' yield ' πŸ‹ ' yield ' πŸ₯­ ' } const fruitGenObj = fruitGen() // generator objects are iterable! for (const fruit of fruitGenObj) { console.log(fruit) } // πŸ‘ // πŸ‰ // πŸ‹ // πŸ₯­ loige 23
  24. function * range (start, end) { for (let i =

    start; i < end; i++) { yield i } } // generators are lazy! for (const i of range(0, Number.MAX_VALUE)) { console.log(i) } const zeroToTen = [...range(0, 11)] loige 24
  25. // generators can be "endless" function * cycle (values) {

    let current = 0 while (true) { yield values[current] current = (current + 1) % values.length } } for (const value of cycle(['even', 'odd'])) { console.log(value) } // even // odd // even // ... loige 25
  26. πŸ“ MINI-SUMMARY loige A generator function returns a generator object

    which is both an iterator and an iterable. A generator function uses `yield` to yield a value and pause its execution. The generator object is used to make progress on an instance of the generator (by calling `next()`). Generator functions are a great way to create custom iterable objects. Generator objects are lazy and they can be endless. 26
  27. πŸ“ EXERCISE(S) loige 02-generators/exercises/zip.js function * take (n, iterable) {

    // take at most n items from iterable and // yield them one by one (lazily) } 02-generators/exercises/zip.js function * zip (iterable1, iterable2) { // consume the two iterables until any of the 2 completes // yield a pair taking 1 item from the first and one from // the second at every step } 27
  28. ➑ ITERATORS & ITERABLES loige 28

  29. ITERATOR OBJ loige An object that acts like a cursor

    to iterate over blocks of data sequentially 29
  30. ITERABLE OBJ loige An object that contains data that can

    be iterated over sequentially 30
  31. THE ITERATOR PROTOCOL An object is an iterator if it

    has a next() method. Every time you call it, it returns an object with the keys done (boolean) and value. loige 31
  32. function createCountdown (from) { let nextVal = from return {

    next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 32
  33. const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:

    3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true } loige 33
  34. πŸ”₯ Generator Objects are iterators (and iterables)! loige 34

  35. function * createCountdown (from) { for (let i = from;

    i >= 0; i--) { yield i } } loige 35
  36. const countdown = createCountdown(3) console.log(countdown.next()) // { done: false, value:

    3 } console.log(countdown.next()) // { done: false, value: 2 } console.log(countdown.next()) // { done: false, value: 1 } console.log(countdown.next()) // { done: false, value: 0 } console.log(countdown.next()) // { done: true, value: undefined } loige 36
  37. THE ITERABLE PROTOCOL An object is iterable if it implements

    the Symbol.iterator method, a zero-argument function that returns an iterator. loige 37
  38. function createCountdown (from) { let nextVal = from return {

    [Symbol.iterator]: () => ({ next () { if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } }) } } loige 38
  39. function createCountdown (from) { return { [Symbol.iterator]: function * ()

    { for (let i = from; i >= 0; i--) { yield i } } } } loige 39
  40. function * createCountdown () { for (let i = from;

    i >= 0; i--) { yield i } } loige πŸ”₯ or just use generators! 40
  41. const countdown = createCountdown(3) for (const value of countdown) {

    console.log(value) } // 3 // 2 // 1 // 0 loige 41
  42. const iterableIterator = { next () { return { done:

    false, value: 'hello' } }, [Symbol.iterator] () { return this } } An object can be an iterable and an iterator at the same time! loige 42
  43. πŸ“ MINI-SUMMARY 1/2 loige An iterator is an object that

    allows us to traverse a collection The iterator protocol specifies that an object is an iterator if it has a `next()` method that returns an object with the shape `{done, value}`. `done` (a boolean) tells us if the iteration is completed `value` represents the value from the current iteration. You can write an iterator as an anonymous object (e.g. returned by a factory function), using classes or using generators. 43
  44. πŸ“ MINI-SUMMARY 2/2 loige The iterable protocol defines what's expected

    for a JavaScript object to be considered iterable. That is an object that holds a collection of data on which you can iterate on sequentially. An object is iterable if it implements a special method called `Symbol.iterator` which returns an iterator. (An object is iterable if you can get an iterator from it!) Generator functions produce objects that are iterable. We saw that generators produce objects that are also iterators. It is possible to have objects that are both iterator and iterable. The trick is to create the object as an iterator and to implement a `Symbol.iterator` that returns the object itself (`this`). 44
  45. πŸ“ EXERCISE(S) loige 04-iterable-protocol/exercises/binarytree.js class BinaryTree { // <implementation provided>

    // ... // make this iterable! } 04-iterable-protocol/exercises/entries.js function entriesIterable (obj) { // Return an iterable that produce key/value pairs // from obj } 45
  46. OK, very cool! But, so far this is all synchronous

    iteration. What about async? πŸ™„ loige 46
  47. ➑ ASYNC ITERATORS & ITERABLES loige 47

  48. THE ASYNC ITERATOR PROTOCOL An object is an async iterator

    if it has a next() method. Every time you call it, it returns a promise that resolves to an object with the keys done (boolean) and value. loige 48
  49. import { setTimeout } from 'node:timers/promises' function createAsyncCountdown (from, delay

    = 1000) { let nextVal = from return { async next () { await setTimeout(delay) if (nextVal < 0) { return { done: true } } return { done: false, value: nextVal-- } } } } loige 49
  50. const countdown = createAsyncCountdown(3) console.log(await countdown.next()) // { done: false,

    value: 3 } console.log(await countdown.next()) // { done: false, value: 2 } console.log(await countdown.next()) // { done: false, value: 1 } console.log(await countdown.next()) // { done: false, value: 0 } console.log(await countdown.next()) // { done: true } loige 50
  51. loige 51

  52. import { setTimeout } from 'node:timers/promises' // async generators "produce"

    async iterators! async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 52
  53. THE ASYNC ITERABLE PROTOCOL An object is an async iterable

    if it implements the `Symbol.asyncIterator` method, a zero-argument function that returns an async iterator. loige 53
  54. import { setTimeout } from 'node:timers/promises' function createAsyncCountdown (from, delay

    = 1000) { return { [Symbol.asyncIterator]: async function * () { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } } } loige 54
  55. HOT TIP πŸ”₯ With async generators we can create objects

    that are both async iterators and async iterables! (We don't need to specify Symbol.asyncIterator explicitly!) loige 55
  56. import { setTimeout } from 'node:timers/promises' // async generators "produce"

    async iterators // (and iterables!) async function * createAsyncCountdown (from, delay = 1000) { for (let i = from; i >= 0; i--) { await setTimeout(delay) yield i } } loige 56
  57. const countdown = createAsyncCountdown(3) for await (const value of countdown)

    { console.log(value) } loige 57
  58. πŸ“ MINI-SUMMARY 1/2 loige Async iterators are the asynchronous counterpart

    of iterators. They are useful to iterate over data that becomes available asynchronously (e.g. coming from a database or a REST API). A good example is a paginated API, we could build an async iterator that gives a new page for every iteration. An object is an async iterator if it has a `next()` method which returns a `Promise` that resolves to an object with the shape: `{done, value}`. The main difference with the iterator protocol is that this time `next()` returns a promise. When we call next we need to make sure we `await` the returned promise. 58
  59. πŸ“ MINI-SUMMARY 2/2 loige The async iterable protocol defines what

    it means for an object to be an async iterable. Once you have an async iterable you can use the `for await ... of` syntax on it. An object is an async iterable if it has a special method called `Symbol.asyncIterator` that returns an async iterator. Async iterables are a great way to abstract paginated data that is available asynchronously or similar operations like pulling jobs from a remote queue. A small spoiler, async iterables can also be used with Node.js streams... 59
  60. πŸ“ EXERCISE loige 06-async-iterable-protocol/exercises/rickmorty.js function createCharactersPaginator () { // return

    an iterator that produces pages // with name of Rick and Morty characters // taken from the API // https://rickandmortyapi.com/api/character } // This is what we want to support πŸ‘‡ const paginator = createCharactersPaginator() for await (const page of paginator) { console.log(page) } 60
  61. ➑ TIPS & ASYNC ITERATORS IN NODE.JS loige 61

  62. function isIterable (obj) { return typeof obj[Symbol.iterator] === 'function' }

    const array = [1, 2, 3] console.log(array, isIterable(array)) // true const genericObj = { foo: 'bar' } console.log(isIterable(genericObj)) // false console.log(isIterable(Object.entries(genericObj))) // true const fakeIterable = { [Symbol.iterator] () { return 'notAnIterator' } } console.log(fakeIterable, isIterable(fakeIterable)) // true 😑 IS THIS OBJECT AN ITERABLE? loige 62
  63. function isAsyncIterable (obj) { return typeof obj[Symbol.asyncIterator] === 'function' }

    IS THIS OBJECT AN ASYNC ITERABLE? loige Are there async iterable objects in Node.js core? 63
  64. console.log( typeof process.stdin[Symbol.asyncIterator] === 'function' ) // true! let bytes

    = 0 for await (const chunk of process.stdin) { bytes += chunk.length } console.log(`${bytes} bytes read from stdin`) READABLE STREAMS! loige 64
  65. import { createWriteStream } from 'node:fs' const dest = createWriteStream('data.bin')

    let bytes = 0 for await (const chunk of process.stdin) { // what if we are writing too much too fast?!! 😱 dest.write(chunk) bytes += chunk.length } dest.end() console.log(`${bytes} written into data.bin`) ⚠ BACKPRESSURE WARNING loige 65
  66. import { createWriteStream } from 'node:fs' import { once }

    from 'node:events' const dest = createWriteStream('data.bin') let bytes = 0 for await (const chunk of process.stdin) { const canContinue = dest.write(chunk) bytes += chunk.length if (!canContinue) { // backpressure, now we stop and we need to wait for drain await once(dest, 'drain') // ok now it's safe to resume writing } } dest.end() console.log(`${bytes} written into data.bin`) loige βœ… handling backpressure like a pro! ... or you can use pipeline() 66
  67. import { pipeline } from 'node:stream/promises' import {createReadStream, createWriteStream} from

    'node:fs' await pipeline( createReadStream('lowercase.txt'), async function* (source) { for await (const chunk of source) { yield await processChunk(chunk) } }, createWriteStream('uppercase.txt') ) console.log('Pipeline succeeded.') loige pipeline() supports async iterables! 😱 67
  68. import { on } from 'node:events' import glob from 'glob'

    // from npm const matcher = glob('**/*.js') for await (const [filePath] of on(matcher, 'match')) { console.log(filePath) } loige creates an async iterable that will yield every time the `match` event happens 68
  69. import { on } from 'node:events' import glob from 'glob'

    // from npm const matcher = glob('**/*.js') for await (const [filePath] of on(matcher, 'match')) { console.log(filePath) } // ⚠ WE WILL NEVER GET HERE πŸ‘‡ console.log('Done') loige This loop doesn't know when to stop! You could pass an AbortController here for more fine-grained control 69
  70. πŸ“ FINAL SUMMARY loige You can check if an object

    is an iterable by checking `typeof obj[Symbol.iterator] === 'function'` You can check if an object is an async iterable with `typeof obj[Symbol.asyncIterator] === 'function'` In both cases, there's no guarantee that the iterable protocol is implemented correctly (the function might not return an iterator πŸ˜₯) Node.js Readable streams are also async iterable objects, so you could use `for await ... of` to consume the data in chunks If you do that and you end up writing data somewhere else, you'll need to handle backpressure yourself. It might be better to use `pipeline()` instead. You can convert Node.js event emitters to async iterable objects by using the `on` function from the module `events`. 70
  71. Cover picture by on Back cover picture by on Camille

    Minouflet Unsplash Antonino Cicero Unsplash fourtheorem.com THANKS! πŸ™Œ loige.link/lets-iter loige nodejsdp.link 71