Upgrade to Pro — share decks privately, control downloads, hide ads and more …

JavaScript Iteration Protocols - Workshop NodeConf EU 2022

Luciano Mammino
October 03, 2022
36

JavaScript Iteration Protocols - Workshop NodeConf EUΒ 2022

How many ways do you know to do iteration with JavaScript and Node.js? While, for loop, for...in, for..of, .map(), .forEach(), streams, iterators, etc! Yes, there are a lot of ways! But did you know that JavaScript has iteration protocols to standardise synchronous and even asynchronous iteration? In this workshop we will learn about these protocols and discover how to build iterators and iterable objects, both synchronous and asynchronous. We will learn about some common use cases for these protocols, explore generators and async generators (great tools for iteration) and finally discuss some hot tips, common pitfalls, and some (more or less successful) wild ideas!

Luciano Mammino

October 03, 2022
Tweet

Transcript

  1. META_SLIDE!
    loige
    loige.link/lets-iter
    2

    View full-size slide

  2. const array = ['foo', 'bar', 'baz']
    for (const item of array) {
    console.log(item)
    }
    loige
    Output:
    foo
    bar
    baz
    3

    View full-size slide

  3. const str = 'foo'
    for (const item of str) {
    console.log(item)
    }
    loige
    Output:
    f
    o
    o
    4

    View full-size slide

  4. const set = new Set(['foo', 'bar', 'baz'])
    for (const item of set) {
    console.log(item)
    }
    loige
    Output:
    foo
    bar
    baz
    5

    View full-size slide

  5. const map = new Map([
    ['foo', 'bar'], ['baz', 'qux']
    ])
    for (const item of map) {
    console.log(item)
    }
    loige
    Output:
    [ 'foo', 'bar' ]
    [ 'baz', 'qux' ]
    6

    View full-size slide

  6. const obj = {
    foo: 'bar',
    baz: 'qux'
    }
    for (const item of obj) {
    console.log(item)
    }
    loige
    Output:
    β›”
    Uncaught TypeError: obj is not iterable
    OMG `for ... of`
    does not work with plain objects!
    😱
    7

    View full-size slide

  7. const obj = {
    foo: 'bar',
    baz: 'qux'
    }
    for (const item of Object.entries(obj)) {
    console.log(item)
    }
    loige
    Output:
    [ 'foo', 'bar' ]
    [ 'baz', 'qux' ]
    8

    View full-size slide

  8. const array = ['foo', 'bar', 'baz']
    console.log(...array)
    loige
    Output:
    foo bar baz
    spread syntax!
    10

    View full-size slide

  9. loige
    Iterators
    Iterables
    Async Iterators
    Async Iterables
    Generator functions
    Async Generators
    πŸ“Έ THE BIGGER PICTURE
    13

    View full-size slide

  10. loige
    Knowing iteration protocols allows us:
    Understand JavaScript better
    Write more modern, interoperable and idiomatic code
    Be able to write our own custom iterables (even async)
    πŸ˜₯ WHY SHOULD WE CARE?
    14

    View full-size slide

  11. WHO IS THIS GUY !?
    πŸ‘‹ I'm Luciano (
    πŸ•πŸ)
    Senior Architect @ fourTheorem (Dublin )
    nodejsdp.link
    πŸ“” Co-Author of Node.js Design Patterns
    πŸ‘‰
    Let's connect!
    (blog)
    (twitter)
    (twitch)
    (github)
    loige.co
    @loige
    loige
    lmammino
    15

    View full-size slide

  12. ALWAYS RE-IMAGINING
    WE ARE A PIONEERING TECHNOLOGY CONSULTANCY FOCUSED ON AWS AND SERVERLESS
    | |
    Accelerated Serverless AI as a Service Platform Modernisation
    loige
    βœ‰ Reach out to us at
    πŸ˜‡ We are always looking for talent:
    [email protected]
    fth.link/careers
    16

    View full-size slide

  13. We host a weekly podcast about AWS
    loige
    awsbites.com
    loige 17

    View full-size slide

  14. πŸ“’ AGENDA
    Generators
    Iterator protocol
    Iterable protocol
    Async Iterator protocol
    Async Iterable protcol
    Real-lifeβ„’ examples
    loige 18

    View full-size slide

  15. LET'S CLONE
    πŸ‘πŸ
    loige
    loige.link/lets-iter-repo
    git clone https://github.com/lmammino/iteration-protocols-workshop.git
    cd iteration-protocols-workshop
    npm i
    19

    View full-size slide

  16. ➑ GENERATORS
    loige 20

    View full-size slide

  17. GENERATOR FN & OBJ
    loige
    function * myGenerator () {
    // generator body
    yield 'someValue'
    // ... do more stuff
    }
    const genObj = myGenerator()
    genObj.next() // -> { done: false, value: 'someValue' }
    21

    View full-size slide

  18. function * fruitGen () {
    yield '
    πŸ‘
    '
    yield '
    πŸ‰
    '
    yield '
    πŸ‹
    '
    yield '
    πŸ₯­
    '
    }
    const fruitGenObj = fruitGen()
    console.log(fruitGenObj.next()) // { value: '
    πŸ‘
    ', done: false }
    console.log(fruitGenObj.next()) // { value: '
    πŸ‰
    ', done: false }
    console.log(fruitGenObj.next()) // { value: '
    πŸ‹
    ', done: false }
    console.log(fruitGenObj.next()) // { value: '
    πŸ₯­
    ', done: false }
    console.log(fruitGenObj.next()) // { value: undefined, done: true }
    loige 22

    View full-size slide

  19. function * fruitGen () {
    yield '
    πŸ‘
    '
    yield '
    πŸ‰
    '
    yield '
    πŸ‹
    '
    yield '
    πŸ₯­
    '
    }
    const fruitGenObj = fruitGen()
    // generator objects are iterable!
    for (const fruit of fruitGenObj) {
    console.log(fruit)
    }
    //
    πŸ‘
    //
    πŸ‰
    //
    πŸ‹
    //
    πŸ₯­
    loige 23

    View full-size slide

  20. function * range (start, end) {
    for (let i = start; i < end; i++) {
    yield i
    }
    }
    // generators are lazy!
    for (const i of range(0, Number.MAX_VALUE)) {
    console.log(i)
    }
    const zeroToTen = [...range(0, 11)]
    loige 24

    View full-size slide

  21. // generators can be "endless"
    function * cycle (values) {
    let current = 0
    while (true) {
    yield values[current]
    current = (current + 1) % values.length
    }
    }
    for (const value of cycle(['even', 'odd'])) {
    console.log(value)
    }
    // even
    // odd
    // even
    // ...
    loige 25

    View full-size slide

  22. πŸ“ MINI-SUMMARY
    loige
    A generator function returns a generator object which is both an iterator and an iterable.
    A generator function uses `yield` to yield a value and pause its execution. The generator
    object is used to make progress on an instance of the generator (by calling `next()`).
    Generator functions are a great way to create custom iterable objects.
    Generator objects are lazy and they can be endless.
    26

    View full-size slide

  23. πŸ“ EXERCISE(S)
    loige
    02-generators/exercises/zip.js
    function * take (n, iterable) {
    // take at most n items from iterable and
    // yield them one by one (lazily)
    }
    02-generators/exercises/zip.js
    function * zip (iterable1, iterable2) {
    // consume the two iterables until any of the 2 completes
    // yield a pair taking 1 item from the first and one from
    // the second at every step
    }
    27

    View full-size slide

  24. ➑ ITERATORS & ITERABLES
    loige 28

    View full-size slide

  25. ITERATOR OBJ
    loige
    An object that acts like a cursor to iterate over blocks of data sequentially
    29

    View full-size slide

  26. ITERABLE OBJ
    loige
    An object that contains data that can be iterated over sequentially
    30

    View full-size slide

  27. THE ITERATOR PROTOCOL
    An object is an iterator if it has a next() method.
    Every time you call it, it returns an object with the
    keys done (boolean) and value.
    loige 31

    View full-size slide

  28. function createCountdown (from) {
    let nextVal = from
    return {
    next () {
    if (nextVal < 0) {
    return { done: true }
    }
    return {
    done: false,
    value: nextVal--
    }
    }
    }
    }
    loige 32

    View full-size slide

  29. const countdown = createCountdown(3)
    console.log(countdown.next())
    // { done: false, value: 3 }
    console.log(countdown.next())
    // { done: false, value: 2 }
    console.log(countdown.next())
    // { done: false, value: 1 }
    console.log(countdown.next())
    // { done: false, value: 0 }
    console.log(countdown.next())
    // { done: true }
    loige 33

    View full-size slide

  30. πŸ”₯
    Generator Objects are iterators (and iterables)!
    loige 34

    View full-size slide

  31. function * createCountdown (from) {
    for (let i = from; i >= 0; i--) {
    yield i
    }
    }
    loige 35

    View full-size slide

  32. const countdown = createCountdown(3)
    console.log(countdown.next())
    // { done: false, value: 3 }
    console.log(countdown.next())
    // { done: false, value: 2 }
    console.log(countdown.next())
    // { done: false, value: 1 }
    console.log(countdown.next())
    // { done: false, value: 0 }
    console.log(countdown.next())
    // { done: true, value: undefined }
    loige 36

    View full-size slide

  33. THE ITERABLE PROTOCOL
    An object is iterable if it implements the
    Symbol.iterator method, a zero-argument function
    that returns an iterator.
    loige 37

    View full-size slide

  34. function createCountdown (from) {
    let nextVal = from
    return {
    [Symbol.iterator]: () => ({
    next () {
    if (nextVal < 0) {
    return { done: true }
    }
    return { done: false, value: nextVal-- }
    }
    })
    }
    }
    loige 38

    View full-size slide

  35. function createCountdown (from) {
    return {
    [Symbol.iterator]: function * () {
    for (let i = from; i >= 0; i--) {
    yield i
    }
    }
    }
    }
    loige 39

    View full-size slide

  36. function * createCountdown () {
    for (let i = from; i >= 0; i--) {
    yield i
    }
    }
    loige
    πŸ”₯ or just use generators!
    40

    View full-size slide

  37. const countdown = createCountdown(3)
    for (const value of countdown) {
    console.log(value)
    }
    // 3
    // 2
    // 1
    // 0
    loige 41

    View full-size slide

  38. const iterableIterator = {
    next () {
    return { done: false, value: 'hello' }
    },
    [Symbol.iterator] () {
    return this
    }
    }
    An object can be an iterable and an iterator at the same time!
    loige 42

    View full-size slide

  39. πŸ“ MINI-SUMMARY 1/2
    loige
    An iterator is an object that allows us to traverse a collection
    The iterator protocol specifies that an object is an iterator if it has a `next()` method that
    returns an object with the shape `{done, value}`.
    `done` (a boolean) tells us if the iteration is completed
    `value` represents the value from the current iteration.
    You can write an iterator as an anonymous object (e.g. returned by a factory function),
    using classes or using generators.
    43

    View full-size slide

  40. πŸ“ MINI-SUMMARY 2/2
    loige
    The iterable protocol defines what's expected for a JavaScript object to be considered
    iterable. That is an object that holds a collection of data on which you can iterate on
    sequentially.
    An object is iterable if it implements a special method called `Symbol.iterator` which
    returns an iterator. (An object is iterable if you can get an iterator from it!)
    Generator functions produce objects that are iterable.
    We saw that generators produce objects that are also iterators.
    It is possible to have objects that are both iterator and iterable. The trick is to create the
    object as an iterator and to implement a `Symbol.iterator` that returns the object itself
    (`this`).
    44

    View full-size slide

  41. πŸ“ EXERCISE(S)
    loige
    04-iterable-protocol/exercises/binarytree.js
    class BinaryTree {
    //
    // ...
    // make this iterable!
    }
    04-iterable-protocol/exercises/entries.js
    function entriesIterable (obj) {
    // Return an iterable that produce key/value pairs
    // from obj
    }
    45

    View full-size slide

  42. OK, very cool!
    But, so far this is all synchronous iteration.
    What about async?
    πŸ™„
    loige 46

    View full-size slide

  43. ➑ ASYNC ITERATORS & ITERABLES
    loige 47

    View full-size slide

  44. THE ASYNC ITERATOR PROTOCOL
    An object is an async iterator if it has a next() method.
    Every time you call it, it returns a promise that resolves
    to an object with the keys done (boolean) and value.
    loige 48

    View full-size slide

  45. import { setTimeout } from 'node:timers/promises'
    function createAsyncCountdown (from, delay = 1000) {
    let nextVal = from
    return {
    async next () {
    await setTimeout(delay)
    if (nextVal < 0) {
    return { done: true }
    }
    return { done: false, value: nextVal-- }
    }
    }
    }
    loige 49

    View full-size slide

  46. const countdown = createAsyncCountdown(3)
    console.log(await countdown.next())
    // { done: false, value: 3 }
    console.log(await countdown.next())
    // { done: false, value: 2 }
    console.log(await countdown.next())
    // { done: false, value: 1 }
    console.log(await countdown.next())
    // { done: false, value: 0 }
    console.log(await countdown.next())
    // { done: true }
    loige 50

    View full-size slide

  47. import { setTimeout } from 'node:timers/promises'
    // async generators "produce" async iterators!
    async function * createAsyncCountdown (from, delay = 1000) {
    for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
    }
    }
    loige 52

    View full-size slide

  48. THE ASYNC ITERABLE PROTOCOL
    An object is an async iterable if it implements the
    `Symbol.asyncIterator` method, a zero-argument
    function that returns an async iterator.
    loige 53

    View full-size slide

  49. import { setTimeout } from 'node:timers/promises'
    function createAsyncCountdown (from, delay = 1000) {
    return {
    [Symbol.asyncIterator]: async function * () {
    for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
    }
    }
    }
    }
    loige 54

    View full-size slide

  50. HOT TIP
    πŸ”₯
    With async generators we can create objects that are
    both async iterators and async iterables!
    (We don't need to specify
    Symbol.asyncIterator explicitly!)
    loige 55

    View full-size slide

  51. import { setTimeout } from 'node:timers/promises'
    // async generators "produce" async iterators
    // (and iterables!)
    async function * createAsyncCountdown (from, delay = 1000) {
    for (let i = from; i >= 0; i--) {
    await setTimeout(delay)
    yield i
    }
    }
    loige 56

    View full-size slide

  52. const countdown = createAsyncCountdown(3)
    for await (const value of countdown) {
    console.log(value)
    }
    loige 57

    View full-size slide

  53. πŸ“ MINI-SUMMARY 1/2
    loige
    Async iterators are the asynchronous counterpart of iterators.
    They are useful to iterate over data that becomes available asynchronously (e.g. coming
    from a database or a REST API).
    A good example is a paginated API, we could build an async iterator that gives a new page
    for every iteration.
    An object is an async iterator if it has a `next()` method which returns a `Promise` that
    resolves to an object with the shape: `{done, value}`.
    The main difference with the iterator protocol is that this time `next()` returns a promise.
    When we call next we need to make sure we `await` the returned promise.
    58

    View full-size slide

  54. πŸ“ MINI-SUMMARY 2/2
    loige
    The async iterable protocol defines what it means for an object to be an async iterable.
    Once you have an async iterable you can use the `for await ... of` syntax on it.
    An object is an async iterable if it has a special method called `Symbol.asyncIterator` that
    returns an async iterator.
    Async iterables are a great way to abstract paginated data that is available
    asynchronously or similar operations like pulling jobs from a remote queue.
    A small spoiler, async iterables can also be used with Node.js streams...
    59

    View full-size slide

  55. πŸ“ EXERCISE
    loige
    06-async-iterable-protocol/exercises/rickmorty.js
    function createCharactersPaginator () {
    // return an iterator that produces pages
    // with name of Rick and Morty characters
    // taken from the API
    // https://rickandmortyapi.com/api/character
    }
    // This is what we want to support
    πŸ‘‡
    const paginator = createCharactersPaginator()
    for await (const page of paginator) {
    console.log(page)
    }
    60

    View full-size slide

  56. ➑ TIPS &
    ASYNC ITERATORS IN NODE.JS
    loige 61

    View full-size slide

  57. function isIterable (obj) {
    return typeof obj[Symbol.iterator] === 'function'
    }
    const array = [1, 2, 3]
    console.log(array, isIterable(array)) // true
    const genericObj = { foo: 'bar' }
    console.log(isIterable(genericObj)) // false
    console.log(isIterable(Object.entries(genericObj))) // true
    const fakeIterable = {
    [Symbol.iterator] () { return 'notAnIterator' }
    }
    console.log(fakeIterable, isIterable(fakeIterable)) // true
    😑
    IS THIS OBJECT AN ITERABLE?
    loige 62

    View full-size slide

  58. function isAsyncIterable (obj) {
    return typeof obj[Symbol.asyncIterator] === 'function'
    }
    IS THIS OBJECT AN ASYNC ITERABLE?
    loige
    Are there async iterable objects in Node.js core?
    63

    View full-size slide

  59. console.log(
    typeof process.stdin[Symbol.asyncIterator] === 'function'
    ) // true!
    let bytes = 0
    for await (const chunk of process.stdin) {
    bytes += chunk.length
    }
    console.log(`${bytes} bytes read from stdin`)
    READABLE STREAMS!
    loige 64

    View full-size slide

  60. import { createWriteStream } from 'node:fs'
    const dest = createWriteStream('data.bin')
    let bytes = 0
    for await (const chunk of process.stdin) {
    // what if we are writing too much too fast?!!
    😱
    dest.write(chunk)
    bytes += chunk.length
    }
    dest.end()
    console.log(`${bytes} written into data.bin`)
    ⚠ BACKPRESSURE WARNING
    loige 65

    View full-size slide

  61. import { createWriteStream } from 'node:fs'
    import { once } from 'node:events'
    const dest = createWriteStream('data.bin')
    let bytes = 0
    for await (const chunk of process.stdin) {
    const canContinue = dest.write(chunk)
    bytes += chunk.length
    if (!canContinue) {
    // backpressure, now we stop and we need to wait for drain
    await once(dest, 'drain')
    // ok now it's safe to resume writing
    }
    }
    dest.end()
    console.log(`${bytes} written into data.bin`)
    loige
    βœ… handling backpressure like a pro!
    ... or you can use pipeline()
    66

    View full-size slide

  62. import { pipeline } from 'node:stream/promises'
    import {createReadStream, createWriteStream} from 'node:fs'
    await pipeline(
    createReadStream('lowercase.txt'),
    async function* (source) {
    for await (const chunk of source) {
    yield await processChunk(chunk)
    }
    },
    createWriteStream('uppercase.txt')
    )
    console.log('Pipeline succeeded.')
    loige
    pipeline() supports async iterables!
    😱
    67

    View full-size slide

  63. import { on } from 'node:events'
    import glob from 'glob' // from npm
    const matcher = glob('**/*.js')
    for await (const [filePath] of on(matcher, 'match')) {
    console.log(filePath)
    }
    loige
    creates an async iterable that will yield
    every time the `match` event happens
    68

    View full-size slide

  64. import { on } from 'node:events'
    import glob from 'glob' // from npm
    const matcher = glob('**/*.js')
    for await (const [filePath] of on(matcher, 'match')) {
    console.log(filePath)
    }
    //
    ⚠
    WE WILL NEVER GET HERE
    πŸ‘‡
    console.log('Done')
    loige
    This loop doesn't know when to stop!
    You could pass an
    AbortController here
    for more fine-grained
    control
    69

    View full-size slide

  65. πŸ“ FINAL SUMMARY
    loige
    You can check if an object is an iterable by checking `typeof obj[Symbol.iterator] === 'function'`
    You can check if an object is an async iterable with `typeof obj[Symbol.asyncIterator] === 'function'`
    In both cases, there's no guarantee that the iterable protocol is implemented correctly (the function
    might not return an iterator
    πŸ˜₯)
    Node.js Readable streams are also async iterable objects, so you could use `for await ... of` to
    consume the data in chunks
    If you do that and you end up writing data somewhere else, you'll need to handle backpressure
    yourself. It might be better to use `pipeline()` instead.
    You can convert Node.js event emitters to async iterable objects by using the `on` function from the
    module `events`.
    70

    View full-size slide

  66. Cover picture by on
    Back cover picture by on
    Camille Minouflet Unsplash
    Antonino Cicero Unsplash
    fourtheorem.com
    THANKS!
    πŸ™Œ
    loige.link/lets-iter
    loige
    nodejsdp.link
    71

    View full-size slide