Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Using Async Iterators in Node.js

Using Async Iterators in Node.js

This talk demonstrates how to use async iterators and the new for-await-of syntax in Node.js 10. It includes some real-world examples, like iterating over the contents of a file, handling a streaming database query, and using SSE to publish the values yielded by an async iterator. The talk also dives into cancellation patterns for promises and async iterators.

Ryan Paul

May 09, 2018
Tweet

More Decks by Ryan Paul

Other Decks in Programming

Transcript

  1. Async Iterators
    REAL WORLD USAGE in NODE.JS 10

    View Slide

  2. Ryan Paul
    http://seg.phault.net

    View Slide

  3. Iterators & Async Functions
    Bringing Together

    View Slide

  4. Availability
    Supported Not Yet
    Chrome 63 Safari (Preview)
    Node.js 10 Edge
    Firefox 55

    View Slide

  5. Async Iterators in Theory
    •Write generator functions that yield promises
    •Loop over async iterators with the new for-
    await-of syntax
    •A useful abstraction for representing streams

    View Slide

  6. Async Iterators in Practice
    const timer = time =>
    new Promise(resolve => setTimeout(resolve, time));
    async function* counter() {
    let counter = 0;
    while (true) {
    await timer(5000);
    yield counter++;
    }
    }

    View Slide

  7. Consuming an Async Iterator
    let iter = counter();
    for await (let item of iter)
    console.log(item)
    Using the for-await-of syntax

    View Slide

  8. Underneath the Sugar
    •An iterator is an object with a next() function
    •The next() function returns an object with value
    and done properties
    •The value is the yielded value and done is a
    boolean indicating whether the iterator is complete

    View Slide

  9. Async Iteration Protocol
    •Use the async iteration protocol to turn any object
    into an async iterator
    •Objects that conform with the protocol can be
    consumed with for-await-of
    •The protocol uses Symbol.asyncIterator

    View Slide

  10. Async Iteration Protocol
    class Example {
    async *[Symbol.asyncIterator]() {
    let counter = 0;
    while (true) {
    await timer(5000);
    yield counter++;
    }
    }
    }
    let example = new Example();
    for await (let item of example)
    console.log(item);

    View Slide

  11. Some Practical Examples
    Real-World Async Iterators

    View Slide

  12. Using Async Iterators in the Real World
    •You can use async iterators today in node.js 10
    •Every ReadableStream instance supports the
    async iteration protocol
    •Useful for I/O and many other everyday tasks

    View Slide

  13. Reading Files with Async Iterators
    let file = fs.createReadStream("file.txt");
    let content = "";
    for await (let chunk of file)
    content += chunk;
    console.log(content);

    View Slide

  14. Streaming Database Query
    • RethinkDB is an open source database for building real-
    time web applications
    • RethinkDB changefeed lets you stream updates from a
    table or query
    • You can wrap a changefeed cursor with an async iterator
    so that it can be consumed with a for-await-of loop

    View Slide

  15. Streaming Database Query
    const r = require("rethinkdbdash")();
    function* stream(cursor) {
    while (true)
    yield cursor.next();
    }
    let query = await r.db("rethinkdb").table("stats").changes();
    for await (let item of stream(query))
    console.log(item);

    View Slide

  16. Serving as an SSE Event Stream
    • Server Sent Events (SSE) is a standard for building real-
    time web applications
    • Broadcast a stream of events in response to an HTTP
    request
    • Easy to consume in the browser with the EventSource
    client API

    View Slide

  17. Serving as an SSE Event Stream
    async function iterStream(res, iter) {
    let header = {"Content-Type": "text/event-stream"};
    res.writeHead(200, header);
    for await (let item of iter)
    res.write(`data: ${item}\n\n`);
    }
    http.createServer((req, res) =>
    iterStream(res, counter())).listen(8000);

    View Slide

  18. Serving as an SSE Event Stream
    let events = new EventSource("/");
    events.onmessage = ev => console.log(ev);
    Create an EventSource instance in the browser and
    attach a function to onmessage to see events

    View Slide

  19. But what happens when the
    client disconnects?

    View Slide

  20. Serving as an SSE Event Stream
    • The generator continues to run when the client
    disconnects
    • We shouldn’t leave a dangling generator every time a
    user loads the page
    • Naive solution: what if we invoke iter.return() when
    the connection closes?

    View Slide

  21. Serving as an SSE Event Stream
    async function iterStream(res, iter) {
    res.connection.on("close", () => iter.return());
    let header = {"Content-Type": "text/event-stream"};
    res.writeHead(200, header);
    for await (let item of iter)
    res.write(`data: ${item}\n\n`);
    }
    Watch for the connection to close and call the
    iter.return() function

    View Slide

  22. Serving as an SSE Event Stream
    •Calling iter.return() doesn’t terminate the
    pending promise
    •The pending promise continues until it rejects or
    resolves
    •We need a way to cancel the pending promise

    View Slide

  23. Promise Cancellation
    Introducing

    View Slide

  24. Promise Cancellation
    function timer(time, cancelled) {
    let handle;
    return new Promise((resolve, reject) => {
    handle = setTimeout(resolve, time);
    cancelled.then(() => clearTimeout(handle));
    });
    }
    let cancel, cancelled = new Promise(resolve => cancel = resolve);
    timer(5000, cancelled);
    cancel();

    View Slide

  25. Promise Cancellation
    let cancel;
    let cancelled = new Promise(resolve => cancel = resolve);
    timer(5000, cancelled);
    cancel();
    Create a promise and assign its resolve function to a
    variable in the parent scope

    View Slide

  26. Promise Cancellation
    function timer(time, cancelled) {
    let handle;
    return new Promise((resolve, reject) => {
    handle = setTimeout(resolve, time);
    cancelled.then(() => clearTimeout(handle));
    });
    }
    Pass the cancelled promise into the function and
    terminate the timer when it resolves

    View Slide

  27. Promise Cancellation
    function timer(time, cancelled) {
    let handle;
    return new Promise((resolve, reject) => {
    handle = setTimeout(resolve, time);
    cancelled.then(() => {
    reject(new Error("Cancelled"));
    clearTimeout(handle);
    });
    });
    }
    Reject the promise so that all downstream subscribers
    know that it was cancelled

    View Slide

  28. SSE Example with Cancellation
    Putting it all Together

    View Slide

  29. SSE Example with Cancellation
    async function iterStream(res, iter, cancel) {
    let header = {"Content-Type": "text/event-stream"};
    res.writeHead(200, header);
    res.connection.on("close", cancel);
    for await (let item of iter)
    res.write(`data: ${item}\n\n`);
    }
    Add an extra parameter for the cancel() function and
    invoke it when the user closes their connection

    View Slide

  30. SSE Example with Cancellation
    async function* counter(cancelled) {
    let counter = 0;
    while (true) {
    try {
    await timer(5000, cancelled);
    yield counter++;
    } catch (err) {
    if (err.message === "Cancelled") break;
    }
    }
    }
    Pass the cancelled promise into the timer and force
    the loop to break when the exception is thrown

    View Slide

  31. SSE Example with Cancellation
    let cancel, cancelled = new Promise(resolve => cancel = resolve);
    http.createServer((req, res) =>
    iterStream(res, counter(cancelled), cancel)).listen(8000);
    Create the cancellation promise and pass it into the
    iterStream() function

    View Slide

  32. Resources
    • My Blog post about cancellation and async iterators:
    http://seg.phault.net/blog/2018/03/async-iterators-
    cancellation/
    • Async Iterators TC39 proposal:
    https://github.com/tc39/proposal-async-iteration
    • Follow me on Twitter: @segphault

    View Slide