Building Async APIs with HTTP/2+ and Mercure

Mercure.rocks is a new protocol allowing to push data updates to web browsers and other HTTP clients in a convenient, fast, reliable and battery-efficient way. It is especially useful to publish real-time updates of resources served through web APIs, to reactive web and mobile apps.

From the ground up, Mercure has been designed to work with technologies not able to maintain persistent connections. It's especially relevant in serverless environments, but is also convenient when using PHP or FastCGI scripts.

Mercure is basically a higher-level replacement for WebSockets. But unlike WebSockets, it leverages the new capabilities of HTTP/2 and HTTP/3.
It has been designed with hypermedia APIs in mind, is auto-discoverable through the Web Linking RFC and is also compatible with GraphQL.
It natively supports authorization, reconnection in case of network issue (with refetching of missed events), subscribing to several topics, topics patterns (using templated URIs)...

Because it is built on top of Server-sent Events and plain old HTTP requests, it is already compatible with all modern browsers, and requires 0 client-side dependencies.

The protocol is open (available as an Internet Draft), and a reference open source implementation of the server written in Go is available. Integrations with popular web frameworks are already available!

During this talk, you'll learn everything you need to get started with the Mercure protocol!

Transcript

1. @dunglas - mercure.rocks
   Building Async APIs with
   HTTP/2+ and
   

   View Slide

2. @dunglas - mercure.rocks
   Kévin Dunglas
   ❏ Founder of Les-Tilleuls.coop
   ❏ Creator of Mercure, Vulcain, API Platform…
   ❏ Symfony Core Team
   @dunglas
   

   View Slide

3. @dunglas - mercure.rocks
   API and cloud experts
   ✊ worker-owned, self-managed cooperative
   50 people, 1,000% growth in 6 years
   contact@les-tilleuls.coop
   Les-Tilleuls.coop
   

   View Slide

4. @dunglas - mercure.rocks
   Async and Real-Time
   APIs?!
   

   View Slide

5. @dunglas - mercure.rocks
   

   View Slide

6. @dunglas - mercure.rocks
   Why a New Protocol?
   

   View Slide

7. © Ilya Grigorik
   High Performance Browser Networking (O'Reilly)
   

   View Slide

8. @dunglas - mercure.rocks
   WebSocket (RFC 6455)
   Benefits over SSE:
   ❏ Full-duplex
   communication
   ❏ Low level: full control
   Drawbacks over SSE:
   ❏ Low level: no native
   ❏ auth
   ❏ re-connection
   ❏ state reconciliation
   ❏ events history
   ❏ Obsoleted by HTTP/2 & 3
   ❏ Hard to secure
   

   View Slide

9. @dunglas - mercure.rocks
   The Persistent Connections Problem
   ❏ WebSocket and SSE rely on persistent connections
   ❏ Serverless platforms (AWS Lambda, Cloud Run, Azure
   functions…), PHP, FastCGI…

   are designed for short-lived connections
   ❏ Persistent, long-lived, concurrent connections are better
   handled by

   dedicated and optimized software and hardware
   

   View Slide

10. View Slide

11. @dunglas - mercure.rocks
    The Mercure Protocol
    

    View Slide

12. @dunglas - mercure.rocks
    

    View Slide

13. @dunglas - mercure.rocks
    Mercure, at a Glance
    ❏ Full-duplex, but plain old HTTP
    ❏ Publish: HTTP POST
    ❏ Subscribe: SSE
    ❏ Built-in: reconnection, retrieving of lost messages, history
    ❏ Auto-discoverable: designed for REST and GraphQL
    ❏ JWT-based authorization mechanism (private updates)
    ❏ Designed for serverless, PHP, FastCGI…
    ❏ End-2-End encryption support
    

    View Slide

14. @dunglas - mercure.rocks
    Stream Events to Clients
    

    View Slide

15. @dunglas - mercure.rocks
    Internet Draft: draft-dunglas-mercure
    

    View Slide

16. View Slide

17. @dunglas - mercure.rocks
    Mercure and HTTP/2+
    

    View Slide

18. @dunglas - mercure.rocks
    © Narayan Prusty
    What is Multiplexing in HTTP/2?
    HTTP/2 Multiplexing
    

    View Slide

19. @dunglas - mercure.rocks
    HTTP/2 global support:

    95% of all internet users
    ...and SSEs also work with HTTP/1.x
    

    View Slide

20. @dunglas - mercure.rocks
    SSE global support:

    100% of all internet users
    there is a polyfill for IE 5.5+
    

    View Slide

21. @dunglas - mercure.rocks
    Publishing
    

    View Slide

22. @dunglas - mercure.rocks
    Publishing
    POST /.well-known/mercure
    Content-Type: application/x-www-form-urlencoded
    Authorization: Bearer:
    topic=&data=
    

    View Slide

23. @dunglas - mercure.rocks
    const http = require('http');
    const querystring = require('querystring');
    const postData = querystring.stringify({
    'topic': 'https://example.com/books/1',
    'data': JSON.stringify({ '@id': 'https://example.com/books/1', status: 'OufOfStock'}),
    });
    const req = http.request({
    hostname: 'demo.mercure.rocks',
    port: '80',
    path: '/.well-known/mercure',
    method: 'POST',
    headers: {
    Authorization: 'Bearer snip',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Content-Length': Buffer.byteLength(postData),
    }
    });
    req.write(postData);req.end();
    Publishing: Node.js
    

    View Slide

24. @dunglas - mercure.rocks
    Publishing: Symfony (PHP)
    

    View Slide

25. @dunglas - mercure.rocks
    Subscribing
    

    View Slide

26. @dunglas - mercure.rocks
    Subscribing: Basic Usage
    const es = new EventSource(
    'https://demo.mercure.rocks/.well-known/mercure?topic=https://example.com/books/1'
    );
    // The callback will be called every time an update is published
    es.onmessage = function (e) {
    console.log(JSON.parse(e.data));
    }
    

    View Slide

27. @dunglas - mercure.rocks
    Subscribing: Several Topics
    const url = new URL('https://demo.mercure.rocks/.well-known/mercure');
    url.searchParams.append('topic', 'https://example.com/books/1')
    url.searchParams.append('topic', 'https://example.com/reviews/21')
    const es = new EventSource(url);
    es.onmessage = function (e) {
    console.log(JSON.parse(e.data));
    }
    

    View Slide

28. @dunglas - mercure.rocks
    Subscribing: using patterns
    const url = new URL('https://demo.mercure.rocks/.well-known/mercure');
    // URI Template (RFC 6560)
    url.searchParams.append('topic', 'https://example.com/books/{id}')
    // Both https://example.com/books/1 and https://example.com/books/33 will match
    const es = new EventSource(url);
    es.onmessage = function (e) {
    console.log(JSON.parse(e.data));
    }
    

    View Slide

29. View Slide

30. @dunglas - mercure.rocks
    The Discovery Mechanism
    

    View Slide

31. @dunglas - mercure.rocks
    Discovery Mechanism
    Web Linking: RFC 5988
    

    View Slide

32. @dunglas - mercure.rocks
    Authorization
    

    View Slide

33. @dunglas - mercure.rocks
    Authorization
    ❏ Uses JSON Web Token (JWT - RFC 7519)
    ❏ An update can be intended for one or several targets
    ❏ Publisher: must be authenticated
    ❏ Subscriber:
    ❏ Can be anonymous (if allowed by the config)
    ❏ Must be authenticated to receive private updates
    ❏ Two transports: cookie and Authorization header
    

    View Slide

34. @dunglas - mercure.rocks
    Authorization
    

    View Slide

35. @dunglas - mercure.rocks
    The Mercure Hub
    

    View Slide

36. @dunglas - mercure.rocks
    Reference Implementation
    ❏ Implements 100% of the Mercure protocol
    ❏ Fast, written in Go
    ❏ Works everywhere: static binaries and Docker
    ❏ Automatic HTTP/2 and HTTPS (Let’s Encrypt)
    ❏ CORS support, CSRF protection
    ❏ Cloud Native (12Factor App)
    ❏ Open source (AGPL)
    ❏ Optional: a server can implement directly the protocol
    

    View Slide

37. @dunglas - mercure.rocks
    Reference Implementation
    https://mercure.rocks
    

    View Slide

38. @dunglas - mercure.rocks
    Starting the Hub
    https://mercure.rocks
    

    View Slide

39. @dunglas - mercure.rocks
    On a Kubernetes Cluster
    

    View Slide

40. @dunglas - mercure.rocks
    Alternative Implementations
    ❏ No hubs: the hub is optional
    ❏ Go library (AGPL)
    ❏ Node.js library (GPL)
    ❏ HA version (managed and on premise)
    

    View Slide

41. @dunglas - mercure.rocks
    iGraal / Glory4Gamers Benchmarks
    ❏ Open Source version (EC2 t3.micro)
    ❏ 40k concurrent connections
    ❏ HA version (on premise)
    ❏ 200k concurrent connections
    

    View Slide

42. View Slide

43. @dunglas - mercure.rocks
    Integrations
    

    View Slide

44. @dunglas - mercure.rocks
    Integrations
    ❏ Official integrations:
    ❏ Symfony, API Platform
    ❏ Libs:
    ❏ Go, JS, Python, Java, Dart, PHP, Laravel
    

    View Slide

45. @dunglas - mercure.rocks
    Example using

    API Platform & React
    

    View Slide

46. @dunglas - mercure.rocks
    Create a Mercure-enabled Web API
    api-platform.com
    

    View Slide

47. You get a fully-featured API supporting:
    ❏ JSON-LD + Hydra + schema.org
    ❏ GraphQL, JSON:API, HAL
    ❏ OpenAPI
    ❏ auth, pagination, filters, validation,
    HTTP/2 push, test framework…
    ❏ and Mercure support!
    MIT license
    

    View Slide

48. @dunglas - mercure.rocks
    Scaffold Mercure-Enabled Clients
    api-platform.com
    

    View Slide

49. @dunglas - mercure.rocks
    Try it!
    bit.ly/2H17hN3
    

    View Slide

50. @dunglas - mercure.rocks
    Unleashing the Power of AsyncAPI
    servers:
    production:
    url: https://demo.mercure.rocks/.well-known/mercure
    protocol: mercure
    channels:
    'https://example.com/books/{id}':
    parameters:
    id:
    schema:
    type: integer
    subscribe:
    message:
    $ref: '#/components/messages/book'
    publish:
    message:
    $ref: '#/components/messages/book'
    

    View Slide

51. @dunglas les-tilleuls.coop
    .rocks
    

    View Slide