Building Async APIs with HTTP/2+ and Mercure

Mercure.rocks is a new protocol allowing to push data updates to web browsers and other HTTP clients in a convenient, fast, reliable and battery-efficient way. It is especially useful to publish real-time updates of resources served through web APIs, to reactive web and mobile apps.

From the ground up, Mercure has been designed to work with technologies not able to maintain persistent connections. It's especially relevant in serverless environments, but is also convenient when using PHP or FastCGI scripts.

Mercure is basically a higher-level replacement for WebSockets. But unlike WebSockets, it leverages the new capabilities of HTTP/2 and HTTP/3.
It has been designed with hypermedia APIs in mind, is auto-discoverable through the Web Linking RFC and is also compatible with GraphQL.
It natively supports authorization, reconnection in case of network issue (with refetching of missed events), subscribing to several topics, topics patterns (using templated URIs)...

Because it is built on top of Server-sent Events and plain old HTTP requests, it is already compatible with all modern browsers, and requires 0 client-side dependencies.

The protocol is open (available as an Internet Draft), and a reference open source implementation of the server written in Go is available. Integrations with popular web frameworks are already available!

During this talk, you'll learn everything you need to get started with the Mercure protocol!

Transcript

1. @dunglas - mercure.rocks Building Async APIs with HTTP/2+ and
2. @dunglas - mercure.rocks Kévin Dunglas ❏ Founder of Les-Tilleuls.coop ❏ Creator of Mercure, Vulcain, API Platform… ❏ Symfony Core Team @dunglas
3. @dunglas - mercure.rocks API and cloud experts ✊ worker-owned, self-managed cooperative 50 people, 1,000% growth in 6 years contact@les-tilleuls.coop Les-Tilleuls.coop
4. @dunglas - mercure.rocks Async and Real-Time APIs?!
5. @dunglas - mercure.rocks
6. @dunglas - mercure.rocks Why a New Protocol?
7. © Ilya Grigorik High Performance Browser Networking (O'Reilly)
8. @dunglas - mercure.rocks WebSocket (RFC 6455) Benefits over SSE: ❏ Full-duplex communication ❏ Low level: full control Drawbacks over SSE: ❏ Low level: no native ❏ auth ❏ re-connection ❏ state reconciliation ❏ events history ❏ Obsoleted by HTTP/2 & 3 ❏ Hard to secure
9. @dunglas - mercure.rocks The Persistent Connections Problem ❏ WebSocket and SSE rely on persistent connections ❏ Serverless platforms (AWS Lambda, Cloud Run, Azure functions…), PHP, FastCGI…
 are designed for short-lived connections ❏ Persistent, long-lived, concurrent connections are better handled by
 dedicated and optimized software and hardware
11. @dunglas - mercure.rocks The Mercure Protocol
12. @dunglas - mercure.rocks
13. @dunglas - mercure.rocks Mercure, at a Glance ❏ Full-duplex, but plain old HTTP ❏ Publish: HTTP POST ❏ Subscribe: SSE ❏ Built-in: reconnection, retrieving of lost messages, history ❏ Auto-discoverable: designed for REST and GraphQL ❏ JWT-based authorization mechanism (private updates) ❏ Designed for serverless, PHP, FastCGI… ❏ End-2-End encryption support
14. @dunglas - mercure.rocks Stream Events to Clients
15. @dunglas - mercure.rocks Internet Draft: draft-dunglas-mercure
17. @dunglas - mercure.rocks Mercure and HTTP/2+
18. @dunglas - mercure.rocks © Narayan Prusty What is Multiplexing in HTTP/2? HTTP/2 Multiplexing
19. @dunglas - mercure.rocks HTTP/2 global support:
 95% of all internet users ...and SSEs also work with HTTP/1.x
20. @dunglas - mercure.rocks SSE global support:
 100% of all internet users there is a polyfill for IE 5.5+
21. @dunglas - mercure.rocks Publishing
22. @dunglas - mercure.rocks Publishing POST /.well-known/mercure Content-Type: application/x-www-form-urlencoded Authorization: Bearer: <JWT> topic=<topic-url>&data=<update-content>
23. @dunglas - mercure.rocks const http = require('http'); const querystring = require('querystring'); const postData = querystring.stringify({ 'topic': 'https://example.com/books/1', 'data': JSON.stringify({ '@id': 'https://example.com/books/1', status: 'OufOfStock'}), }); const req = http.request({ hostname: 'demo.mercure.rocks', port: '80', path: '/.well-known/mercure', method: 'POST', headers: { Authorization: 'Bearer snip', 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': Buffer.byteLength(postData), } }); req.write(postData);req.end(); Publishing: Node.js
24. @dunglas - mercure.rocks Publishing: Symfony (PHP)
25. @dunglas - mercure.rocks Subscribing
26. @dunglas - mercure.rocks Subscribing: Basic Usage const es = new EventSource( 'https://demo.mercure.rocks/.well-known/mercure?topic=https://example.com/books/1' ); // The callback will be called every time an update is published es.onmessage = function (e) { console.log(JSON.parse(e.data)); }
27. @dunglas - mercure.rocks Subscribing: Several Topics const url = new URL('https://demo.mercure.rocks/.well-known/mercure'); url.searchParams.append('topic', 'https://example.com/books/1') url.searchParams.append('topic', 'https://example.com/reviews/21') const es = new EventSource(url); es.onmessage = function (e) { console.log(JSON.parse(e.data)); }
28. @dunglas - mercure.rocks Subscribing: using patterns const url = new URL('https://demo.mercure.rocks/.well-known/mercure'); // URI Template (RFC 6560) url.searchParams.append('topic', 'https://example.com/books/{id}') // Both https://example.com/books/1 and https://example.com/books/33 will match const es = new EventSource(url); es.onmessage = function (e) { console.log(JSON.parse(e.data)); }
30. @dunglas - mercure.rocks The Discovery Mechanism
31. @dunglas - mercure.rocks Discovery Mechanism Web Linking: RFC 5988
32. @dunglas - mercure.rocks Authorization
33. @dunglas - mercure.rocks Authorization ❏ Uses JSON Web Token (JWT - RFC 7519) ❏ An update can be intended for one or several targets ❏ Publisher: must be authenticated ❏ Subscriber: ❏ Can be anonymous (if allowed by the config) ❏ Must be authenticated to receive private updates ❏ Two transports: cookie and Authorization header
34. @dunglas - mercure.rocks Authorization
35. @dunglas - mercure.rocks The Mercure Hub
36. @dunglas - mercure.rocks Reference Implementation ❏ Implements 100% of the Mercure protocol ❏ Fast, written in Go ❏ Works everywhere: static binaries and Docker ❏ Automatic HTTP/2 and HTTPS (Let’s Encrypt) ❏ CORS support, CSRF protection ❏ Cloud Native (12Factor App) ❏ Open source (AGPL) ❏ Optional: a server can implement directly the protocol
37. @dunglas - mercure.rocks Reference Implementation https://mercure.rocks
38. @dunglas - mercure.rocks Starting the Hub https://mercure.rocks
39. @dunglas - mercure.rocks On a Kubernetes Cluster
40. @dunglas - mercure.rocks Alternative Implementations ❏ No hubs: the hub is optional ❏ Go library (AGPL) ❏ Node.js library (GPL) ❏ HA version (managed and on premise)
41. @dunglas - mercure.rocks iGraal / Glory4Gamers Benchmarks ❏ Open Source version (EC2 t3.micro) ❏ 40k concurrent connections ❏ HA version (on premise) ❏ 200k concurrent connections
43. @dunglas - mercure.rocks Integrations
44. @dunglas - mercure.rocks Integrations ❏ Official integrations: ❏ Symfony, API Platform ❏ Libs: ❏ Go, JS, Python, Java, Dart, PHP, Laravel
45. @dunglas - mercure.rocks Example using
 API Platform & React
46. @dunglas - mercure.rocks Create a Mercure-enabled Web API api-platform.com
47. You get a fully-featured API supporting: ❏ JSON-LD + Hydra + schema.org ❏ GraphQL, JSON:API, HAL ❏ OpenAPI ❏ auth, pagination, filters, validation, HTTP/2 push, test framework… ❏ and Mercure support! MIT license
48. @dunglas - mercure.rocks Scaffold Mercure-Enabled Clients api-platform.com
49. @dunglas - mercure.rocks Try it! bit.ly/2H17hN3
50. @dunglas - mercure.rocks Unleashing the Power of AsyncAPI servers: production: url: https://demo.mercure.rocks/.well-known/mercure protocol: mercure channels: 'https://example.com/books/{id}': parameters: id: schema: type: integer subscribe: message: $ref: '#/components/messages/book' publish: message: $ref: '#/components/messages/book'
51. @dunglas les-tilleuls.coop .rocks