Event-driven API revolution is happening now

This talk was presented by Matthew O'Riordan at ASC API Specifications conference in Vancouver, Oct 2019.

In this talk I talk about what an event-driven API is, why it adds complexity compared to more traditional REST APIs, why in spite of this additional complexity we need to tackle event-driven APIs due to demand from developers. Then I move onto 3 common areas of unexpected complexity which is largely unspecified, yet needs to consider when publishing event-driven APIs, The goal of the talk was to inform users about complexity that exists, and how to navigate that complexity once you understand it.

Transcript

1. Event-driven API revolution is happening now Understanding the technical challenges and missing policies needed to successfully deliver streaming APIs
2. My journey with event-driven APIs Technical co-founder of Ably Built and scaled event-driven APIs for developers. Board member OpenAPI initiative
3. Page 3 Ably - Serious realtime infrastructure We all know what an event-driven API looks like... Delivery person Logistics provider Retailer Parcel arrived! (on device) Customer Ping
4. Page 4 Ably - Serious realtime infrastructure Pull (request/response) is stateless & simpler Data source Worker Server Consumers
5. Page 5 Ably - Serious realtime infrastructure Event-driven is more demanding on producers Worker hot spots Additional complexity: Backoffice scheduling & management Short value window Delivery logistics Acme Fresh Lunch boxes
6. Page 6 Ably - Serious realtime infrastructure Complexity inversion with event-driven APIs Pull (REST) API Event-driven API Consumer Producer Trigger requests and maintain state API Gateway / CDN Fan out push, maintain state, and fault tolerant. Complexity Wait for updates Stateless response
7. Page 7 Ably - Serious realtime infrastructure If complexity is evil - can we avoid it? Probably not. Event-driven data exceeds all data produced in 2019 (circa 40Zb)
8. Page 8 Ably - Serious realtime infrastructure Event-driven data demand is here, now By 2021, cumulative event-driven data exceeds all previous event-driven data produced and consumer, ever
9. Page 9 Ably - Serious realtime infrastructure Source of new event-driven data
10. Page 10 Ably - Serious realtime infrastructure Criticality of this data Compound Annual Growth Rate All data Potentially Critical Critical Hyper-Critical 30% 37% 39% 54%
11. Lifecycle of an event-driven API
12. Page 12 Ably - Serious realtime infrastructure Infrastructure Publishing Documentation / Spec Three distinct functions when publishing a realtime API Distribute Deploy Authentication Pull & push protocols Onboarding + dev tooling Scaling & performance Manage Access control Instrumentation Billing Monetization
13. Page 13 Ably - Serious realtime infrastructure Three distinct functions when publishing a realtime API Distribute Deploy Authentication Pull & push protocols Onboarding + dev tooling Scaling & performance Publishing Infrastructure Documentation / Spec Manage Access control Instrumentation Billing Monetization
14. Distribution layer complexity 1. Integrity vs Latency 2. Throughput 3. Push vs Pull subscriptions Three commonly unforeseen but important areas of complexity in event-driven API delivery.
15. Distribution layer complexity #1 Integrity vs Latency
16. Page 16 Ably - Serious realtime infrastructure Integrity vs Latency Happy Path Your servers message bus API Interface 15 updates a minute
17. Page 17 Ably - Serious realtime infrastructure Latency prioritised: Congestion & connectivity issues Backpressure Your servers message bus Less Happy Path
18. Page 18 Ably - Serious realtime infrastructure Spec solution #1 - Agree on backpressure control Time: 0s 10s 20s 30s 40s 50s 60s 70s 80s Backpressure control Bandwidth limited
19. Managing backpressure Server-side • TCP buffer size with high watermark Network layer Client-side • ACK from subscribing clients Application layer • Stream polling Common techniques used to manage backpressure.
20. Page 20 Ably - Serious realtime infrastructure Spec solution #2 - Conflation rules Pricing Scores GPS locations Time: 1s 2s 3s 4s Great for:
21. Page 21 Ably - Serious realtime infrastructure Simplified example of conflation on a stream
22. Page 22 Ably - Serious realtime infrastructure Integrity vs Latency: Integrity priority spec Ordered message stream Auditors & Legal message bus
23. Page 23 Ably - Serious realtime infrastructure Integrity vs Latency: Integrity priority spec Latency focussed Integrity focussed Latency & Integrity agnostic Ordered message stream Auditors & Legal message bus
24. Page 24 Ably - Serious realtime infrastructure Integrity vs Latency in summary • Consumers decide on integrity vs latency • Producers of data can choose both latency and integrity • Latency over integrity is generally simpler • Backpressure needs to be handled - always - and defined.
25. Distribution layer complexity #2 Throughput
26. Page 26 Ably - Serious realtime infrastructure Throughput - narrow pipe Multiple producers Fat message bus Single stream Consumer 500 msgs p/s Max 1,000 msgs p/s per stream
27. Page 27 Ably - Serious realtime infrastructure Throughput: #1 solved using queues Advantages: Relatively simple Scales easily Disadvantages: No ordering No data-consumer affinity At least once delivery Potentially two design patterns in play - pub/sub and queueing. Consumer workers Queue 500 msgs p/s Fat message bus Multiple producers 500 msgs p/s
28. Page 28 Ably - Serious realtime infrastructure Advantages: Data stream integrity (ordering) Single pattern pub/sub Exactly-once delivery Consumer-data affinity Disadvantages: Sharding is complex Expanding or contracting shards is hard Consumer state may be needed Throughput: #2 solved with sharding Shard consumers Consumer group 625 msg p/s 1 2 4 3 Throughput sharding Multiple producers Fat message bus
29. Distribution layer complexity #3 Push vs Pull Subscriptions
30. Page 30 Ably - Serious realtime infrastructure Push vs Pull subscriptions AMQP API Gateway Push service Client initiated Producer initiated Pull subscription protocols Push subscription protocols SSE Websockets MQTT HTTP Webhook WebSub Your message bus
31. Page 31 Ably - Serious realtime infrastructure Examples of Push subscription protocols Stateless over HTTP Queue based Stream based Webhooks AMQP Kafka WebSub AWS SQS AWS Kinesis Serverless function invocation MQTT Integrations (Zapier etc)
32. Page 32 Ably - Serious realtime infrastructure Push vs Pull specifications AMQP API Gateway Push service Client initiated Producer initiated Pull subscription protocols Push subscription protocols SSE Websockets MQTT HTTP Webhook WebSub Your message bus
33. Page 33 Ably - Serious realtime infrastructure Push vs Pull specifications AMQP API Gateway Push service Client initiated Producer initiated Pull subscription protocols Push subscription protocols SSE Websockets MQTT HTTP Webhook WebSub Your message bus AsyncAPI OpenAPI callbacks (sort of)
34. Page 34 Ably - Serious realtime infrastructure State of play for push & pull subscriptions Push subscriptions • High throughput Target can be load balanced. • Reduces consumer complexity Expose an endpoint and receive • Not supported in AsyncAPI / OpenAPI You’re on your own with this one. • Adds producer complexity Hard to manage downstream issues, rate limiting, instrumentation for consumers. Pull subscriptions • Good for on demand Well suited for devices such as mobiles & browsers with varying online state. • Simple Simple for consumers and producers. • Low throughput per subscriber Not suitable for high throughput, without sharding or queueing. • Supported mostly in AsyncAPI AsyncAPI is continuing to add support for more protocols.
35. Page 35 Ably - Serious realtime infrastructure What next for event-driven APIs? #1 Give developers what they want - better event-driven integrations Event driven integration frustration up 2% this year Source: State of API Integration Cloud Elements
36. Page 36 Ably - Serious realtime infrastructure What next for event-driven APIs? #2 Avoid complexity • Focus on necessary complexity • Delay complexity that is not needed to day, but plan for it • Open source and cloud solutions exist to address complexity
37. Page 37 Ably - Serious realtime infrastructure #3 Improve specifications and define policies • Don’t downstream complexity to consumers • Let consumers choose integrity or latency & push or pull subscriptions. • Agree policies around streams under duress • Upstream these policies as proposals to OpenAPI & AsyncAPI What next for event-driven APIs?
38. Page 38 Ably - Serious realtime infrastructure #4 Help build an open realtime connected world • Treat Event-driven APIs as first-class API citizens • Set your (open) data free What next?
39. Thank you Event-driven API revolution is happening now www.ably.io Me: @mattheworiordan @ablyrealtime Shameless plugs: go.ably.io/asc-odsp go.ably.io/asc-api-management