Building Event-Driven Services

40 min talk covering how to build event streaming applications / microservices.

Transcript

1. 1 Building Event Driven Services with Apache Kafka, Kafka Streams & KSQL Ben Stopford @benstopford
2. 2 There is a book! http://bit.ly/designing-event-driven-systems
3. 3 Event Driven Architectures Business Events Event Sourcing DDD Stream Processing
4. 4 Today’s ecosystems get pretty big 4 • 2.2 trillion messages per day (6 Petabytes) • Up to 400 Microservices pre cluster. • 20-200 Brokers per cluster
5. 5 Today’s ecosystems get pretty big 5 • 1 billion messages per day • 20,000 messages per second • 100 teams
6. 6 Stream Processing Event Driven Architectures Business Events Event Sourcing DDD
7. 7 KAFKA Serving Layer (Cassandra etc.) Kafka Streams / KSQL Streaming Platforms Data is embedded in each engine High Throughput Messaging Clustered Java App
8. 8 apps_opened app_crashes unstable_apps crashed_per_day opened_per_day (b) Crashes, Per App, Per day (a) Apps Opened, Per App, Per day (c) Unstable Applications Streaming Pipeline
9. 9 KAFKA Serving Layer (Cassandra etc.) Kafka Streams / KSQL Streaming Platforms Data is embedded in each engine High Throughput Messaging Clustered Java App
10. 10 An event log is a simple idea Messages are added at the end of the log Old New
11. 11 Readers have a position all of their own Sally is here George is here Fred is here Old New Scan Scan Scan
12. 12 You can rewind and replay, just like Tivo! Old New Sally is here Scan
13. 13 The hard part: Tying it all together!
14. 14 Many ”logs” over many machines Producing Services Kafka Consuming Services
15. 15 Resistant to Failure Producing Services Kafka Consuming Services
16. 16 KAFKA Serving Layer (Cassandra etc.) Kafka Streams / KSQL Streaming Platforms Data is embedded in each engine High Throughput Messaging Clustered Java App
17. 17 Streaming Example apps_opened opened_per_day
18. 18 CREATE TABLE opened_per_day AS SELECT app_id, count(*) FROM apps_opened WINDOW TUMBLING (SIZE 1 DAY) GROUP BY app_id; apps_opened opened_per_day
19. 19 CREATE TABLE opened_per_day AS SELECT app_id, count(*) FROM apps_opened WINDOW TUMBLING (SIZE 1 DAY) GROUP BY app_id; apps_opened opened_per_day
20. 20 CREATE TABLE opened_per_day AS SELECT app_id, count(*) FROM apps_opened WINDOW TUMBLING (SIZE 1 DAY) GROUP BY app_id; apps_opened opened_per_day
21. 21 CREATE TABLE opened_per_day AS SELECT app_id, count(*) FROM apps_opened WINDOW TUMBLING (SIZE 1 DAY) GROUP BY app_id; apps_opened opened_per_day
22. 22 CREATE TABLE opened_per_day AS SELECT app_id, count(*) FROM apps_opened WINDOW TUMBLING (SIZE 1 DAY) GROUP BY app_id; apps_opened opened_per_day
23. 23 CREATE TABLE opened_per_day AS SELECT app_id, count(*) FROM apps_opened WINDOW TUMBLING (SIZE 1 DAY) GROUP BY app_id; apps_opened opened_per_day
24. 24 Streaming is manipulating events in flight, at scale.
25. 25 Stream Processing Event Driven Architectures Business Events Event Sourcing DDD
26. 26 Ecosystems App Increasingly we build ecosystems
27. 27 SOA / Microservices / EDA Customer Service Shipping Service
28. 28 The Problem is DATA
29. 29 Most services share the same core facts. Catalog Most services live in here
30. 30 Events have two hats Notification Data replication
31. 31 Buying an iPad (with REST/RPC) Submit Order shipOrder() getCustomer() Orders Service Shipping Service Customer Service Webserver
32. 32 Events for Notification Only Message Broker (Kafka) Submit Order Order Created getCustomer() REST Notification Orders Service Shipping Service Customer Service Webserver KAFKA
33. 33 Pluggability Submit Order Order Created Orders Service Shipping Service Customer Service Webserver KAFKA Repricing getCustomer() REST Notification
34. 34 Events for Data Locality Customer Updated Submit Order Order Created Data is replicated Orders Service Shipping Service Customer Service Webserver KAFKA
35. 35 Events have two hats Notification Data replication
36. 36 Stateless / Stateful Stream Processing Relates to these hats
37. 37 Message Broker (Kafka) Submit Order Order Created getCustomer() REST/RPC Orders Service Customer Service Webserver KAFKA KStreams Shipping Service Stateless Stream Processing Notification Kafka Steams / KSQL
38. 38 Message Broker (Kafka) Submit Order Order Created Orders Service Customer Service Webserver KAFKA KStreams Shipping Service KTable Customer Updated Stateful Stream Processing Data replication
39. 39 Message Broker (Kafka) Orders Stream (Buffer) KAFKA KStreams Shipping Service Customers (Buffer All) Streams & Tables Join
40. 40 KSQL ~ KStreams
41. 41 Streaming is about 1. Joining & Operating on Streams 2. Joining & Operating on Materialized Tables On Notification Data Replication
42. 42 Kafka: a Streaming Platform The Log Connectors Connectors Producer Consumer Streaming Engine
43. 43 8 Steps to Streaming Services
44. 44 1. Use events to decouple and collaborate
45. 45 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Event Collaboration Order Completed CQRS
46. 46 2. Use Connect (& CDC) to evolve away from legacy
47. 47 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Make Legacy Datasets Available via the Log Order Completed CQRS Connect Products
48. 48 3. Use the Single Writer Principal
49. 49 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service State changes to a topic owned by one service Connect Products Schema Registry Order Completed
50. 50 Orders Service Email Service T1 T2 T3 T4 REST Service T5 Local consistency points in the absence of Global Consistency
51. 51 4. Use Kafka as a Shared Source of Truth (Messaging that Remembers)
52. 52 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Shared Source of Truth Connect Products Schema Registry Order Completed Reporting
53. 53 KAFKA Order Requested Order Validated Order Received Browser Webserver Orders Service Product Catalogue stored in 3 places Connect Products Schema Registry Order Completed Reporting Reporting view may be “thinner”
54. 54 5. Move Data to Code
56. 56 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Stock Stock Materialize Stock ‘View’ Inside Service KAFKA
57. 57 Kafka has several features for reducing the need to move data on startup - Standby Replicas - Disk Checkpoints - Compacted topics
58. 58 6. Use State Stores as local ‘database’, backed up in Kafka
59. 59 Connect Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks State stores behave like local databases KAFKA State Store
60. 60 7. Use Transactions to tie All Interactions Together
61. 61 Connect TRANSACTION Order Requested Order Validated Order Completed Order Received Products Browser Webserver Schema Registry Orders Service Reserved Stocks Stock Stock Reserved Stocks Transactions KAFKA
62. 62 8. Evolve and Grow
63. 63 Kafka KAFKA Finance Front Office Operations Tiered Contexts
64. 64 Span regions or clouds
65. 65 Handle Disconnectedness
66. 66 So…
67. 67 Stream Processing Optimize for complexity vs optimize for scale Event Driven Architectures
68. 68 Events provide the key to evolutionary architectures Notification Data replication
69. 69 Spectrum of use cases Finer Grained, Collaborative, Connected Courser Grained, Non-collaborative, Disconnected Notification Data Replication
70. 70 Streaming is the toolset for dealing with events at scale
71. 71 • Broadcast events • Retain them in the log • Evolve the event-stream with streaming functions • Recasting the event stream into views when you need to query. Event Driven Services
72. 72 Find out more Book: http://bit.ly/designing-event-driven-systems Software: https://confluent.io/download/ Code: http://bit.ly/kafka-microservice-examples Twitter: @benstopford