Practical Event Driven Architecture

The trade-offs of using events to shape a fan-out architecture and spread data to multiple cache systems, isolating traffic-serving from data-processing pipelines.

This talk was idealized while working at heycar (https://hey.car) but presented while working at Airbnb (https://airbnb.com), that's why the slides carry the icon and information from the latter.

Transcript

1. HOSTED BY MARCELO BOEIRA · BERLIN, GERMANY Practical Event Driven Architecture
2. context San Francisco, California
4. Context car dealerships
5. Context car dealerships inventory
6. Context website car dealerships inventory
7. Context website car dealerships customers inventory
8. Context website car dealerships customers inventory
9. Context website car dealerships customers inventory lead
10. Context website car dealerships customers inventory lead
11. Context website car dealerships customers inventory lead time-to-market performance
12. traditional Santorini, Greece
13. Traditional Approach MONOLITH car dealerships
14. Traditional Approach MONOLITH car dealerships inventory
15. Traditional Approach MONOLITH application car dealerships inventory
16. Traditional Approach MONOLITH database application car dealerships inventory
17. Traditional Approach MONOLITH database application car dealerships customers inventory
18. Traditional Approach MONOLITH database application car dealerships customers inventory zip (csv + images) CMS
19. Traditional Approach MONOLITH database application car dealerships customers inventory zip (csv + images) time-to-market performance CMS
20. Traditional Approach MONOLITH application inventory customers
21. Traditional Approach MONOLITH application inventory customers
22. Traditional Approach MONOLITH database application inventory customers
23. Traditional Approach MONOLITH database application inventory customers
24. Traditional Approach MONOLITH application inventory customers database
25. Traditional Approach MONOLITH database application inventory customers
26. Traditional Approach MONOLITH database application inventory customers
27. Traditional Approach MONOLITH database application inventory customers
28. Traditional Approach MONOLITH database application inventory customers Traffic Serving Logic
29. Traditional Approach MONOLITH database application inventory customers Business Logic Traffic Serving Logic
30. Traditional Approach MONOLITH database application inventory customers Business Logic Hard to Scale Traffic Serving Logic
31. Traditional Approach MONOLITH database application inventory customers Business Logic Hard to Scale Hard to Orchestrate Traffic Serving Logic
32. Traditional Approach MONOLITH database application inventory customers Business Logic Hard to Scale Hard to Orchestrate Traffic Serving Logic Single point of failure
33. Traditional Approach ingestion traffic serving TRAFFIC FLOW
34. Traditional Approach monolith ingestion traffic serving TRAFFIC FLOW
35. Traditional Approach monolith ingestion traffic serving write TRAFFIC FLOW
36. Traditional Approach database monolith ingestion traffic serving write TRAFFIC FLOW
37. Traditional Approach database monolith ingestion traffic serving write read TRAFFIC FLOW
38. Traditional Approach database monolith ingestion traffic serving write read TRAFFIC FLOW UNRELIABLE
39. ? ingestion traffic serving ? TRAFFIC FLOW Experimental Approach
40. database ? ingestion traffic serving write ? TRAFFIC FLOW Experimental Approach
41. database ? ingestion traffic serving write ? TRAFFIC FLOW consistency Experimental Approach
42. database ? ingestion traffic serving write ? TRAFFIC FLOW durability consistency Experimental Approach
43. database ? ingestion traffic serving write read ? ? TRAFFIC FLOW durability consistency Experimental Approach
44. database ? ingestion traffic serving write read ? ? TRAFFIC FLOW performance durability consistency Experimental Approach
45. database ? ingestion traffic serving write read ? ? TRAFFIC FLOW reliability performance durability consistency Experimental Approach
46. database ? ingestion traffic serving write read ? ? TRAFFIC FLOW isolation reliability performance durability consistency Experimental Approach isolation reliability
47. database ? ingestion traffic serving write read ? ? TRAFFIC FLOW replication isolation reliability performance durability consistency Experimental Approach replication isolation reliability
48. database ? ingestion traffic serving write read cache ? TRAFFIC FLOW replication isolation reliability performance durability consistency Experimental Approach replication isolation reliability
49. database ? ingestion traffic serving write read cache ? TRAFFIC FLOW replication isolation reliability performance durability consistency Experimental Approach replication isolation reliability ?
50. database ? ingestion traffic serving write read cache ? TRAFFIC FLOW replication isolation reliability performance durability consistency Experimental Approach replication isolation reliability Events
51. database ? ingestion traffic serving write read cache ? TRAFFIC FLOW replication isolation reliability performance durability consistency Experimental Approach replication isolation reliability Fan-out
52. events Tokyo, Japan
53. Experimental Approach DISTRIBUTED car dealerships inventory
54. Experimental Approach DISTRIBUTED car dealerships inventory ingestion-batch
55. Experimental Approach DISTRIBUTED ingestion-service car dealerships inventory ingestion-batch
56. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip ingestion-batch
57. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip WHY S3? ingestion-batch
58. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip WHY S3? ingestion-batch
59. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip sqs WHY S3? ingestion-batch Queue vs Stream
60. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip sqs WHY S3? ingestion-batch extraction-service Queue vs Stream
61. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip sqs WHY S3? ingestion-batch extraction-service Queue vs Stream Replay
62. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip sqs WHY S3? ingestion-batch extraction-service Queue vs Stream Replay FIFO
63. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip sqs WHY S3? ingestion-batch extraction-service Queue vs Stream Replay At Least Once Delivery FIFO
64. Experimental Approach DISTRIBUTED s3 bucket ingestion-service car dealerships inventory 2019/12/14/dealership-a/batch-000001.zip sqs WHY S3? ingestion-batch extraction-service Queue vs Stream Replay Retry At Least Once Delivery FIFO
65. Example Queue Consumer CODE SNIPPET loop do event = sqs.pop result = process(event) if result.success? event.ack else event.return end end Business Logic Remove from the queue Return to the queue { Add metrics for monitoring
66. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service
67. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service single consumer
68. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service single consumer CSV
69. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db single consumer CSV
70. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db single consumer CSV ?
71. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db single consumer CSV ? consumer-a consumer-b consumer-c
72. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db single consumer CSV ? consumer-a consumer-b consumer-c ordering multiple-consumers
73. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db single consumer CSV consumer-a consumer-b consumer-c ordering multiple-consumers stream
74. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db single consumer CSV stream Dual write problem Consistency
75. listing-service listing-db HTTP POST Experimental Approach SERVICE INTERNALS
76. listing-service listing-db HTTP POST Write Ahead Log Experimental Approach SERVICE INTERNALS
77. listing-service listing-db changes-service HTTP POST Slave Replica Write Ahead Log Experimental Approach SERVICE INTERNALS
78. listing-service listing-db changes-service HTTP POST Slave Replica Write Ahead Log Experimental Approach SERVICE INTERNALS
79. listing-service listing-db changes-service HTTP POST Slave Replica Write Ahead Log listing-stream Experimental Approach SERVICE INTERNALS
80. listing-service listing-db changes-service HTTP POST Slave Replica Write Ahead Log listing-stream Experimental Approach SERVICE INTERNALS Service Unit Replication Slot CDC - change-data-capture < Unaware
81. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db single consumer CSV
82. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db changes-service listing-stream single consumer JSON payloads* CSV
83. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service listing-db changes-service listing-stream single consumer JSON payloads* CSV
84. Experimental Approach DISTRIBUTED s3 bucket sqs extraction-service listing-service image-batch-service listing-db image-db changes-service changes-service listing-stream image-stream single consumer JSON payloads* s3-images CSV Pictures
85. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service Experimental Approach
86. Example Stream Consumer CODE SNIPPET loop do record = kinesis.scan_after(position || horizon) result = process(record) if result.success? position = record.position end end Business Logic Update Cursor { Add metrics for monitoring
87. Experimental Approach DISTRIBUTED listing-stream image-stream cpu-intensive-service Experimental Approach
88. Experimental Approach DISTRIBUTED listing-stream image-stream cpu-intensive-service Experimental Approach
89. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service Experimental Approach
90. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service Experimental Approach
91. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service financing-service Experimental Approach
92. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service financing-service image-classification-service Experimental Approach
93. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service financing-service image-classification-service Experimental Approach dealership-db vehicle-db pricing-db financing-db image-classification-db
94. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service financing-service image-classification-service Experimental Approach dealership-db vehicle-db pricing-db financing-db image-classification-db customers front-end ?
95. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service financing-service image-classification-service Experimental Approach dealership-db vehicle-db pricing-db financing-db image-classification-db dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream
96. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service financing-service image-classification-service Experimental Approach dealership-db vehicle-db pricing-db financing-db image-classification-db dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream }
97. Experimental Approach DISTRIBUTED listing-stream image-stream dealership-service vehicle-service pricing-service financing-service image-classification-service Experimental Approach dealership-db vehicle-db pricing-db financing-db image-classification-db aggregation-service dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream }
98. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream } listing-stream
99. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream } id listing dealership financing … a23-… JSON JSON JSON d08-… JSON JSON JSON … listing-stream
100. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream } id listing dealership financing … a23-… JSON JSON JSON d08-… JSON JSON JSON … listing-stream
101. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream } id listing dealership financing … a23-… JSON JSON JSON d08-… JSON JSON JSON … listing-stream
102. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service dealership-stream vehicle-stream pricing-stream financing-stream image-classification-stream } id listing dealership financing … a23-… JSON JSON JSON d08-… JSON JSON JSON … listing-stream Aggregate / Hydrated Document
103. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service aggregation-stream
104. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service aggregation-stream elastic-search
105. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service aggregation-stream elastic-search front-end
106. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service aggregation-stream elastic-search {
107. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service aggregation-stream data-warehouse elastic-search {
108. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service aggregation-stream data-warehouse analytics / dashboards elastic-search {
109. Experimental Approach DISTRIBUTED Experimental Approach aggregation-service aggregation-stream data-warehouse analytics / dashboards marketing feeds elastic-search {
110. overview Tokyo, Japan Cairo, Egypt
111. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service
112. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service Isolation
113. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service Isolation
114. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service Isolation Scalable Traffic Serving
115. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service
116. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service replayable events
117. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service latency
118. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service scalable components
119. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service scalable components infra > code
120. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service bottleneck
121. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service bottleneck sharding
122. Experimental Approach FINAL OVERVIEW Experimental Approach customers inventory { } queue stream to stream workers core stream Ingestion Fan-out Traffic Serving aggregate-stream cache front-end ingestion-entrypoint listing-service aggregation-service
123. • CDC - Change Data Capture - Data Replication (e.g.: Write Ahead Log) • Queues vs Streams - Guarantees (At least once, at most once,…) - Single vs Multiple Consumers • Sharding (e.g.: parallelized processing) • Eventual Consistency (e.g.: a trade-off on performance to scale distributed systems) • Latency (e.g.: how long it takes for processing and fan-out of your data) • Stream Editing (Benthos) • Binary Serialization / Schemas - e.g.: Thrift, Protobufs… Take Home Topics INTERESTING SUBJECTS TO STUDY
124. thank you Tokyo, Japan Cappadocia, Turkey
125. Practical Event Driven Architecture Marcelo Boeira - 2019