In progress v1

Transcript

1. Microservices in a Streaming World
3. Centralisation
4. Introversion
5. decentralisation Warehouse Analytics
6. Systems are better Connected
7. Which is good! duplication inefficiency error
8. Embrace the wider context
9. Many Architectural Styles
10. Shared State polling a central oracle
11. Shared State with Transport Services talk through broker, but share a database
12. Mediator / Workflow single workflow walks a variety of services/workers through a set of steps
13. Event Driven Architecture Services listen for interesting events, do something useful, announce their results, the process continues
14. SOA / Microservices Message Broker
15. Pros and Cons
16. combinations
17. combinations Withdraw £100 Account Service General Ledger Customer Statements Fraud Detection Check Funds Async Message Broker
18. Services generally eschew shared state
19. Distributed Programming is Hard
20. ?? V.S.
21. How do we put these things together?
22. Synchronous Communication
23. Request/Response Request Response
24. Request/Response + Registry Registry Request Response
25. Request Response Withdraw £100 Account Service Check Funds
26. Asynchronous Communication
27. Queues
28. Tools: Queue Producer Consumer
29. Async / Decoupled
30. Point to Point Service A Service B
31. Scaling Consumption Instance 2 Instance 1 Single message allocation has scalability issues
32. Batched Allocation Instance 1 Instance 2
33. Lose Ordering Guarantees Fail! Instance 1 Instance 2
34. Topics
35. Topics Retain Ordering Trades Buys Sells Broker Instance 1 Instance 2
36. Even when services fail Trades Buys Sells Fail! Broker We retain ordering, but we have to detect & reprovision Instance 1 Instance 2
37. Filters don’t scale well Lots of consumers => Lots of filters Slow consumers Broker => Queues are Databases* *Queues Are Databases - Jim Gray 1995 => Lots of indexes => Random disk access
38. Three Implications
39. Multi-Consumer Queues Lose Ordering Guarantees Fail! Worker 1 Worker 2
40. Trades Buys Sells Topics don’t degrade well Broker
41. Trades Buys Sells Messages are Transient Broker
42. Is there another way?
43. Think back to the queue example Batch Batch
44. Alternative: Shard on the way in
45. Each shard is a queue Strong Ordering (in shard). Good concurrency.
46. Each consuming service is assigned a “personal set” of queues each little queue is sent to only one consumer in a group
47. This means services can naturally rebalance Service instance dies Data is redirected
48. Sharded In, Sharded Out
49. Reduces to a globally ordered queue
50. This is a Distributed Log Kafka is one example
51. some other nice properties
52. Fault Tolerance
53. The Log Single seek & scan Append only messages don’t need to be transient!
54. Linearly Scalable
55. • Strong partition-based ordering • Scalable data retention • Scalable multiprocessing • Always on • Dynamic Consumer Rebalancing
56. So how is this useful to microservices?
57. Build ‘Always On’ Services Rely on Fault Tolerant Broker
58. Load Balance Services Load Balance Services (with strong ordering)
59. Fault Tolerant Services Services automatically fail over
60. Services can retain data in the log Rewind & Replay
61. Retain data in “table-like” compacted streams Older K1 K1 K1 K2 K2 K2 K1 V1 V1 V2 V3 V2 V4 V3 K1 V4 K2 V3 Log file Exchange Rates Exchange Rates Keep all versions of key Only keep latest key Just a simple journal Compacted Stream
62. What does that mean exactly? In Kafka it’s termed Log Compaction
63. Exchange Rate Service Exchange Rate Service USD/GBP = 0.71 EUR/GBP = 0.77 USD/INR = 67.7 USD/AUD = 1.38 EUR/JPY = 114.41 …
64. Option1: Request Response rate for USD/GBP? 0.71 Exchange Rate Service
65. Option 2: Publish Subscribe Exchange Rate Service Accumulate current state ETL
66. Option 3: Accumulate in Compacted Stream Exchange Rate Service Publish all rate events Broker keeps latest version of exchange rates in a compacted stream Get all exchange rates New exchange rates
67. Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered
68. … adding in stream processing
69. Max(price) From orders where ccy=‘GBP’ over 1 day window emitting every second What is stream processing? continuous queries.
70. What is stream processing engine? Data Index Query Engine Query Engine vs Database Finite, well defined source Stream Processor Infinite, poorly defined source
71. Windowing For unordered or unpredictable streams Sliding Fixed
72. Features: similar to database query engine Join Filter Aggr- egate View Window
73. Tables & Streams stream Compacted stream Join Streaming Data Stored Data KStream KTable
74. A little example…
75. Buying Lunch Abroad Payments Service Exchange Rates Service Buy Notification Service Amount in ££ $$ $$ Text Message: ££ $$
76. ETL Option Payments Service Exchange Rates Service Buy Amount in ££ ETL ETL Join etc
77. Request-Response Option Payments Service Exchange Rates Service Buy Amount in ££ Join etc
78. Stream Processor Option Payments Service Exchange Rates Service Buy Stream Processor join etc
79. Buying Lunch Abroad Payments Exchange Rates • filter(ccy<>’GBP’) • join on ccy • Calculate payment in GBP • Send text message
80. Streams & Compacted Streams Payments Exchange Rates Looks like a table (compacted stream) Looks like an infinite stream
81. In Process DB Topic Compacted Topic
82. Backed by the broker Topic Compacted Topic
83. Scales Out (MPP)
84. These tools are pretty handy for managing disconnected services
85. Joining Services Payments Exchange Rates Join Streams Compacted Streams
86. Talk your own data model Data Stream View Query
87. Express Lateness 9am 5pm Late trades
88. Keep Services Consistent
89. Big Global Bag of State in the Sky Problem: No BGBSS
90. How to you provide the accuracy of this
91. In this?
92. The problem is failure
93. Duplicate messages are inevitable
94. Idempotence try 1 try 2 try 3 try 4
95. Stream processors have to solve this problem Risk of duplication at all points in the chain
96. Idempotent Topics Idempotent Topic write many times (against key) emit only once one part of a wider, and ongoing, ‘Exactly Once’ effort
97. Service-based approaches need this too Centralised consistency model Distributed consistency model
98. Simple Approaches Just a library (over Kafka)
99. There is much more to stream processing it is grounded in the world of big-data analytics
100. So what do we have?
101. Microservices push us away from global state
102. Big Global Bag of State in the Sky Away from BGBSS’s
103. This means data is increasingly remote
104. Sure, you can collect it all copy copy copy copy copy copy copy ETL ETL ETL ETL ETL ETL
105. But do you really want to?
106. Better to embrace decentralistion
107. We need a decentralised toolset to do this
108. Queued Delivery System Ordered queue
109. Scales Horizontally
110. Scales Horizontally
111. Scales Horizontally
112. Scales Horizontally
113. Built In Fault Tolerance
114. Runs Always On
115. For Services Too
116. Scales Horizontally load balance services
117. rebalancing on failure
118. Scales Horizontally with history stored in the Log
119. Scales Horizontally Extending to any number of services
120. Scales Horizontally With any data throughput
121. Scales Horizontally With any data throughput
122. Scales Horizontally With any data throughput
123. Scales Horizontally And tools for dealing with streams
124. Scales Horizontally the declarative processing of data
125. Scales Horizontally with in process storage
126. Scales Horizontally As resilient as the system that backs it
127. Scaling services out horizontally
128. Scales Horizontally to easily combining streams from real time services
129. or join with compacted streams
130. Scales Horizontally with strong ordering and repeatability guarantees
131. Embrace Decentralisation Keep it simple, Keep it moving