In progress v1 - Speaker Deck

Slide 1

Slide 1 text

Microservices in a Streaming World

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Centralisation

Slide 4

Slide 4 text

Introversion

Slide 5

Slide 5 text

decentralisation Warehouse Analytics

Slide 6

Slide 6 text

Systems are better Connected

Slide 7

Slide 7 text

Which is good! duplication inefﬁciency error

Slide 8

Slide 8 text

Embrace the wider context

Slide 9

Slide 9 text

Many Architectural Styles

Slide 10

Slide 10 text

Shared State polling a central oracle

Slide 11

Slide 11 text

Shared State with Transport Services talk through broker, but share a database

Slide 12

Slide 12 text

Mediator / Workﬂow single workﬂow walks a variety of services/workers through a set of steps

Slide 13

Slide 13 text

Event Driven Architecture Services listen for interesting events, do something useful, announce their results, the process continues

Slide 14

Slide 14 text

SOA / Microservices Message Broker

Slide 15

Slide 15 text

Pros and Cons

Slide 16

Slide 16 text

combinations

Slide 17

Slide 17 text

combinations Withdraw £100 Account Service General Ledger Customer Statements Fraud Detection Check Funds Async Message Broker

Slide 18

Slide 18 text

Services generally eschew shared state

Slide 19

Slide 19 text

Distributed Programming is Hard

Slide 20

Slide 20 text

?? V.S.

Slide 21

Slide 21 text

How do we put these things together?

Slide 22

Slide 22 text

Synchronous Communication

Slide 23

Slide 23 text

Request/Response Request Response

Slide 24

Slide 24 text

Request/Response + Registry Registry Request Response

Slide 25

Slide 25 text

Request Response Withdraw £100 Account Service Check Funds

Slide 26

Slide 26 text

Asynchronous Communication

Slide 27

Slide 27 text

Queues

Slide 28

Slide 28 text

Tools: Queue Producer Consumer

Slide 29

Slide 29 text

Async / Decoupled

Slide 30

Slide 30 text

Point to Point Service A Service B

Slide 31

Slide 31 text

Scaling Consumption Instance 2 Instance 1 Single message allocation has scalability issues

Slide 32

Slide 32 text

Batched Allocation Instance 1 Instance 2

Slide 33

Slide 33 text

Lose Ordering Guarantees Fail! Instance 1 Instance 2

Slide 34

Slide 34 text

Topics

Slide 35

Slide 35 text

Topics Retain Ordering Trades Buys Sells Broker Instance 1 Instance 2

Slide 36

Slide 36 text

Even when services fail Trades Buys Sells Fail! Broker We retain ordering, but we have to detect & reprovision Instance 1 Instance 2

Slide 37

Slide 37 text

Filters don’t scale well Lots of consumers => Lots of ﬁlters Slow consumers Broker => Queues are Databases* *Queues Are Databases - Jim Gray 1995 => Lots of indexes => Random disk access

Slide 38

Slide 38 text

Three Implications

Slide 39

Slide 39 text

Multi-Consumer Queues Lose Ordering Guarantees Fail! Worker 1 Worker 2

Slide 40

Slide 40 text

Trades Buys Sells Topics don’t degrade well Broker

Slide 41

Slide 41 text

Trades Buys Sells Messages are Transient Broker

Slide 42

Slide 42 text

Is there another way?

Slide 43

Slide 43 text

Think back to the queue example Batch Batch

Slide 44

Slide 44 text

Alternative: Shard on the way in

Slide 45

Slide 45 text

Each shard is a queue Strong Ordering (in shard). Good concurrency.

Slide 46

Slide 46 text

Each consuming service is assigned a “personal set” of queues each little queue is sent to only one consumer in a group

Slide 47

Slide 47 text

This means services can naturally rebalance Service instance dies Data is redirected

Slide 48

Slide 48 text

Sharded In, Sharded Out

Slide 49

Slide 49 text

Reduces to a globally ordered queue

Slide 50

Slide 50 text

This is a Distributed Log Kafka is one example

Slide 51

Slide 51 text

some other nice properties

Slide 52

Slide 52 text

Fault Tolerance

Slide 53

Slide 53 text

The Log Single seek & scan Append only messages don’t need to be transient!

Slide 54

Slide 54 text

Linearly Scalable

Slide 55

Slide 55 text

• Strong partition-based ordering • Scalable data retention • Scalable multiprocessing • Always on • Dynamic Consumer Rebalancing

Slide 56

Slide 56 text

So how is this useful to microservices?

Slide 57

Slide 57 text

Build ‘Always On’ Services Rely on Fault Tolerant Broker

Slide 58

Slide 58 text

Load Balance Services Load Balance Services (with strong ordering)

Slide 59

Slide 59 text

Fault Tolerant Services Services automatically fail over

Slide 60

Slide 60 text

Services can retain data in the log Rewind & Replay

Slide 61

Slide 61 text

Retain data in “table-like” compacted streams Older K1 K1 K1 K2 K2 K2 K1 V1 V1 V2 V3 V2 V4 V3 K1 V4 K2 V3 Log ﬁle Exchange Rates Exchange Rates Keep all versions of key Only keep latest key Just a simple journal Compacted Stream

Slide 62

Slide 62 text

What does that mean exactly? In Kafka it’s termed Log Compaction

Slide 63

Slide 63 text

Exchange Rate Service Exchange Rate Service USD/GBP = 0.71 EUR/GBP = 0.77 USD/INR = 67.7 USD/AUD = 1.38 EUR/JPY = 114.41 …

Slide 64

Slide 64 text

Option1: Request Response rate for USD/GBP? 0.71 Exchange Rate Service

Slide 65

Slide 65 text

Option 2: Publish Subscribe Exchange Rate Service Accumulate current state ETL

Slide 66

Slide 66 text

Option 3: Accumulate in Compacted Stream Exchange Rate Service Publish all rate events Broker keeps latest version of exchange rates in a compacted stream Get all exchange rates New exchange rates

Slide 67

Slide 67 text

Service Backbone Scalable, Fault Tolerant, Concurrent, Strongly Ordered

Slide 68

Slide 68 text

… adding in stream processing

Slide 69

Slide 69 text

Max(price) From orders where ccy=‘GBP’ over 1 day window emitting every second What is stream processing? continuous queries.

Slide 70

Slide 70 text

What is stream processing engine? Data Index Query Engine Query Engine vs Database Finite, well defined source Stream Processor Infinite, poorly defined source

Slide 71

Slide 71 text

Windowing For unordered or unpredictable streams Sliding Fixed

Slide 72

Slide 72 text

Features: similar to database query engine Join Filter Aggr- egate View Window

Slide 73

Slide 73 text

Tables & Streams stream Compacted stream Join Streaming Data Stored Data KStream KTable

Slide 74

Slide 74 text

A little example…

Slide 75

Slide 75 text

Buying Lunch Abroad Payments Service Exchange Rates Service Buy Notiﬁcation Service Amount in ££ $$ $$ Text Message: ££ $$

Slide 76

Slide 76 text

ETL Option Payments Service Exchange Rates Service Buy Amount in ££ ETL ETL Join etc

Slide 77

Slide 77 text

Request-Response Option Payments Service Exchange Rates Service Buy Amount in ££ Join etc

Slide 78

Slide 78 text

Stream Processor Option Payments Service Exchange Rates Service Buy Stream Processor join etc

Slide 79

Slide 79 text

Buying Lunch Abroad Payments Exchange Rates • ﬁlter(ccy<>’GBP’) • join on ccy • Calculate payment in GBP • Send text message

Slide 80

Slide 80 text

Streams & Compacted Streams Payments Exchange Rates Looks like a table (compacted stream) Looks like an inﬁnite stream

Slide 81

Slide 81 text

In Process DB Topic Compacted Topic

Slide 82

Slide 82 text

Backed by the broker Topic Compacted Topic

Slide 83

Slide 83 text

Scales Out (MPP)

Slide 84

Slide 84 text

These tools are pretty handy for managing disconnected services

Slide 85

Slide 85 text

Joining Services Payments Exchange Rates Join Streams Compacted Streams

Slide 86

Slide 86 text

Talk your own data model Data Stream View Query

Slide 87

Slide 87 text

Express Lateness 9am 5pm Late trades

Slide 88

Slide 88 text

Keep Services Consistent

Slide 89

Slide 89 text

Big Global Bag of State in the Sky Problem: No BGBSS

Slide 90

Slide 90 text

How to you provide the accuracy of this

Slide 91

Slide 91 text

In this?

Slide 92

Slide 92 text

The problem is failure

Slide 93

Slide 93 text

Duplicate messages are inevitable

Slide 94

Slide 94 text

Idempotence try 1 try 2 try 3 try 4

Slide 95

Slide 95 text

Stream processors have to solve this problem Risk of duplication at all points in the chain

Slide 96

Slide 96 text

Idempotent Topics Idempotent Topic write many times (against key) emit only once one part of a wider, and ongoing, ‘Exactly Once’ effort

Slide 97

Slide 97 text

Service-based approaches need this too Centralised consistency model Distributed consistency model

Slide 98

Slide 98 text

Simple Approaches Just a library (over Kafka)

Slide 99

Slide 99 text

There is much more to stream processing it is grounded in the world of big-data analytics

Slide 100

Slide 100 text

So what do we have?

Slide 101

Slide 101 text

Microservices push us away from global state

Slide 102

Slide 102 text

Big Global Bag of State in the Sky Away from BGBSS’s

Slide 103

Slide 103 text

This means data is increasingly remote

Slide 104

Slide 104 text

Sure, you can collect it all copy copy copy copy copy copy copy ETL ETL ETL ETL ETL ETL

Slide 105

Slide 105 text

But do you really want to?

Slide 106

Slide 106 text

Better to embrace decentralistion

Slide 107

Slide 107 text

We need a decentralised toolset to do this

Slide 108

Slide 108 text

Queued Delivery System Ordered queue

Slide 109

Slide 109 text

Scales Horizontally

Slide 110

Slide 110 text

Scales Horizontally

Slide 111

Slide 111 text

Scales Horizontally

Slide 112

Slide 112 text

Scales Horizontally

Slide 113

Slide 113 text

Built In Fault Tolerance

Slide 114

Slide 114 text

Runs Always On

Slide 115

Slide 115 text

For Services Too

Slide 116

Slide 116 text

Scales Horizontally load balance services

Slide 117

Slide 117 text

rebalancing on failure

Slide 118

Slide 118 text

Scales Horizontally with history stored in the Log

Slide 119

Slide 119 text

Scales Horizontally Extending to any number of services

Slide 120

Slide 120 text

Scales Horizontally With any data throughput

Slide 121

Slide 121 text

Scales Horizontally With any data throughput

Slide 122

Slide 122 text

Scales Horizontally With any data throughput

Slide 123

Slide 123 text

Scales Horizontally And tools for dealing with streams

Slide 124

Slide 124 text

Scales Horizontally the declarative processing of data

Slide 125

Slide 125 text

Scales Horizontally with in process storage

Slide 126

Slide 126 text

Scales Horizontally As resilient as the system that backs it

Slide 127

Slide 127 text

Scaling services out horizontally

Slide 128

Slide 128 text

Scales Horizontally to easily combining streams from real time services

Slide 129

Slide 129 text

or join with compacted streams

Slide 130

Slide 130 text

Scales Horizontally with strong ordering and repeatability guarantees

Slide 131

Slide 131 text

Embrace Decentralisation Keep it simple, Keep it moving