Asynchronous Service Oriented Design

Slide 1

Slide 1 text

Asynchronous Service-Oriented Design Patterns for Building and Operating Flexible Services

Slide 2

Slide 2 text

@jpignata

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

P ro b l e m Statement

Slide 6

Slide 6 text

“Our Ruby on Rails application is difﬁcult to change.”

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

monolith

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

monorail

Slide 12

Slide 12 text

“Amazon.com started [in 1996] as a monolithic application, running on a Web server, talking to a database on the back end. This application evolved to hold all the business logic, all the display logic, and all the functionality that Amazon eventually became famous for. This went on until 2001 when it became clear that the front-end application couldn’t scale anymore.” http://queue.acm.org/detail.cfm?id=1142065 A Conversation with Werner Vogels

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Asynchronous Service-Oriented Design

Slide 15

Slide 15 text

message passing

Slide 16

Slide 16 text

a medium is used to exchange messages

Slide 17

Slide 17 text

Patterns

Slide 18

Slide 18 text

Service Mailbox Pattern

Slide 19

Slide 19 text

clients write messages to a queue

Slide 20

Slide 20 text

a service processes messages from the queue

Slide 21

Slide 21 text

FIFO Queue

Slide 22

Slide 22 text

ﬁrst in, ﬁrst out

Slide 23

Slide 23 text

queue.enqueue(item)

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

queue.dequeue()

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

Application Service Mailbox Service dequeue enqueue

Slide 28

Slide 28 text

Feature: Message Delivery

Slide 29

Slide 29 text

Best Friends Group

Slide 30

Slide 30 text

"What is up?" "Steve: What is up?"

Slide 31

Slide 31 text

"Heading to drinks." "Tara: Heading to drinks."

Slide 32

Slide 32 text

iOS (APNS) Android (C2DM / GCM) Windows (WPNS) SMS (SIP / SMPP / HTTP)

Slide 33

Slide 33 text

messages should be delivered exactly once

Slide 34

Slide 34 text

messages should be delivered as soon as possible

Slide 35

Slide 35 text

drop messages after 2 hours if undeliverable

Slide 36

Slide 36 text

GroupMe Twilio

Slide 37

Slide 37 text

GroupMe Resque Twilio Redis

Slide 38

Slide 38 text

GroupMe Resque Twilio Redis APNS C2DM

Slide 39

Slide 39 text

throughput choppy and low

Slide 40

Slide 40 text

large number of processes

Slide 41

Slide 41 text

different rates of change

Slide 42

Slide 42 text

Transport Service

Slide 43

Slide 43 text

high-speed eventmachine resque worker

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

GroupMe Redis Transport APNS C2DM GCM SMS WPNS

Slide 46

Slide 46 text

uses a redis list as a queue

Slide 47

Slide 47 text

resque serialization format and key names

Slide 48

Slide 48 text

GroupMe Redis Transport RPUSH resque:queue:apns [serialized item] BLPOP resque:queue:apns

Slide 49

Slide 49 text

redis lists provide the exactly- once semantics required

Slide 50

Slide 50 text

higher, more even throughput

Slide 51

Slide 51 text

easy migration path

Slide 52

Slide 52 text

same redis dependency as resque

Slide 53

Slide 53 text

Event Stream Pattern

Slide 54

Slide 54 text

File

Slide 55

Slide 55 text

log.append(item)

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

log.read(offset, total) [

Slide 58

Slide 58 text

Service Service Service Event Stream Service Service Service Subscribe Publish

Slide 59

Slide 59 text

Feature: News Feed

Slide 60

Slide 60 text

Best Friends Group

Slide 61

Slide 61 text

Pat: Something pithy and funny about the current sporting event 2 Pat: Rails is Benihana.

Slide 62

Slide 62 text

Steve and Tara liked your message: Something pithy and funny about the current sporting event Steve and Tara liked your message: Rails is Benihana.

Slide 63

Slide 63 text

class NewsFeedController < ApplicationController def index render likes: Like.find_all_by_recipient_id(current_user_id), # other locals ... end end

Slide 64

Slide 64 text

slow database queries

Slide 65

Slide 65 text

low-value news feed reads competed for database time

Slide 66

Slide 66 text

unnecessarily real-time

Slide 67

Slide 67 text

News Feed Service

Slide 68

Slide 68 text

Kafka

Slide 69

Slide 69 text

GroupMe Message Service Kafka News Feed Service Relationship Service Cache

Slide 70

Slide 70 text

Apache Kafka is a distributed, high-throughput pub/sub messaging system

Slide 71

Slide 71 text

messages are appended to a topic by a producer

Slide 72

Slide 72 text

consumers iterate through the topic starting at a given byte offset

Slide 73

Slide 73 text

picture a distributed log ﬁle that you can write to and tail from other services across the network

Slide 74

Slide 74 text

components publish user events to the event stream

Slide 75

Slide 75 text

the news feed service creates content from these events where necessary

Slide 76

Slide 76 text

multi-consumer

Slide 77

Slide 77 text

rewindable

Slide 78

Slide 78 text

Beneﬁts

Slide 79

Slide 79 text

scalability

Slide 80

Slide 80 text

each process consumes a uniform number of resources

Slide 81

Slide 81 text

inﬁnite horizontal scale is usually a lie

Slide 82

Slide 82 text

services allow us to add targeted capacity

Slide 83

Slide 83 text

the services scale individually

Slide 84

Slide 84 text

agility

Slide 85

Slide 85 text

updates can deployed independently

Slide 86

Slide 86 text

deploys are low-impact because our messaging layer acts as a buffer

Slide 87

Slide 87 text

reusability

Slide 88

Slide 88 text

any component that needs to send messages can use the same interface

Slide 89

Slide 89 text

cheap experimentation

Slide 90

Slide 90 text

data ownership silos naturally form in larger organizations

Slide 91

Slide 91 text

conway’s law

Slide 92

Slide 92 text

“organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations”

Slide 93

Slide 93 text

publishing to event streams can mitigate data silos

Slide 94

Slide 94 text

all notable events within a system are published to an event stream

Slide 95

Slide 95 text

experimental projects can read from the event stream and use the data in unanticipated ways

Slide 96

Slide 96 text

GroupMe Message Service Kafka Experimental Likes Leaderboard Relationship Service Experimental Email Newsletter Experimental Photo Gallery

Slide 97

Slide 97 text

publishing to an event stream reduces the coupling between teams

Slide 98

Slide 98 text

data availability removes barriers to exploration

Slide 99

Slide 99 text

back pressure

Slide 100

Slide 100 text

some partners mandate our maximum delivery rate

Slide 101

Slide 101 text

message-based service boundaries are natural buffers

Slide 102

Slide 102 text

groupme / pace

Slide 103

Slide 103 text

worker = Pace::Worker.new(queue: "sms", jobs_per_second: 100) worker.pause worker.resume worker.pause(0.5)

Slide 104

Slide 104 text

automatic backoff on failure

Slide 105

Slide 105 text

we pause if we exceed a threshold of remote failures to prevent compounding the problem

Slide 106

Slide 106 text

messages are buffered in the queue and ﬂush when the system stablizes

Slide 107

Slide 107 text

since redis’ working set must ﬁt in memory, this must be monitored closely

Slide 108

Slide 108 text

interoperability

Slide 109

Slide 109 text

SMS Provider SIP Receiver SIP Redis Mobile Originated Consumer SMS

Slide 110

Slide 110 text

SMS Provider SIP Receiver SIP Redis Mobile Originated Consumer SMS

Slide 111

Slide 111 text

Concerns

Slide 112

Slide 112 text

monitoring

Slide 113

Slide 113 text

QoS

Slide 114

Slide 114 text

QoS = [msg exited timestamp] - [msg entered timestamp]

Slide 115

Slide 115 text

nginx GroupMe Transport Messaging Provider Message Entered Timestamp Captured Message Exited Timestamp Captured

Slide 116

Slide 116 text

QoS = 1370098106.90963 - 1370098106.493368

Slide 117

Slide 117 text

QoS = ~416ms

Slide 118

Slide 118 text

No content

Slide 119

Slide 119 text

Queue Size

Slide 120

Slide 120 text

LLEN [queue name]

Slide 121

Slide 121 text

often zero unless failures or throttling requirements have caused the senders to apply back pressure

Slide 122

Slide 122 text

delivery guarantees

Slide 123

Slide 123 text

exactly once at least once at most once

Slide 124

Slide 124 text

business requirements drive what delivery guarantees must be maintained

Slide 125

Slide 125 text

interface synchronization

Slide 126

Slide 126 text

your client and service must agree on the format of the messages

Slide 127

Slide 127 text

ruby objects are used to represent messages and serialize themselves for transport

Slide 128

Slide 128 text

consumers and producers work with the same objects which enforce a consistent interface

Slide 129

Slide 129 text

these clients are shared via a gem which is versioned along with the service

Slide 130

Slide 130 text

failure modes

Slide 131

Slide 131 text

the behavior of your system during failure

Slide 132

Slide 132 text

avoid cascading failure

Slide 133

Slide 133 text

what happens when redis is unavailable?

Slide 134

Slide 134 text

or memory ﬁlls up?

Slide 135

Slide 135 text

durability

Slide 136

Slide 136 text

(and performance)

Slide 137

Slide 137 text

redis by default isn’t very durable

Slide 138

Slide 138 text

even with AOF and replication, message loss is still possible

Slide 139

Slide 139 text

kafka doesn’t acknowledge produced messages

Slide 140

Slide 140 text

Learnings

Slide 141

Slide 141 text

you can go a long way with a monolith

Slide 142

Slide 142 text

messaging is a tool for service-oriented systems building

Slide 143

Slide 143 text

there are alternatives to HTTP when designing cooperating systems

Slide 144

Slide 144 text

backgrounded work is a prime candidate for service extraction

Slide 145

Slide 145 text

service extraction provides many of the promised beneﬁts

Slide 146

Slide 146 text

as with all engineering decisions: there are trade-offs to make to realize these beneﬁts

Slide 147

Slide 147 text

you must explicitly plan for failure cases

Slide 148

Slide 148 text

failure is another mode of operation

Slide 149

Slide 149 text

the system must bend but not break

Slide 150

Slide 150 text

you won’t know how broken something is until you measure it

Slide 151

Slide 151 text

look at percentile slices rather than averages

Slide 152

Slide 152 text

No content

Slide 153

Slide 153 text

you will have to experience system failure to build resilience into the system and your team

Slide 154

Slide 154 text

18) Failure free operations require experience with failure. Recognizing hazard and successfully manipulating system operations to remain inside the tolerable performance boundaries requires intimate contact with failure. http://www.ctlab.org/documents/How%20Complex%20Systems%20Fail.pdf How Complex Systems Fail

Slide 155

Slide 155 text

we prefer smaller building blocks like redis and kafka over message brokers like rabbitmq, qpid, etc