Slide 1
Slide 1 text
Event-Driven Architecture
Event-Driven Architecture
Messaging Patterns for Ruby Microservices
https://bit.ly/2RSudnT
Kirill Shevchenko
Kirill Shevchenko
Slide 2
Slide 2 text
Who I am
Who I am
Ruby/JS developer, Tech Lead at
Slide 3
Slide 3 text
Who I am
Who I am
Ruby/JS developer, Tech Lead at
Two years with Microservices
Slide 4
Slide 4 text
Who I am
Who I am
Ruby/JS developer, Tech Lead at
Two years with Microservices
Maintainer of Ruby/Rails digest on
Slide 5
Slide 5 text
Who I am
Who I am
Ruby/JS developer, Tech Lead at
Two years with Microservices
Maintainer of Ruby/Rails digest on
Live in Dnipro, Ukraine
Slide 6
Slide 6 text
Who I am
Who I am
Ruby/JS developer, Tech Lead at
Two years with Microservices
Maintainer of Ruby/Rails digest on
Live in Dnipro, Ukraine
kirill_shevch
Slide 7
Slide 7 text
Who I am
Who I am
Ruby/JS developer, Tech Lead at
Two years with Microservices
Maintainer of Ruby/Rails digest on
Live in Dnipro, Ukraine
kirillshevch
kirill_shevch
Slide 8
Slide 8 text
Agenda
Agenda
Road from Monolith to Microservices
Communication Patterns
Message Brokers
Publish/Subscribe
Testing and Monitoring
Slide 9
Slide 9 text
Microservices Are
Microservices Are
Something You Grow Into,
Something You Grow Into,
Not Begin With
Not Begin With
Slide 10
Slide 10 text
Let The Adventure
Let The Adventure
Begin
Begin
Slide 11
Slide 11 text
"Hello World" Setup
"Hello World" Setup
+
Slide 12
Slide 12 text
"Hello World" Setup
"Hello World" Setup
Slide 13
Slide 13 text
Background Jobs
Background Jobs
+
Slide 14
Slide 14 text
Background Jobs
Background Jobs
Slide 15
Slide 15 text
Cache
Cache
Slide 16
Slide 16 text
Business logic size
Business logic size
Orders?
Slide 17
Slide 17 text
Business logic size
Business logic size
Products?
Orders?
Slide 18
Slide 18 text
Business logic size
Business logic size
Analytics?
Products?
Orders?
Slide 19
Slide 19 text
Business logic size
Business logic size
Analytics?
Products?
Orders?
Users?
Slide 20
Slide 20 text
Applying DDD to Microservices
Applying DDD to Microservices
Slide 21
Slide 21 text
Pattern: Database Per Service
Pattern: Database Per Service
Slide 22
Slide 22 text
Pattern: Database Per Service
Pattern: Database Per Service
Slide 23
Slide 23 text
Pattern: Database Per Service
Pattern: Database Per Service
Slide 24
Slide 24 text
Messaging Patterns
Messaging Patterns
Slide 25
Slide 25 text
Sync vs Async Approach
Sync vs Async Approach
Two Ways
Two Ways
Slide 26
Slide 26 text
Sync vs Async Approach
Sync vs Async Approach
Two Ways
Two Ways
The publisher writes a message and waits until reader
confirm the reading
Slide 27
Slide 27 text
Sync vs Async Approach
Sync vs Async Approach
Two Ways
Two Ways
The publisher writes a message and waits until reader
confirm the reading
The writer doesn't wait for the reader, you can read
the message at any time
Slide 28
Slide 28 text
Synchronous Calls
Synchronous Calls
Probably, easiest communication pattern to
Probably, easiest communication pattern to
implement is simply calling another service
implement is simply calling another service
synchronously, usually via REST.
synchronously, usually via REST.
Service 1 Service 2
REST
Slide 29
Slide 29 text
Synchronous Calls
Synchronous Calls
Protocols
Protocols
HTTP/RPC
HTTP/RPC
Slide 30
Slide 30 text
Synchronous Calls
Synchronous Calls
Protocols
Protocols
HTTP/RPC
HTTP/RPC
except for
except for
streaming
streaming
Slide 31
Slide 31 text
HTTP/REST IS GREAT
HTTP/REST IS GREAT
Easy to understand (text protocol)
Web infrastructure already built on top of HTTP
Great tooling for testing, inspection, modification
Loose coupling between clients/server makes changes easy
High-quality http implementations in every language
Slide 32
Slide 32 text
Synchronous Calls
Synchronous Calls
Problems?
Problems?
You need wait for a response
You need to handle errors (like timeouts)
You work with many different APIs
You you use a lot of different API wrappers
Slide 33
Slide 33 text
A gem for building API Wrappers in Ruby
A gem for building API Wrappers in Ruby
Slide 34
Slide 34 text
ApiStruct: Configuration
ApiStruct: Configuration
ApiStruct::Settings.configure do |config|
config.endpoints = {
some_api: {
root: 'https://someapi.dev/v1'
}
}
end
Slide 35
Slide 35 text
ApiStruct: Client
ApiStruct: Client
class OrdersClients < ApiStruct::Client
some_api :orders
def index
get
end
def show(id)
get(id)
end
end
Slide 36
Slide 36 text
ApiStruct: Entity
ApiStruct: Entity
class Order < ApiStruct::Entity
client_service OrdersClient
attr_entity :id, :state, :total_price
end
Slide 37
Slide 37 text
ApiStruct: Entity
ApiStruct: Entity
class Order < ApiStruct::Entity
client_service OrdersClient
attr_entity :id, :state, :total_price
end
order = Order.show(1)
# => {"id"=>1, "state"=>"shipping",
"total_price"=>"99"}
Slide 38
Slide 38 text
Event-Driven Architecture
Event-Driven Architecture
(
(EDA
EDA)
)
Slide 39
Slide 39 text
Сommunication Сoncepts
Сommunication Сoncepts
Messages
Messages
Events
Events
Commands
Commands
Slide 40
Slide 40 text
Messages
Messages
The basic unit of communication in Message Brokers
The basic unit of communication in Message Brokers
Slide 41
Slide 41 text
Messages
Messages
The basic unit of communication in Message Brokers
The basic unit of communication in Message Brokers
Literally can be anything
Literally can be anything
Slide 42
Slide 42 text
Messages
Messages
The basic unit of communication in Message Brokers
The basic unit of communication in Message Brokers
Literally can be anything
Literally can be anything
An ID, a string, an object, a
An ID, a string, an object, a
command, an event or whatever.
command, an event or whatever.
Slide 43
Slide 43 text
Commands
Commands
It's a one-to-one communication between publisher
It's a one-to-one communication between publisher
and subscriber
and subscriber
Slide 44
Slide 44 text
Commands
Commands
It's a one-to-one communication between publisher
It's a one-to-one communication between publisher
and subscriber
and subscriber
Event
Publisher
Subscriber
Slide 45
Slide 45 text
Events
Events
An event is a message which informs listeners about
An event is a message which informs listeners about
something which has happened
something which has happened
Slide 46
Slide 46 text
Events
Events
An event is a message which informs listeners about
An event is a message which informs listeners about
something which has happened
something which has happened
Event
Publisher
Subscriber 1 Subscriber 2
Slide 47
Slide 47 text
Topic
Topic Queue
Queue
vs
Slide 48
Slide 48 text
Topic
Topic Queue
Queue
vs
All subscribers will receive a
All subscribers will receive a
copy of the message
copy of the message
Slide 49
Slide 49 text
All subscribers will receive a
All subscribers will receive a
copy of the message
copy of the message
A single message will be
A single message will be
received by exactly one
received by exactly one
consumer
consumer
Topic
Topic Queue
Queue
vs
Slide 50
Slide 50 text
Queues
Queues
Pros
Pros
Simple messaging pattern with a
transparent communication flow
Messages can be recovered by
putting them back on the queue
Slide 51
Slide 51 text
Cons
Cons
Only one consumer can get the
message
Implies a coupling between
producer and consumer as it’s
an one-to-one relation
Queues
Queues
Slide 52
Slide 52 text
Topics
Topics
Pros
Pros
Multiple consumers can get a
message
Decoupling between producer
and consumers (publish-and-
subscribe pattern)
Slide 53
Slide 53 text
Cons
Cons
More complicated
communication flow
A message cannot be recovered
for a single listener
Topics
Topics
Slide 54
Slide 54 text
Virtual Topics
Virtual Topics
Pros
Pros
Multiple consumers can get a message
Decoupling between producer and
consumers (publish-and-subscribe
pattern)
Messages can be recovered by putting
them back on the queue
Slide 55
Slide 55 text
Virtual Topics
Virtual Topics
Cons
Cons
Might require additional
configuration in the broker
Slide 56
Slide 56 text
FIFO (First-In-First-Out)
FIFO (First-In-First-Out)
Queues
Queues
The order in which messages are sent
and received is strictly preserved.
Slide 57
Slide 57 text
Message Brokers
Message Brokers
Slide 58
Slide 58 text
Checklist
Checklist
Services discovery
Messaging
Load balancing
Logging
Queuing
Slide 59
Slide 59 text
ACTIVE
Apache
MQ
Slide 60
Slide 60 text
Amazon SQS
Amazon SQS
Fully managed message queue
Fully managed message queue
Slide 61
Slide 61 text
Amazon SQS
Amazon SQS
Fully managed message queue
Fully managed message queue
Pros
Pros Cons
Cons
Slide 62
Slide 62 text
Amazon SQS
Amazon SQS
Fully managed message queue
Fully managed message queue
Pros
Pros
Available as a cloud service
Cons
Cons
Slide 63
Slide 63 text
Amazon SQS
Amazon SQS
Fully managed message queue
Fully managed message queue
Pros
Pros
Available as a cloud service
Cons
Cons
FIFO queues support only 300
messages per second
Slide 64
Slide 64 text
Amazon SQS
Amazon SQS
Fully managed message queue
Fully managed message queue
Pros
Pros
Available as a cloud service
Cons
Cons
FIFO queues support only 300
messages per second
No ability to use FIFO with
Pub/Sub (SNS)
Slide 65
Slide 65 text
ACTIVE
Apache
MQ
Slide 66
Slide 66 text
ActiveMQ
ActiveMQ
Pros
Pros
It has two main concepts: topics
and queues
Slide 67
Slide 67 text
ActiveMQ
ActiveMQ
Pros
Pros
It has two main concepts: topics
and queues
Load Balancing: replication,
horizontal scaling
Slide 68
Slide 68 text
ActiveMQ
ActiveMQ
Pros
Pros
It has two main concepts: topics
and queues
Load Balancing: replication,
horizontal scaling
Support for most protocols
Slide 69
Slide 69 text
ActiveMQ
ActiveMQ
Pros
Pros
It has two main concepts: topics
and queues
Load Balancing: replication,
horizontal scaling
Support for most protocols
Cons
Cons
Need backups
Slide 70
Slide 70 text
ActiveMQ
ActiveMQ
Pros
Pros
It has two main concepts: topics
and queues
Load Balancing: replication,
horizontal scaling
Support for most protocols
Cons
Cons
Need backups
Might require additional
configuration(FIFO, Virtual Topics)
Slide 71
Slide 71 text
Protocols
Protocols
Slide 72
Slide 72 text
Protocols
Protocols
OpenWire
OpenWire
STOMP
STOMP
AMQP
AMQP
MQTT
MQTT
What to сhoose?
What to сhoose?
WebSocket
WebSocket
XMPP
XMPP
Slide 73
Slide 73 text
ActiveMQ
ActiveMQ
Instance can provide
Instance can provide
all protocols at the
all protocols at the
same time
same time
Slide 74
Slide 74 text
ActiveMQ
ActiveMQ
Endpoints
Endpoints
Slide 75
Slide 75 text
Publish/Subscribe
Publish/Subscribe
pattern and Ruby
pattern and Ruby
Slide 76
Slide 76 text
Publishing is simply
Publishing is simply
options = {
command: 'update_order',
data: { order_id: 1, state: :paid }
}
redis = Redis.new
redis.publish(:orders, options.to_json)
Slide 77
Slide 77 text
But we need handle
But we need handle
pubishing result
pubishing result
options = {
command: 'update_order',
data: { order_id: 1, state: :paid }
}
redis = Redis.new
redis.publish(:orders, options.to_json).tap do |r|
Raven.capture_message('Publish failed', extra: options) if r.zero?
end
Slide 78
Slide 78 text
Pub/Sub pattern
Pub/Sub pattern
redis = Redis.new
redis.subscribe(:orders) do |on|
on.message do |channel, message|
# call the command
end
end
Slide 79
Slide 79 text
Pub/Sub pattern
Pub/Sub pattern
redis = Redis.new
Thread.new do
redis.subscribe(:orders) do |on|
on.message do |channel, message|
begin
# call the command
rescue StandardError => e
Raven.capture_exception(e)
end
end
end
end
Slide 80
Slide 80 text
Testing
Testing
Slide 81
Slide 81 text
No content
Slide 82
Slide 82 text
Real life problems
Real life problems
No one really knows how to test
microservices
Slide 83
Slide 83 text
Real life problems
Real life problems
No one really knows how to test
microservices
No one really knows how to test
infrastructure for microservices
Slide 84
Slide 84 text
Solutions?
Solutions?
Unit testing for each service with hard
mocking
Slide 85
Slide 85 text
Solutions?
Solutions?
Unit testing for each service with hard
mocking
Up the server with all services for
running integration testing
Slide 86
Slide 86 text
Monitoring
Monitoring
No more single system, no more centralized
monitoring
Slide 87
Slide 87 text
Monitoring
Monitoring
No more single system, no more centralized
monitoring
Logs and metrics collecting (Beats, Statsd,
Elasticsearch)
Slide 88
Slide 88 text
Monitoring
Monitoring
No more single system, no more centralized
monitoring
Logs and metrics collecting (Beats, Statsd,
Elasticsearch)
Aggregation (Logstash)
Slide 89
Slide 89 text
Monitoring
Monitoring
No more single system, no more centralized
monitoring
Logs and metrics collecting (Beats, Statsd,
Elasticsearch)
Aggregation (Logstash)
Visualization (Kibana, Grafana)
Slide 90
Slide 90 text
Monitoring
Monitoring
No more single system, no more centralized
monitoring
Logs and metrics collecting (Beats, Statsd,
Elasticsearch)
Aggregation (Logstash)
Visualization (Kibana, Grafana)
Store and maintain all this data and systems
Slide 91
Slide 91 text
Thanks!
Thanks!