Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[RubyWine #1] Event-Driven Architecture and Messaging Patterns for Ruby Microservices

[RubyWine #1] Event-Driven Architecture and Messaging Patterns for Ruby Microservices

How to choose a communication protocol for services? Synchronous vs Asynchronous approach. HTTP, Brokered Messaging and gRPC. Issues when working with Threads and asynchronous message queues in Ruby and their solutions. Logging and monitoring of transactional integrity. Publish/Subscribe messaging systems and a brief comparison of them.

Video - https://youtu.be/e9AAUy4kkek

3adc73c22d225bfff22c9144e37ca78c?s=128

Kirill Shevchenko

April 13, 2019
Tweet

More Decks by Kirill Shevchenko

Other Decks in Programming

Transcript

  1. Event-Driven Architecture Event-Driven Architecture Messaging Patterns for Ruby Microservices https://bit.ly/2RSudnT

    Kirill Shevchenko Kirill Shevchenko
  2. Who I am Who I am Ruby/JS developer, Tech Lead

    at
  3. Who I am Who I am Ruby/JS developer, Tech Lead

    at Two years with Microservices
  4. Who I am Who I am Ruby/JS developer, Tech Lead

    at Two years with Microservices Maintainer of Ruby/Rails digest on
  5. Who I am Who I am Ruby/JS developer, Tech Lead

    at Two years with Microservices Maintainer of Ruby/Rails digest on Live in Dnipro, Ukraine
  6. Who I am Who I am Ruby/JS developer, Tech Lead

    at Two years with Microservices Maintainer of Ruby/Rails digest on Live in Dnipro, Ukraine kirill_shevch
  7. Who I am Who I am Ruby/JS developer, Tech Lead

    at Two years with Microservices Maintainer of Ruby/Rails digest on Live in Dnipro, Ukraine kirillshevch kirill_shevch
  8. Agenda Agenda Road from Monolith to Microservices Communication Patterns Message

    Brokers Publish/Subscribe Testing and Monitoring
  9. Microservices Are Microservices Are Something You Grow Into, Something You

    Grow Into, Not Begin With Not Begin With
  10. Let The Adventure Let The Adventure Begin Begin

  11. "Hello World" Setup "Hello World" Setup +

  12. "Hello World" Setup "Hello World" Setup

  13. Background Jobs Background Jobs +

  14. Background Jobs Background Jobs

  15. Cache Cache

  16. Business logic size Business logic size Orders?

  17. Business logic size Business logic size Products? Orders?

  18. Business logic size Business logic size Analytics? Products? Orders?

  19. Business logic size Business logic size Analytics? Products? Orders? Users?

  20. Applying DDD to Microservices Applying DDD to Microservices

  21. Pattern: Database Per Service Pattern: Database Per Service

  22. Pattern: Database Per Service Pattern: Database Per Service

  23. Pattern: Database Per Service Pattern: Database Per Service

  24. Messaging Patterns Messaging Patterns

  25. Sync vs Async Approach Sync vs Async Approach Two Ways

    Two Ways
  26. Sync vs Async Approach Sync vs Async Approach Two Ways

    Two Ways The publisher writes a message and waits until reader confirm the reading
  27. Sync vs Async Approach Sync vs Async Approach Two Ways

    Two Ways The publisher writes a message and waits until reader confirm the reading The writer doesn't wait for the reader, you can read the message at any time
  28. Synchronous Calls Synchronous Calls Probably, easiest communication pattern to Probably,

    easiest communication pattern to implement is simply calling another service implement is simply calling another service synchronously, usually via REST. synchronously, usually via REST. Service 1 Service 2 REST
  29. Synchronous Calls Synchronous Calls Protocols Protocols HTTP/RPC HTTP/RPC

  30. Synchronous Calls Synchronous Calls Protocols Protocols HTTP/RPC HTTP/RPC except for

    except for streaming streaming
  31. HTTP/REST IS GREAT HTTP/REST IS GREAT Easy to understand (text

    protocol) Web infrastructure already built on top of HTTP Great tooling for testing, inspection, modification Loose coupling between clients/server makes changes easy High-quality http implementations in every language
  32. Synchronous Calls Synchronous Calls Problems? Problems? You need wait for

    a response You need to handle errors (like timeouts) You work with many different APIs You you use a lot of different API wrappers
  33. A gem for building API Wrappers in Ruby A gem

    for building API Wrappers in Ruby
  34. ApiStruct: Configuration ApiStruct: Configuration ApiStruct::Settings.configure do |config| config.endpoints = {

    some_api: { root: 'https://someapi.dev/v1' } } end
  35. ApiStruct: Client ApiStruct: Client class OrdersClients < ApiStruct::Client some_api :orders

    def index get end def show(id) get(id) end end
  36. ApiStruct: Entity ApiStruct: Entity class Order < ApiStruct::Entity client_service OrdersClient

    attr_entity :id, :state, :total_price end
  37. ApiStruct: Entity ApiStruct: Entity class Order < ApiStruct::Entity client_service OrdersClient

    attr_entity :id, :state, :total_price end order = Order.show(1) # => {"id"=>1, "state"=>"shipping", "total_price"=>"99"}
  38. Event-Driven Architecture Event-Driven Architecture ( (EDA EDA) )

  39. Сommunication Сoncepts Сommunication Сoncepts Messages Messages Events Events Commands Commands

  40. Messages Messages The basic unit of communication in Message Brokers

    The basic unit of communication in Message Brokers
  41. Messages Messages The basic unit of communication in Message Brokers

    The basic unit of communication in Message Brokers Literally can be anything Literally can be anything
  42. Messages Messages The basic unit of communication in Message Brokers

    The basic unit of communication in Message Brokers Literally can be anything Literally can be anything An ID, a string, an object, a An ID, a string, an object, a command, an event or whatever. command, an event or whatever.
  43. Commands Commands It's a one-to-one communication between publisher It's a

    one-to-one communication between publisher and subscriber and subscriber
  44. Commands Commands It's a one-to-one communication between publisher It's a

    one-to-one communication between publisher and subscriber and subscriber Event Publisher Subscriber
  45. Events Events An event is a message which informs listeners

    about An event is a message which informs listeners about something which has happened something which has happened
  46. Events Events An event is a message which informs listeners

    about An event is a message which informs listeners about something which has happened something which has happened Event Publisher Subscriber 1 Subscriber 2
  47. Topic Topic Queue Queue vs

  48. Topic Topic Queue Queue vs All subscribers will receive a

    All subscribers will receive a copy of the message copy of the message
  49. All subscribers will receive a All subscribers will receive a

    copy of the message copy of the message A single message will be A single message will be received by exactly one received by exactly one consumer consumer Topic Topic Queue Queue vs
  50. Queues Queues Pros Pros Simple messaging pattern with a transparent

    communication flow Messages can be recovered by putting them back on the queue
  51. Cons Cons Only one consumer can get the message Implies

    a coupling between producer and consumer as it’s an one-to-one relation Queues Queues
  52. Topics Topics Pros Pros Multiple consumers can get a message

    Decoupling between producer and consumers (publish-and- subscribe pattern)
  53. Cons Cons More complicated communication flow A message cannot be

    recovered for a single listener Topics Topics
  54. Virtual Topics Virtual Topics Pros Pros Multiple consumers can get

    a message Decoupling between producer and consumers (publish-and-subscribe pattern) Messages can be recovered by putting them back on the queue
  55. Virtual Topics Virtual Topics Cons Cons Might require additional configuration

    in the broker
  56. FIFO (First-In-First-Out) FIFO (First-In-First-Out) Queues Queues The order in which

    messages are sent and received is strictly preserved.
  57. Message Brokers Message Brokers

  58. Checklist Checklist Services discovery Messaging Load balancing Logging Queuing

  59. ACTIVE Apache MQ

  60. Amazon SQS Amazon SQS Fully managed message queue Fully managed

    message queue
  61. Amazon SQS Amazon SQS Fully managed message queue Fully managed

    message queue Pros Pros Cons Cons
  62. Amazon SQS Amazon SQS Fully managed message queue Fully managed

    message queue Pros Pros Available as a cloud service Cons Cons
  63. Amazon SQS Amazon SQS Fully managed message queue Fully managed

    message queue Pros Pros Available as a cloud service Cons Cons FIFO queues support only 300 messages per second
  64. Amazon SQS Amazon SQS Fully managed message queue Fully managed

    message queue Pros Pros Available as a cloud service Cons Cons FIFO queues support only 300 messages per second No ability to use FIFO with Pub/Sub (SNS)
  65. ACTIVE Apache MQ

  66. ActiveMQ ActiveMQ Pros Pros It has two main concepts: topics

    and queues
  67. ActiveMQ ActiveMQ Pros Pros It has two main concepts: topics

    and queues Load Balancing: replication, horizontal scaling
  68. ActiveMQ ActiveMQ Pros Pros It has two main concepts: topics

    and queues Load Balancing: replication, horizontal scaling Support for most protocols
  69. ActiveMQ ActiveMQ Pros Pros It has two main concepts: topics

    and queues Load Balancing: replication, horizontal scaling Support for most protocols Cons Cons Need backups
  70. ActiveMQ ActiveMQ Pros Pros It has two main concepts: topics

    and queues Load Balancing: replication, horizontal scaling Support for most protocols Cons Cons Need backups Might require additional configuration(FIFO, Virtual Topics)
  71. Protocols Protocols

  72. Protocols Protocols OpenWire OpenWire STOMP STOMP AMQP AMQP MQTT MQTT

    What to сhoose? What to сhoose? WebSocket WebSocket XMPP XMPP
  73. ActiveMQ ActiveMQ Instance can provide Instance can provide all protocols

    at the all protocols at the same time same time
  74. ActiveMQ ActiveMQ Endpoints Endpoints

  75. Publish/Subscribe Publish/Subscribe pattern and Ruby pattern and Ruby

  76. Publishing is simply Publishing is simply options = { command:

    'update_order', data: { order_id: 1, state: :paid } } redis = Redis.new redis.publish(:orders, options.to_json)
  77. But we need handle But we need handle pubishing result

    pubishing result options = { command: 'update_order', data: { order_id: 1, state: :paid } } redis = Redis.new redis.publish(:orders, options.to_json).tap do |r| Raven.capture_message('Publish failed', extra: options) if r.zero? end
  78. Pub/Sub pattern Pub/Sub pattern redis = Redis.new redis.subscribe(:orders) do |on|

    on.message do |channel, message| # call the command end end
  79. Pub/Sub pattern Pub/Sub pattern redis = Redis.new Thread.new do redis.subscribe(:orders)

    do |on| on.message do |channel, message| begin # call the command rescue StandardError => e Raven.capture_exception(e) end end end end
  80. Testing Testing

  81. None
  82. Real life problems Real life problems No one really knows

    how to test microservices
  83. Real life problems Real life problems No one really knows

    how to test microservices No one really knows how to test infrastructure for microservices
  84. Solutions? Solutions? Unit testing for each service with hard mocking

  85. Solutions? Solutions? Unit testing for each service with hard mocking

    Up the server with all services for running integration testing
  86. Monitoring Monitoring No more single system, no more centralized monitoring

  87. Monitoring Monitoring No more single system, no more centralized monitoring

    Logs and metrics collecting (Beats, Statsd, Elasticsearch)
  88. Monitoring Monitoring No more single system, no more centralized monitoring

    Logs and metrics collecting (Beats, Statsd, Elasticsearch) Aggregation (Logstash)
  89. Monitoring Monitoring No more single system, no more centralized monitoring

    Logs and metrics collecting (Beats, Statsd, Elasticsearch) Aggregation (Logstash) Visualization (Kibana, Grafana)
  90. Monitoring Monitoring No more single system, no more centralized monitoring

    Logs and metrics collecting (Beats, Statsd, Elasticsearch) Aggregation (Logstash) Visualization (Kibana, Grafana) Store and maintain all this data and systems
  91. Thanks! Thanks!