Decoupling Distributed Ruby Applications with RabbitMQ

As a software project grows, so does its complexity. Micro-services are all the rage for breaking up code into smaller, more manageable pieces. But how does one iterate to a micro-service architecture? How does one keep code not only distributed, but decoupled? In this talk I'll explain how a message bus can be used to facilitate not only service communication, but service development. I'll focus on RabbitMQ, though the opinions presented can be applied to other technologies.

Transcript

1. Decoupling Distributed Ruby Applications with RabbitMQ Eric Saxby ecdysone sax
2. Who am I? I computer for a living I have done a wide range of jobs, some without computers My first job out of a biology degree was Flash development, after learning Flash in a weekend
3. Who am I? I computer for a living I have done a wide range of jobs, some without computers My first job out of a biology degree was Flash development, after learning Flash in a weekend My first job out of a biology degree was as an usher for a circus
4. Who am I? I computer for a living I have done a wide range of jobs, some without computers My second job out of a biology degree was Flash development, after learning Flash in a weekend My first job out of a biology degree was as an usher for a circus
5. Who am I? I computer for a living I have done a wide range of jobs, some without computers My second job out of a biology degree was Flash development, after learning Flash in a weekend My first job out of a biology degree was as an usher for a circus I've done software development for 19 years, but have only been a full-time programmer for 8
6. Who am I? I computer for a living I have done a wide range of jobs, some without computers My second job out of a biology degree was Flash development, after learning Flash in a weekend My first job out of a biology degree was as an usher for a circus I've done software development for 19 years, but have only been a full-time programmer for 8 Toby
7. Who am I? I've worked with Ruby for 7 1/2 years I've worked on projects with tens of millions of active users I've worked on projects with hundreds of millions of annual revenue I've worked on projects with billions of database rows I've worked in development and operations I hate being paged
8. Who am I? I've worked with Ruby for 7 1/2 years I've worked on projects with tens of millions of active users I've worked on projects with hundreds of millions of annual revenue I've worked on projects with billions of database rows I've worked in development and operations I hate being paged Toby
9. Who am I? Most of the work that taught me about this subject was done at ModCloth and Wanelo I am currently consulting at Pivotal, on CloudFoundry
10. RabbitMQ RabbitMQ
11. Message Bus Pattern Message Bus Pattern
12. What is a message bus?
13. What is a message bus?
14. What is a message bus?
15. What is a message bus?
16. What is a message bus?
17. What is a message bus?
18. What is a message bus?
19. What is a message bus? A bucket, into which an application can put messages Other applications can choose to listen for specific types of messages, based on some metadata attached to the message
20. Let us look closer... The producer knows about the bucket, and it knows about the message The consumer knows about the bucket, and it knows about the message The producer and consumer do not know about each other Multiple consumers can listen for the same message, without changing the producer
21. Why do I want this? Create applications that send messages to each other, without knowing about each other Break apart an application that is too big (by some arbitrary definition) Application has business logic that is not intrinsic to its main purpose Introduce new business logic that current applications don't care about Start to consume events from something else, but don't want to add data translation, authentication to current applications
22. Why do I want this? Want to deploy multiple apps without dropping messages Some functionality can to be scaled entirely differently from others Some components require different availability than others
23. Why do I want this? Persistent TCP connections can be much more performant than HTTP when > millions messages per minute
25. Millions of Cats is a picture book written and illustrated by Wanda Gág in 1928. The book won a Newbery Honor award in 1929, one of the few picture books to do so. Millions of Cats is the oldest American picture book still in print.[1] wikipedia
26. A bucket, into which an application can put messages Other applications can choose to listen for specific types of messages, based on some metadata attached to the message
27. A bucket, into which an application can put messages Other applications can choose to listen for specific types of messages, based on some metadata attached to the message If you are like me (and everyone I've worked with), as soon as you start reading documentation, you will jump to incorrect conclusions We are biased by experiences with other tools New tools come with new vocabulary. Even the best documentation does not make sense until you see a technology fail.
28. What is not a message bus? Remote Procedure Calls Background jobs / asynchronous workers — chained HTTP requests count as RPC I assume clients are written to mask the HTTP (see hexagonal architecture)
29. RPC An event happens in Application A The event is not complete unless Application B knows of it Application A tells Application B about the event Application B responds to say that everything is OK
30. RPC not a message bus? The client Application A has implicit knowledge of Application B which needs to be available For N services, each application must know about N with retry logic and error handling
31. Background jobs Sidekiq, Resque, Delayed Job Used when can/should be done outside of a web request Used when application is not a web application Independent daemon pulls messages from a queue, routes them to specific classes based on metadata
34. Background jobs not a message bus? Producer generates messages Consumer listens for message based on metadata Producer has intimate knowledge of the consumer multiple consumers require changing the producer to generate multiple messages Designed for message passing between processes of a single application
35. What is a message bus?
36. Not specific to a particular technology Kafka RabbitMQ (AMQP) Amazon SNS and SQS Alternate AMQP implementations (SwiftMQ, ActiveMQ, Apollo) Build your own! * * don't actually build your own
37. Central piece of infrastructure, where producers can publish messages and consumers can subscribe to messages Used in a way that decouples producers from consumers
38. RabbitMQ RabbitMQ
39. Easy to start development Few dependencies (Erlang) Ruby protocol gems are mature Scales to production loads
40. AMQP terminology Exchange Broker Queue Binding Message
41. AMQP terminology Exchange Broker Queue Binding Message Process running on a node
42. AMQP terminology Exchange Broker Queue Binding Message Process running on a node; can be clustered
43. AMQP terminology Exchange Broker Queue Binding Message Process running on a node; can be clustered Place where messages can be published
44. AMQP terminology Exchange Broker Queue Binding Message Process running on a node; can be clustered Place where messages can be published Registered in an exchange with a binding. Matched messages are copied into a queue.
45. AMQP terminology Exchange Broker Queue Binding Message Process running on a node; can be clustered Place where messages can be published Registered in an exchange with a binding. Matched messages are copied into a queue. Rules used by exchange to route messages
46. AMQP terminology Exchange Broker Queue Binding Message Process running on a node; can be clustered Place where messages can be published Registered in an exchange with a binding. Matched messages are copied into a queue. Rules used by exchange to route messages Arbitrary payload; includes routing_key metadata
47. AMQP terminology Routing key arbitrary.dot.delimited text.describing the.message
48. AMQP terminology Virtual Host An isolated environment containing groups of users, exchanges, queues A bucket of buckets
49. AMQP terminology Producers and consumers connect to a vhost on a broker Producers and consumers connect to an exchange in the vhost Producers publish messages to the exchange Consumers register queues with bindings in the exchange The exchange uses the bindings to route each message to 0 or more queues
50. Types of exchanges Types of exchanges Direct Direct Fanout Fanout Topic Topic
51. Direct Exchange When queue binds, it includes a routing key K If message routing key R == K, exchange routes it to queue Each application registers its own queue Producer can target an application by using the correct routing key
52. https://www.rabbitmq.com/tutorials/amqp-concepts.html
53. Problems with Direct Exchange Producer knows about the consumers To target multiple consumers, duplicate messages must be published Adding a consumer requires changing publisher
54. Fanout Exchange Each message is written to every queue Routing keys are ignored Useful when multiple apps care about the same message Producer knows nothing about consumers
55. https://www.rabbitmq.com/tutorials/amqp-concepts.html
56. Problems with Fanout Exchange If a queue exists, it will receive every message on the exchange If an application does not care about every message, it must ignore them Introduces more work than is necessary
57. Topic Exchange Queue is bound with a routing key pattern Exchange uses binding to put messages in matching queues Messages that are not matched are dropped some.arbitrary.routing.* some.arbitrary.# single word wildcard 0 or more word wildcard some.arbitrary.routing.key exact match
58. Yay Topic Exchanges! Producer does not know about consumers Consumers can listen for specific messages Adding new messages is cheap
59. Message lifecycle Producer writes message to exchange; may wait for acknowledgement Exchanges compares bindings, writes message to 0 or more queues Queue pushes messages to a waiting consumer Consumer acknowledges message Producer can optionally wait for a consumer receipt
60. Resource utilization Messages consume RAM With durable queues, routing messages to queues causes disk I/O Don't let queues back up! If disk I/O becomes a problem, scale by adding brokers
61. Scaling and availability Brokers can be clustered Queues can be mirrored in a cluster When a primary disappears, the oldest replica is promoted Any node can receive operations, which actually occur on the master New mirrors only receive new messages
62. Or... Pretend all nodes are independent Producers connect through HAProxy or ELB When a node disappears, producers reconnect to another node Deploy consumers directly connecting to each node
63. Keep queues empty Combine RabbitMQ with another tool, like Sidekiq Consumer receives message, routes it to Sidekiq, acknowledges Do as little work in the RabbitMQ consumer daemon as possible Scale application work separately from message bus resources
64. AMQP in Ruby amqp gem — eventmachine bunny gem — not eventmachine amqp gem great for standalone daemons or other eventmachine processes amqp gem not great for non-evented processes, like unicorn produce messages with bunny consume messages with amqp
65. Demo Demo https://github.com/sax/talk-notes/tree/master/2016/demo
66. What could go wrong? What could go wrong?
67. Availability Once a message bus is introduced, it must be highly available When RabbitMQ goes down, you are in a bad place Ensure that queues do not back up! Focusing on a single high available service may be easier than making everything resilient
68. Pattern matching Don't use * when you mean # Every other thing related to computers uses *, so mistakes are easy Double check all routing keys all the time
69. Grammar and routing keys thing.create things.create thing.created things.created thingie.crated Inconsistency leads to mistakes When routing keys come from external sources, double check everything!
70. Naming queues and exchanges my.topic product.related.routing.key application.product.action What will be the least confusing, and help the most with debugging? What is generic across applications, and what is specific to an application?
71. Properties can't be changed after the fact Properties are set at queue creation Different queue properties don't necessarily raise errors Changing properties requires creating new queues, then deleting the old
72. Queues don't delete themselves An application may be stopped, but its queues keep receiving messages Changing queue names
74. Opinions coded into a gem named lapine https://github.com/messagebus/lapine
76. https://www.flickr.com/photos/30581630@N04/5387336231
77. https://www.flickr.com/photos/dhendrix/6906652333
78. https://www.flickr.com/photos/davidmaitland/3796363158
79. https://www.flickr.com/photos/eiriknewth/1238351905/
80. https://www.flickr.com/photos/studiospecialplace/2848127770