Message queue overview

Transcript

1. Message queue overview
2. A starting use case Let’s create a reason to talk about message queues 1
3. www.leblogdefrancis.com • A blog of articles • Registered users can start a thread on an article • When an user reply on a thread, a notification must be sent to every participants of this thread 3
4. Straightforward implementation 1. Get all the users that already commented on the thread 2. Iterate on them 3. Check if they have notifications enabled 4. An infinity of possible checks actually... 5. Send the notifications 4
5. Main issues with it • Bad for performance • Not decoupled • Not scalable in an efficient way • Bad for exceptions handling • Less reliability • Not flexible • Is it essential to send the notifications there ? 5
6. How can we improve it? Spoiler alert : message queues can help us 2
7. Current synchronous workflow 7
8. Go asynchronous Run the expensive tasks later, in the background. 8
9. Better asynchronous workflow 9
10. Cron? • Not adapted • Less functionalities • Not instantaneous • Not scalable 10
11. Message queue! • Something sends a message • The message ends up in a queue • Something process the message 11
12. Theory Not too much I promise <3 3
13. A common language • A producer publishes a message • A message broker accepts and forwards a message • A consumer consumes / receives a message 13
15. The message • A message is some data • It should encapsulate everything consumers might need • In our example use case : ◦ The id / uuid of the reply ◦ The content of the notification and the query to find the recipients 15
16. The producer • A program that sends the message • In our example use case : ◦ Our application ◦ In a dedicated service ◦ In an event listener 16
17. The message broker • An intermediary program that handle the message between the producer and the consumer • Objective : decoupling the awareness between producers and consumers • In our example use case : ◦ An existing one 17
18. The consumer • A program that waits to receive the message • This program logically does the expensive task • In our example use case : ◦ Our application ◦ In a dedicated service ◦ Another application ◦ In another language ◦ On another server 18
20. The queue • Inside the message broker • Managed by the message broker • FIFO (First In First Out) 20
21. Why is it better? Do not trust me, try it yourselves! 4
22. Performant and decoupled • Performance : ◦ Publishing a message takes a few milliseconds ◦ Sending the notifications does not block the response • Decoupled : ◦ Not perfect in our example but : ◦ The publish and consume parts are totally separated ◦ So extracting the consumer would be way more easier 22
23. Scalable and recoverable • Scalable : ◦ We just have to spawn new instances of our consumer • Better exceptions handling : ◦ Because of the retry mechanism ◦ And the dead letter exchange mechanism ◦ By splitting one message into several small ones 23
24. Reliable and flexible • Reliable : ◦ If our application (producer) is down, all the currently waiting messages can still be processed • Flexible : ◦ We can replace our current consumer implementation ◦ We can add a new behavior with a new consumer 24
25. Do not confuse More vocabulary to speak like a pro 5
26. Standards / protocols • AMQP (Asynchronous Message Queue Protocol) • MQTT (Message Queuing Telemetry Transport) • STOMP (Streaming Text Oriented Messaging Protocol) • And more... 26
27. Implementations & abstractions • php-amqplib • JMS (Java Message Service) • Symfony Messenger • And many more... 27
28. Message brokers • Apache Kafka • Apache ActiveMQ • Apache Qpid And dozens more... 28
29. Message queuing services • Amazon Web Services SQS (Simple Queue Service) • Google Cloud Platform Pub/Sub • Microsoft Azure Storage queues & Service Bus queues • And more... 29
30. Common use cases Directly taken from the RabbitMQ tutorials 6
31. Quick focus on RabbitMQ • Open Source message broker, originally for the AMQP protocol • The most widely deployed Open Source message broker in the world • Written in Erlang, a concurrent and functional programming language 31
32. Mature system • TLS / SSL support • Clustering • Authentication and authorisation mechanisms • And more... 32
33. The exchange • The producer actually sends the message to an exchange • The exchange knows what to do with the message (forwards it to one or several queues, or ignore it) • 4 Different types : direct, topic, headers and fanout 33
35. Work queue • Most simple case like our example • Delay an expensive task • Nameless exchange “” 35
37. Publish / Subscribe • The same message for several consumers • Exchange type : fanout • In our example : ◦ We want to send an email in addition to the notification ◦ We just need to create a new consumer 37
39. Routing • Receive messages selectively • Bind an exchange and a queue with a routing key • Exchange type : direct 39
41. Topics • Receive message selectively but based on a pattern • The routing key contains a specific format with special characters • Exchange type : topic 41
43. Bonus Because I still have time 7
44. Warning ! • Message queue might be overkill • Using an actual implementation is not that easy • Infrastructure setup is not negligible 44
45. My previous real usage cases • Imports • Exports • Bulk actions 45
46. UNIX power • There are two common message queue implementations in UNIX • The first one, System V IPC messages, was done in 1983 • In other words : message queue is not new at all! 46
47. 47 Thank you Any questions?