Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Message queue overview

Message queue overview

A7e7c34aaa3ff7eb359b6449fb8bb043?s=128

Thomas Calvet

November 16, 2018
Tweet

Transcript

  1. Message queue overview

  2. A starting use case Let’s create a reason to talk

    about message queues 1
  3. www.leblogdefrancis.com • A blog of articles • Registered users can

    start a thread on an article • When an user reply on a thread, a notification must be sent to every participants of this thread 3
  4. Straightforward implementation 1. Get all the users that already commented

    on the thread 2. Iterate on them 3. Check if they have notifications enabled 4. An infinity of possible checks actually... 5. Send the notifications 4
  5. Main issues with it • Bad for performance • Not

    decoupled • Not scalable in an efficient way • Bad for exceptions handling • Less reliability • Not flexible • Is it essential to send the notifications there ? 5
  6. How can we improve it? Spoiler alert : message queues

    can help us 2
  7. Current synchronous workflow 7

  8. Go asynchronous Run the expensive tasks later, in the background.

    8
  9. Better asynchronous workflow 9

  10. Cron? • Not adapted • Less functionalities • Not instantaneous

    • Not scalable 10
  11. Message queue! • Something sends a message • The message

    ends up in a queue • Something process the message 11
  12. Theory Not too much I promise <3 3

  13. A common language • A producer publishes a message •

    A message broker accepts and forwards a message • A consumer consumes / receives a message 13
  14. 14

  15. The message • A message is some data • It

    should encapsulate everything consumers might need • In our example use case : ◦ The id / uuid of the reply ◦ The content of the notification and the query to find the recipients 15
  16. The producer • A program that sends the message •

    In our example use case : ◦ Our application ◦ In a dedicated service ◦ In an event listener 16
  17. The message broker • An intermediary program that handle the

    message between the producer and the consumer • Objective : decoupling the awareness between producers and consumers • In our example use case : ◦ An existing one 17
  18. The consumer • A program that waits to receive the

    message • This program logically does the expensive task • In our example use case : ◦ Our application ◦ In a dedicated service ◦ Another application ◦ In another language ◦ On another server 18
  19. 19

  20. The queue • Inside the message broker • Managed by

    the message broker • FIFO (First In First Out) 20
  21. Why is it better? Do not trust me, try it

    yourselves! 4
  22. Performant and decoupled • Performance : ◦ Publishing a message

    takes a few milliseconds ◦ Sending the notifications does not block the response • Decoupled : ◦ Not perfect in our example but : ◦ The publish and consume parts are totally separated ◦ So extracting the consumer would be way more easier 22
  23. Scalable and recoverable • Scalable : ◦ We just have

    to spawn new instances of our consumer • Better exceptions handling : ◦ Because of the retry mechanism ◦ And the dead letter exchange mechanism ◦ By splitting one message into several small ones 23
  24. Reliable and flexible • Reliable : ◦ If our application

    (producer) is down, all the currently waiting messages can still be processed • Flexible : ◦ We can replace our current consumer implementation ◦ We can add a new behavior with a new consumer 24
  25. Do not confuse More vocabulary to speak like a pro

    5
  26. Standards / protocols • AMQP (Asynchronous Message Queue Protocol) •

    MQTT (Message Queuing Telemetry Transport) • STOMP (Streaming Text Oriented Messaging Protocol) • And more... 26
  27. Implementations & abstractions • php-amqplib • JMS (Java Message Service)

    • Symfony Messenger • And many more... 27
  28. Message brokers • Apache Kafka • Apache ActiveMQ • Apache

    Qpid And dozens more... 28
  29. Message queuing services • Amazon Web Services SQS (Simple Queue

    Service) • Google Cloud Platform Pub/Sub • Microsoft Azure Storage queues & Service Bus queues • And more... 29
  30. Common use cases Directly taken from the RabbitMQ tutorials 6

  31. Quick focus on RabbitMQ • Open Source message broker, originally

    for the AMQP protocol • The most widely deployed Open Source message broker in the world • Written in Erlang, a concurrent and functional programming language 31
  32. Mature system • TLS / SSL support • Clustering •

    Authentication and authorisation mechanisms • And more... 32
  33. The exchange • The producer actually sends the message to

    an exchange • The exchange knows what to do with the message (forwards it to one or several queues, or ignore it) • 4 Different types : direct, topic, headers and fanout 33
  34. 34

  35. Work queue • Most simple case like our example •

    Delay an expensive task • Nameless exchange “” 35
  36. 36

  37. Publish / Subscribe • The same message for several consumers

    • Exchange type : fanout • In our example : ◦ We want to send an email in addition to the notification ◦ We just need to create a new consumer 37
  38. 38

  39. Routing • Receive messages selectively • Bind an exchange and

    a queue with a routing key • Exchange type : direct 39
  40. 40

  41. Topics • Receive message selectively but based on a pattern

    • The routing key contains a specific format with special characters • Exchange type : topic 41
  42. 42

  43. Bonus Because I still have time 7

  44. Warning ! • Message queue might be overkill • Using

    an actual implementation is not that easy • Infrastructure setup is not negligible 44
  45. My previous real usage cases • Imports • Exports •

    Bulk actions 45
  46. UNIX power • There are two common message queue implementations

    in UNIX • The first one, System V IPC messages, was done in 1983 • In other words : message queue is not new at all! 46
  47. 47 Thank you Any questions?