Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Message queue overview

Message queue overview

Thomas Calvet

November 16, 2018
Tweet

More Decks by Thomas Calvet

Other Decks in Programming

Transcript

  1. www.leblogdefrancis.com • A blog of articles • Registered users can

    start a thread on an article • When an user reply on a thread, a notification must be sent to every participants of this thread 3
  2. Straightforward implementation 1. Get all the users that already commented

    on the thread 2. Iterate on them 3. Check if they have notifications enabled 4. An infinity of possible checks actually... 5. Send the notifications 4
  3. Main issues with it • Bad for performance • Not

    decoupled • Not scalable in an efficient way • Bad for exceptions handling • Less reliability • Not flexible • Is it essential to send the notifications there ? 5
  4. Message queue! • Something sends a message • The message

    ends up in a queue • Something process the message 11
  5. A common language • A producer publishes a message •

    A message broker accepts and forwards a message • A consumer consumes / receives a message 13
  6. 14

  7. The message • A message is some data • It

    should encapsulate everything consumers might need • In our example use case : ◦ The id / uuid of the reply ◦ The content of the notification and the query to find the recipients 15
  8. The producer • A program that sends the message •

    In our example use case : ◦ Our application ◦ In a dedicated service ◦ In an event listener 16
  9. The message broker • An intermediary program that handle the

    message between the producer and the consumer • Objective : decoupling the awareness between producers and consumers • In our example use case : ◦ An existing one 17
  10. The consumer • A program that waits to receive the

    message • This program logically does the expensive task • In our example use case : ◦ Our application ◦ In a dedicated service ◦ Another application ◦ In another language ◦ On another server 18
  11. 19

  12. The queue • Inside the message broker • Managed by

    the message broker • FIFO (First In First Out) 20
  13. Performant and decoupled • Performance : ◦ Publishing a message

    takes a few milliseconds ◦ Sending the notifications does not block the response • Decoupled : ◦ Not perfect in our example but : ◦ The publish and consume parts are totally separated ◦ So extracting the consumer would be way more easier 22
  14. Scalable and recoverable • Scalable : ◦ We just have

    to spawn new instances of our consumer • Better exceptions handling : ◦ Because of the retry mechanism ◦ And the dead letter exchange mechanism ◦ By splitting one message into several small ones 23
  15. Reliable and flexible • Reliable : ◦ If our application

    (producer) is down, all the currently waiting messages can still be processed • Flexible : ◦ We can replace our current consumer implementation ◦ We can add a new behavior with a new consumer 24
  16. Standards / protocols • AMQP (Asynchronous Message Queue Protocol) •

    MQTT (Message Queuing Telemetry Transport) • STOMP (Streaming Text Oriented Messaging Protocol) • And more... 26
  17. Message queuing services • Amazon Web Services SQS (Simple Queue

    Service) • Google Cloud Platform Pub/Sub • Microsoft Azure Storage queues & Service Bus queues • And more... 29
  18. Quick focus on RabbitMQ • Open Source message broker, originally

    for the AMQP protocol • The most widely deployed Open Source message broker in the world • Written in Erlang, a concurrent and functional programming language 31
  19. Mature system • TLS / SSL support • Clustering •

    Authentication and authorisation mechanisms • And more... 32
  20. The exchange • The producer actually sends the message to

    an exchange • The exchange knows what to do with the message (forwards it to one or several queues, or ignore it) • 4 Different types : direct, topic, headers and fanout 33
  21. 34

  22. Work queue • Most simple case like our example •

    Delay an expensive task • Nameless exchange “” 35
  23. 36

  24. Publish / Subscribe • The same message for several consumers

    • Exchange type : fanout • In our example : ◦ We want to send an email in addition to the notification ◦ We just need to create a new consumer 37
  25. 38

  26. Routing • Receive messages selectively • Bind an exchange and

    a queue with a routing key • Exchange type : direct 39
  27. 40

  28. Topics • Receive message selectively but based on a pattern

    • The routing key contains a specific format with special characters • Exchange type : topic 41
  29. 42

  30. Warning ! • Message queue might be overkill • Using

    an actual implementation is not that easy • Infrastructure setup is not negligible 44
  31. UNIX power • There are two common message queue implementations

    in UNIX • The first one, System V IPC messages, was done in 1983 • In other words : message queue is not new at all! 46