Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Do More With Message Queue

Do More With Message Queue

Introducing message queue system, and explain how message queue can be used for queuing tasks. This is especially useful for web application to perform tasks in an asynchronously manner.

Leong Hean Hong

August 01, 2015
Tweet

More Decks by Leong Hean Hong

Other Decks in Programming

Transcript

  1. Agenda • Problem • Solution • Message Queue • Queue

    Systems • Protocols • Benchmark • Choosing Queue System • Example
  2. Problem • An author publishes an article on your website

    • The website needs to ◦ send push notifications to 2000 followers ◦ send email notification to 2000 followers ◦ optimise images in the article ◦ announce it on various (social) media When user clicked ‘Submit’ button, how long does she have to wait?
  3. Solution • Show a publish success message to user immediately

    • Schedule the tasks to be performed later • Perform multiple tasks concurrently
  4. Producer • Request for tasks to be performed • Tasks

    may not be performed immediately upon request
  5. Message • Contains information about a task • Information must

    be understood by task processor (consumer)
  6. Message Broker • Manage multiple queues • Receive messages and

    store in queue(s) • Allow consumer to retrieve messages
  7. Consumer • Retrieve message from message broker • Process the

    task • Producer and consumer might be the same application
  8. • RabbitMQ • ActiveMQ (most popular) • SQS (Amazon Simple

    Queue Service) • ZeroMQ (decentralized, blazing fast) • HornetQ • Apollo • QPID Reference: http://queues.io/ Queue Systems
  9. Protocols • AMQP ◦ Advanced Message Queuing Protocol ◦ Most

    popular • STOMP ◦ Streaming Text Oriented Messaging Protocol
  10. Choosing Queue System • Persistent / non-persistent • Setup/integration effort

    • (Community) support • Licensing cost • Maintenance effort/cost • Performance
  11. Example 1. Web app sends a message to RabbitMQ server.

    a. Message contains worker’s URL in custom AMQP header b. Message content might be ID of posted article 2. We developed an app (Message Forwarder) that constantly a. get message from RabbitMQ server b. forwards message to worker using HTTP POST c. multiple messages are sent concurrently
  12. Example • Worker is another web app that do work

    upon receiving message. ◦ Sometime it is the the original sender web app itself • Worker might be a webservice, and URL is the API endpoint