Do More With Message Queue

Introducing message queue system, and explain how message queue can be used for queuing tasks. This is especially useful for web application to perform tasks in an asynchronously manner.

Transcript

1. Do More With Message Queue Leong Hean Hong (mrleong.net) 2nd Aug, 2015
2. Objective Optimizing web application using message queue
3. Agenda • Problem • Solution • Message Queue • Queue Systems • Protocols • Benchmark • Choosing Queue System • Example
4. Problem • An author publishes an article on your website • The website needs to ◦ send push notifications to 2000 followers ◦ send email notification to 2000 followers ◦ optimise images in the article ◦ announce it on various (social) media When user clicked ‘Submit’ button, how long does she have to wait?
5. Solution • Show a publish success message to user immediately • Schedule the tasks to be performed later • Perform multiple tasks concurrently
6. Queue your tasks
7. Message Queue
8. Producer • Request for tasks to be performed • Tasks may not be performed immediately upon request
9. Message • Contains information about a task • Information must be understood by task processor (consumer)
10. Message Broker • Manage multiple queues • Receive messages and store in queue(s) • Allow consumer to retrieve messages
11. Consumer • Retrieve message from message broker • Process the task • Producer and consumer might be the same application
12. • RabbitMQ • ActiveMQ (most popular) • SQS (Amazon Simple Queue Service) • ZeroMQ (decentralized, blazing fast) • HornetQ • Apollo • QPID Reference: http://queues.io/ Queue Systems
13. Protocols • AMQP ◦ Advanced Message Queuing Protocol ◦ Most popular • STOMP ◦ Streaming Text Oriented Messaging Protocol
14. Benchmark Reference: http://bit.ly/1MsOu70
15. Choosing Queue System • Persistent / non-persistent • Setup/integration effort • (Community) support • Licensing cost • Maintenance effort/cost • Performance
16. Example Custom made message forwarder: https://github.com/hongster/message-forwarder
17. Example 1. Web app sends a message to RabbitMQ server. a. Message contains worker’s URL in custom AMQP header b. Message content might be ID of posted article 2. We developed an app (Message Forwarder) that constantly a. get message from RabbitMQ server b. forwards message to worker using HTTP POST c. multiple messages are sent concurrently
18. Example • Worker is another web app that do work upon receiving message. ◦ Sometime it is the the original sender web app itself • Worker might be a webservice, and URL is the API endpoint