Message Queues : A
Primer
Mike Willbanks | Barnes & Noble
Slide 2
Slide 2 text
2
• Talk
Slides will be posted after the talk.
• Me
Sr. Web Architect Manager at NOOK Developer
Open Source Contributor
Where you can find me:
• Twitter: mwillbanks G+: Mike Willbanks
• IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com
• GitHub: https://github.com/mwillbanks
Housekeeping…
Slide 3
Slide 3 text
3
• Message Queues?
• Protocols and Software
• Picking a Message Queue
• Best Practices
Agenda
Slide 4
Slide 4 text
Message Queues?
Slide 5
Slide 5 text
5
A Definition
“Message queues and mailboxes are software-engineering
components used for interprocess communication, or for inter-
thread communication within the same process. They use a
queue for messaging – the passing of control or of content.”
Slide 6
Slide 6 text
6
What is messaging?
“Messaging describes the sending and receiving of data (in the
form of messages) between systems. Messages are exchanged
between programs or applications, similar to the way people
communicate by email but with guarantees on delivery, speed,
security and the absence of spam.”
Slide 7
Slide 7 text
7
General Anatomy
Task
Producer Consumer
Messages
Messages
Messages
Producer creates a message and pushes it to the queue; the
consumer reads from the queue and processes the message.
Slide 8
Slide 8 text
8
l
Pub/Sub
l
FIFO buffer
l
Push / Pull
l
A way to communicate between applications / systems.
l
A way to decouple components.
l
A way to offload work.
Describing Message Queues
Slide 9
Slide 9 text
9
l
Offload Heavy Work
l
Integration with Legacy Systems
l
Asynchronous Processing
l
Parallel Processing
l
Process consistency
l
Scalability
Why to use a Message Queue
Slide 10
Slide 10 text
10
l
“Write programs that work together.”
l
“Do it in the background.”
Unix Foundations
Slide 11
Slide 11 text
11
Why it matters
l
Web systems need to be geared to run things
asynchronously.
l
Distribution of load
l
System integrity
Slide 12
Slide 12 text
12
Enterprise Integration Patterns
Understanding messaging patterns will greatly
assist you in building out great systems for
messaging.
http://www.eaipatterns.com/
Slide 13
Slide 13 text
Common Messaging Patterns
Slide 14
Slide 14 text
14
Publish-Subscribe
Slide 15
Slide 15 text
15
Request-Reply
Slide 16
Slide 16 text
16
Message Router
Slide 17
Slide 17 text
Message queue examples
• You’ve seen them before; they are used in most
applications to help them scale.
Slide 18
Slide 18 text
18
Slide 19
Slide 19 text
19
Slide 20
Slide 20 text
20
Slide 21
Slide 21 text
Generic Use Cases
• When to make use of message queues in case you’re
wondering.
24
Video Processing
Resampling, Audio Overlay, Type Conversion…
Slide 25
Slide 25 text
25
Analytics
Web Server Logs, Log Aggregation, PHP Errors, etc.
Slide 26
Slide 26 text
26
Integrations
Save local first; push second.
Slide 27
Slide 27 text
Protocols
• AMQP
• STOMP
• XMPP
• Vendor Specific
Slide 28
Slide 28 text
28
AMQP
Advanced Message Queuing Protocol
Slide 29
Slide 29 text
29
l
AMQP Working Group (Community and
Vendor)
l
Platform agnostic protocol.
l
Completely open, interoperable and broadly
applicable.
l
Many severs available and many client
libraries.
Overview of AMQP
Slide 30
Slide 30 text
30
How it Works
Producer, Exchange, Queue, Consumer
Slide 31
Slide 31 text
31
l
AMQP utilizes exchanges, queues and
bindings.
l
An exchange are routers with routing tables.
l
A binding defines the routing rules.
l
A queue is where the messages wait for a
consumer.
How it Works
Slide 32
Slide 32 text
32
Exchanges
Slide 33
Slide 33 text
33
l
Fanout Exchange
l
No routing keys involved. Any message that is sent to the
exchange is sent to all queues bound to that exchange.
l
Direct Exchange
l
Routing keys involved. A queue binds to the exchange to
request messages that match a routing key exactly.
l
Topic Exchange
l
Routing keys involved. A queue binds to the exchange to
request messages that match a routing key pattern.
Understanding Exchanges
Slide 34
Slide 34 text
34
Implementations
www.rabbitmq.com
Very popular and common message queue owned by
VMware.
qpid.apache.org
Long standing project; apache foundation.
www.openamq.org
Long standing project; ZeroMQ partner, no news since
2009.
Slide 35
Slide 35 text
35
l
An exchange, queue and bindings must be defined
first. Publishing can then commence after.
l
Create the queue
l
Create the exchange
l
Bind to the queue.
Building a Queue
Slide 36
Slide 36 text
36
l
Default behavior is no persistence.
l
How important are the messages?
l
Just about all items have a level of persistence if you
would like them to survive on reboot.
l
Mark exchanges, queues and messages as DURABLE.
Persistence?
40
STOMP
Simple (or Streaming) Text Orientated Messaging Protocol
Slide 41
Slide 41 text
41
Overview
l
Simple protocol
l
Behaviors follow very simple commands.
l
Most message queues can communicate over
STOMP.
Slide 42
Slide 42 text
42
How It Works
l
When you send in a message, you tell it which queue to
go to.
l
When you subscribe you request a queue.
Connect Send Disconnect
/queue/
msg
P
H
P
S
T
O
M
P
S
E
R
V
E
R
Connect Subscribe Disconnect
/queue/
msg
Read
Ack
Slide 43
Slide 43 text
43
Sever Implementations
activemq.apache.org
One of the oldest message queues existing;
a apache foundation project
activemq.apache.org/apollo
Next generation ActiveMQ
www.rabbitmq.com
Very popular and common message queue owned by
VMware.
www.jboss.org/hornetq
Supported by Red Hat Middle Ware division, picking
up steam.
Slide 44
Slide 44 text
44
l
pecl install stomp
l
That was easy J
PECL Stomp
Slide 45
Slide 45 text
45
STOMP Client
Slide 46
Slide 46 text
46
STOMP Worker
Slide 47
Slide 47 text
47
XMPP
Extensible Messaging and Presence Protocol
(Although not really a “Message Queue”)
Slide 48
Slide 48 text
48
Overview
l
Best for real-time data.
l
Leveraging pub/sub can turn it into more of a generic message
system.
l
Multiple libraries
l
JAXL - https://github.com/abhinavsingh/JAXL
l
Xmpp - https://github.com/alexmace/Xmpp
Slide 49
Slide 49 text
49
XEP-0060: Publish-Subscribe
l
Specification for implementing Publish Subscribe
models.
l
Extension to the original XMPP specification.
Publish
Subscribe
Sub1 Sub2 Sub3
to, id,
message
from, to, id,
message
Slide 50
Slide 50 text
50
Publish
Slide 51
Slide 51 text
51
Message
Slide 52
Slide 52 text
Special Message Queues
• Various others; including Gearman, ZeroMQ, etc.
Slide 53
Slide 53 text
53
l
There are job servers available that are more
flexible or more specific.
l
Extreme Flexibility
l
Job Severs
l
Cloud Messaging
Overview
Slide 54
Slide 54 text
54
ZeroMQ
The ultimate in message queue flexibility.
Socket library that acts as a concurrency framework.
Contains a PHP extension.
Slide 55
Slide 55 text
55
Several Types of Queues
Request / Reply
Publish / Subscribe
Parallel Pipeline
Fair Queuing
Slide 56
Slide 56 text
56
ZeroMQ Client Example
Slide 57
Slide 57 text
57
ZeroMQ Worker Example
Slide 58
Slide 58 text
58
l
Application framework for farming out work.
l
Job sever for asynchronous or synchronous messages.
Gearman
Slide 59
Slide 59 text
59
l
Pass a job to the job server
l
Worker receives the job and processes
l
Ability to persist if enabled; default is in-memory.
Gearman
Slide 60
Slide 60 text
60
Gearman Client Example
Slide 61
Slide 61 text
61
Gearman Worker Example
Slide 62
Slide 62 text
62
l
Asynchronous Job Queue
l
Good scheduling system (aka delays).
l
User land PHP clients.
Beanstalkd
Slide 63
Slide 63 text
63
l
Pass a job to the job server
l
Worker receives the job and processes
l
Ability to persist to binlog; default is in-memory
Beanstalkd
Application Code
Beanstalkd Client API
(Pheanstalk)
Beanstalkd Server
Beanstalkd Worker API
(Pheanstalk)
Worker Application Code
Slide 64
Slide 64 text
64
Pheanstalk Client
Slide 65
Slide 65 text
65
Pheanstalk Worker
Slide 66
Slide 66 text
Picking a Message Queue
• What you want to look for.
68
Standards
• A recognized standard?
AMQP? STOMP? XMPP?
• How many developers?
Will an unfortunate event kill off the product?
• Undocumented protocol?
Forget about it!
Slide 69
Slide 69 text
69
Delivery Policies
Store Message Saved?
Process Message ACK?
Receive Message TTL?
Slide 70
Slide 70 text
70
Message Policies
• Handling in-active messages?
• Messages that failed processing?
• Time to live?
• Retry?
• Ability to check message status?
74
Messages
• Formatting
JSON or XML are great options.
• Please no serialized PHP objects.
• Message Size
Only as large as necessary.
• Don’t send a binary object through the queue.
Slide 75
Slide 75 text
75
Workers
• Dumb as possible!
Single message type
• Easier to scale.
Single operation
• Easy to debug and far more flexible.
Slide 76
Slide 76 text
76
PHP Daemons
• Prevent Memory Leaks
Detection
Cycle Workers
• Handle Signals
Properly shutdown!
Watch out for OS service kills
• Sleeping is good J
Slide 77
Slide 77 text
77
Slide 78
Slide 78 text
78
Environment Integration
Web Server
Database
Worker Server
Message Queue
Web Server -> Message Queue
Worker Server -> Message Queue
Slide 79
Slide 79 text
79
High Availability
Web Server
Database
Worker Server
Message Queue
Message Queue
Worker Server
Send to 1
Insert multiple message queue servers + multiple worker
nodes. Each worker node can connect to as many message
queue servers as necessary.
Slide 80
Slide 80 text
80
Supervisord
Process monitoring, logging and more!