Share
Embed
Transcript
90 slides
-
Task Producer Consumer Messages Messages Messages WHAT IS A QUEUE?
• Pub/Sub • FIFO buffer • Push / Pull • A way to communicate between applications / systems. • A way to decouple components. • A way to offload work. -
THE INTEROP CASE Request Conditioning Service Response Response Web Service
Call Send Response -
SO WHY QUEUE • User experience • System Security •
Load Distribution • System Reliability -
AMQP • AMQP Working Group (Community and Vendor) • Platform
agnostic protocol. • Completely open, interoperable and broadly applicable. • Many severs available and many client libraries. • Best generally known in RabbitMQ. -
STOMP • Simple protocol • Behaviors follow very simple commands.
• Most message queues can communicate over STOMP. -
Connect Send Disconnect /queue/ msg P H P S T
O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack -
SQS • Simplistic protocol, HTTP- based. • Supports delays, timers,
and multiple policies. • Combine with SNS to implement patterns. -
ZEROMQ • The ultimate in message queue flexibility. • Socket
library that acts as a concurrency framework. -
XMPP • Best for real-time data. • Leveraging pub/sub can
turn it into more of a generic message system. • Multiple libraries available. -
BEANSTALKD • Asynchronous Job Queue • Supports delays • Many
PHP clients exist -
REDIS • Supports pub/sub • Supports atomic lists which people
build queues off of -
YOUR DATABASE • The poor mans queue. • Generally leveraging
memory tables. -
A list of 30+ message queue implementations. NOTE: NOT ALL
ARE "REAL" MESSAGE QUEUES. queues.io -
PULL VS. PUSH • Always PULL, whenever possible in back-end
applications. • Push eliminates several benefits, however, can be useful with web-workers and front-end applications. -
DELIVERY • Is the delivery guaranteed? • If a message
cannot be delivered how it it handled? -
ROUTING • Multiple routing scenarios • Fanout • Direct •
Topic • Broadcast -
BATCHING • Do it later but in bulk (credit card
processing) • Can be done via scheduling (jenkins) -
PUSHING VIA STOMP <?php class UserService { public function save(MyUser
$user) { $this->db->save(MyUser $user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', json_encode([ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ])); } } -
FETCHING VIA STOMP <?php $stomp = new Stomp('tcp://localhost:61613'); $stomp->subscribe('/queue/email'); while
(true) { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body); mail($email->to, $email->subject, $email->message, $email->headers); } -
PUSHING VIA AMQP <?php class UserService { public function save(MyUser
$user) { $this->db->save(MyUser $user); $amqp = new AMQPConnection(); $amqp->connect(); $ex = new AMQPExchange(); $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT); $q = new AMQPQueue($amqp); $q->declare('email'); $ex->bind('email', 'routing.key'); $ex->publish(json_encode([ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]), 'routing.key'); } } -
FETCHING VIA AMQP <?php $amqp = new AMQPConnection(); $amqp->connect(); $ex
= new AMQPExchange(); $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT); $q = new AMQPQueue($amqp); $q->declare('email'); $ex->bind('email', 'routing.key'); while ($messages = $q->consume()) { foreach ($messages as $message) { $email = json_decode($message['message_body']); mail($email->to, $email->subject, $email->message, $email->headers); } } -
WORKER CONSIDERATIONS • Should do ONE thing and ONE thing
well. • Should attempt to be as quick as possible in handling that type. • Should be able to be scaled horizontally. -
HANDLING WORKERS • Prevent Memory Leaks • memory_get_usage • Handle
Signals! • pcntl_signal -
interface QueueInterface { public function __construct(Stomp $stomp, String $queue); public
function dispatch(); public function publish : bool(array $message); public function work(StompFrame $message); } -
class AbstractQueue implements QueueInterface { protected $stomp; protected $queue; protected
$signal; public function __construct(Stomp $stomp, string $queue) { $this->stomp = $stomp; $this->queue = $queue; } protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); } if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } } protected function signal(int $signal) { $this->signal = $signal; } -
public function dispatch() { $this->prepare(); while (true) { if ($this->signal)
{ break ; } if (!$this->stomp->hasFrame()) { $this->wait(); continue ; } $frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); } $this->stomp->ack($frame); } } protected function wait() { sleep(1); } protected function validate(StompFrame $message): bool { return false; } public function publish(array $message): bool { return $this->stomp->send($this->queue, json_encode($message)); } -
class EmailQueue extends AbstractQueue { public function validate(StompFrame $message): bool
{ if (!array_key_exists('to', $message)) { return false; } return true; } public function work(StompFrame $message) { $mail = json_decode($message); mail($mail->to, $mail->subject, $mail->message); } } -
<?php declare(ticks=1); include 'vendor/autoload.php'; $app = Zend\Mvc\Application::init(include 'config/application.config.php'); $sm =
$app->getServiceManager(); if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker <name>\n\n"); exit(1); } $name = $argv[1]; try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (\Exception $e) { fprintf(STDERR, "%s\n", $msg); exit(1); } $consumer = null; echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0); -
SERVICES TRIGGER EVENTS use Zend\EventManager\EventManagerAwareTrait; class UserService { use EventManagerAwareTrait;
public function save(MyUser $user) { $this->db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } } -
ATTACH EVENTS use Zend\ServiceManager\ServiceManager; $sm = new ServiceManager(); $service =
$sm->get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish(json_encode[ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); }); -
HANDLING PROGRESS UPDATES • Keep track of item state using
a cache server or database. • Utilize events on your processing to provide updates to the UI. -
HOW MIGHT WE UPDATE? $message = json_decode($stomp->readFrame()->body); $proc = proc_open($command,
$descriptorSpec, $pipes); stream_set_blocking($pipes[1], 0); while (!feof($pipes[1])) { $content = fgets($pipes[1], 1024); if (preg_match('/something-i-(care)-about/', $content, $results)) { // provide update to DB, cache, etc. } } -
JENKINS: AMAZON EC2 • Install the Amazon EC2 Plugin •
Leverage images and use spot instances • Configure spot instance to deploy latest code on boot • Configure jenkins to boot up additional spot instances if jobs are waiting. -
JENKINS: JOBS • Handlers • 1 job per handler with:
• Execute concurrent builds if necessary • Restrict where this project can be run (spot instance label). • Scheduled job to watch queue lengths and available workers • queue length > available workers after x time = execute job • queue length < available workers after x time = kill job -
SYSTEMD • Integrated into the OS. • Monitors and handles
starting, restarting and dependencies of processes. • Integrated logging. -
SERVICE FILE EXAMPLE # /etc/systemd/system/my_worker.service [Unit] Description=My Worker Service After=network.target
[Service] Type=simple User=unix-user WorkingDirectory=/path/to/working/dir ExecStart=/path/to/worker --opt1 Restart=on-failure # or always, on-abort, etc [Install] WantedBy=multi-user.target -
HOW TO USE IT? # controlling systemctl start|stop|status|enable| disable my-worker.service
# logs journalctl -f -u my-worker.service -
SUPERVISOR • Daemon that runs on the server. • Monitors
programs and keeps them running in case of failure. • Handles logging. • If systemd is unavailable or a bit too scary :) -
EXAMPLE PROGRAM CONFIGURATION [program:emailworker] command=/usr/bin/php /var/www/worker "MyProject\Queue\Email" process_name=%(program_name)s_%(process_num)d numprocs=2 numprocs_start=2
user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15 -
QUEUE BACKUP • Queue length continues to grow and you're
not processing enough...