Queue It - Speaker Deck

Slide 1

Slide 1 text

QUEUE IT!

Slide 2

Slide 2 text

Task Producer Consumer Messages Messages Messages WHAT IS A QUEUE? • Pub/Sub • FIFO buffer • Push / Pull • A way to communicate between applications / systems. • A way to decouple components. • A way to ofﬂoad work.

Slide 3

Slide 3 text

WHY QUEUE?

Slide 4

Slide 4 text

THE LONG RUNNING CASE Request Long-process Send Response Response

Slide 5

Slide 5 text

THE INTEROP CASE Request Conditioning Service Response Response Web Service Call Send Response

Slide 6

Slide 6 text

NOT YOUR DATABASE Request Process Response Send Response DB

Slide 7

Slide 7 text

IOT Data S3 Possible Action Process Event

Slide 8

Slide 8 text

SO WHY QUEUE • User experience • System Security • Load Distribution • System Reliability

Slide 9

Slide 9 text

IN PRACTICE You’ve seen this before…

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

WHAT YOU MIGHT QUEUE

Slide 14

Slide 14 text

COMMUNICATIONS • Emails • SMS • Push Notiﬁcations

Slide 15

Slide 15 text

IMAGES • Conversions • Resize • Thumbnail • Watermark

Slide 16

Slide 16 text

VIDEO • Conversion • Resampling • Audio overlay

Slide 17

Slide 17 text

IOT • Receive messages from devices and process responses

Slide 18

Slide 18 text

PATTERNS

Slide 19

Slide 19 text

POINT TO POINT Point to Point Channel Receiver Sender

Slide 20

Slide 20 text

PUBLISH / SUBSCRIBE Publiser Subscribe Channel Subscriber Subscriber Subscriber

Slide 21

Slide 21 text

MESSAGE BUS Application Application Application Message Bus

Slide 22

Slide 22 text

PIPELINE Sender Receiver Point to Point Channel Receiver Point to Point Channel

Slide 23

Slide 23 text

INVALID MESSAGE Channel Receiver Sender X Invalid Message Channel

Slide 24

Slide 24 text

PROTOCOLS Or implementations for that matter.

Slide 25

Slide 25 text

AMQP • AMQP Working Group (Community and Vendor) • Platform agnostic protocol. • Completely open, interoperable and broadly applicable. • Many severs available and many client libraries. • Best generally known in RabbitMQ.

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

KAFKA • Works with "streams" of data. • Limited messaging patterns.

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

STOMP • Simple protocol • Behaviors follow very simple commands. • Most message queues can communicate over STOMP.

Slide 30

Slide 30 text

Connect Send Disconnect /queue/ msg P H P S T O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack

Slide 31

Slide 31 text

SQS • Simplistic protocol, HTTP- based. • Supports delays, timers, and multiple policies. • Combine with SNS to implement patterns.

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

SPECIAL PURPOSE Many queue implementations exist that don’t necessarily sit under standards…

Slide 34

Slide 34 text

ZEROMQ • The ultimate in message queue ﬂexibility. • Socket library that acts as a concurrency framework.

Slide 35

Slide 35 text

SOCKET.IO • Real-time bidirectional event-based communication • Largely leverages pub/sub

Slide 36

Slide 36 text

XMPP • Best for real-time data. • Leveraging pub/sub can turn it into more of a generic message system. • Multiple libraries available.

Slide 37

Slide 37 text

BEANSTALKD • Asynchronous Job Queue • Supports delays • Many PHP clients exist

Slide 38

Slide 38 text

REDIS • Supports pub/sub • Supports atomic lists which people build queues off of

Slide 39

Slide 39 text

YOUR DATABASE • The poor mans queue. • Generally leveraging memory tables.

Slide 40

Slide 40 text

A list of 30+ message queue implementations. NOTE: NOT ALL ARE "REAL" MESSAGE QUEUES. queues.io

Slide 41

Slide 41 text

CONSIDERATIONS How do we evaluate our options…

Slide 42

Slide 42 text

PULL VS. PUSH • Always PULL, whenever possible in back-end applications. • Push eliminates several beneﬁts, however, can be useful with web-workers and front-end applications.

Slide 43

Slide 43 text

DURABILITY • Memory residence • Persistence • Restart survival

Slide 44

Slide 44 text

SECURITY • Authentication • Queue permissions / restrictions

Slide 45

Slide 45 text

DELIVERY • Is the delivery guaranteed? • If a message cannot be delivered how it it handled?

Slide 46

Slide 46 text

ROUTING • Multiple routing scenarios • Fanout • Direct • Topic • Broadcast

Slide 47

Slide 47 text

BATCHING • Do it later but in bulk (credit card processing) • Can be done via scheduling (jenkins)

Slide 48

Slide 48 text

RECEIPT • Do you get an acknowledgement of receipt?

Slide 49

Slide 49 text

IMPLEMENTING QUEUES

Slide 50

Slide 50 text

STARTING POINTS

Slide 51

Slide 51 text

PUSHING VIA STOMP db->save(MyUser $user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', json_encode([ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ])); } }

Slide 52

Slide 52 text

FETCHING VIA STOMP subscribe('/queue/email'); while (true) { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body); mail($email->to, $email->subject, $email->message, $email->headers); }

Slide 53

Slide 53 text

PUSHING VIA AMQP db->save(MyUser $user); $amqp = new AMQPConnection(); $amqp->connect(); $ex = new AMQPExchange(); $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT); $q = new AMQPQueue($amqp); $q->declare('email'); $ex->bind('email', 'routing.key'); $ex->publish(json_encode([ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]), 'routing.key'); } }

Slide 54

Slide 54 text

FETCHING VIA AMQP connect(); $ex = new AMQPExchange(); $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT); $q = new AMQPQueue($amqp); $q->declare('email'); $ex->bind('email', 'routing.key'); while ($messages = $q->consume()) { foreach ($messages as $message) { $email = json_decode($message['message_body']); mail($email->to, $email->subject, $email->message, $email->headers); } }

Slide 55

Slide 55 text

MESSAGES

Slide 56

Slide 56 text

MESSAGE CONSIDERATIONS • Message Format • Message Contents

Slide 57

Slide 57 text

SERIALIZE O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s: 3:"baz";}}}}

Slide 58

Slide 58 text

WORKERS

Slide 59

Slide 59 text

WORKER CONSIDERATIONS • Should do ONE thing and ONE thing well. • Should attempt to be as quick as possible in handling that type. • Should be able to be scaled horizontally.

Slide 60

Slide 60 text

HANDLING WORKERS • Prevent Memory Leaks • memory_get_usage • Handle Signals! • pcntl_signal

Slide 61

Slide 61 text

ABSTRACTIONS We want to make this easy…

Slide 62

Slide 62 text

SYMFONY MESSENGER

Slide 63

Slide 63 text

LARAVEL QUEUES

Slide 64

Slide 64 text

BERNARD General queue implementation

Slide 65

Slide 65 text

interface QueueInterface { public function __construct(Stomp $stomp, String $queue); public function dispatch(); public function publish : bool(array $message); public function work(StompFrame $message); }

Slide 66

Slide 66 text

class AbstractQueue implements QueueInterface { protected $stomp; protected $queue; protected $signal; public function __construct(Stomp $stomp, string $queue) { $this->stomp = $stomp; $this->queue = $queue; } protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); } if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } } protected function signal(int $signal) { $this->signal = $signal; }

Slide 67

Slide 67 text

public function dispatch() { $this->prepare(); while (true) { if ($this->signal) { break ; } if (!$this->stomp->hasFrame()) { $this->wait(); continue ; } $frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); } $this->stomp->ack($frame); } } protected function wait() { sleep(1); } protected function validate(StompFrame $message): bool { return false; } public function publish(array $message): bool { return $this->stomp->send($this->queue, json_encode($message)); }

Slide 68

Slide 68 text

class EmailQueue extends AbstractQueue { public function validate(StompFrame $message): bool { if (!array_key_exists('to', $message)) { return false; } return true; } public function work(StompFrame $message) { $mail = json_decode($message); mail($mail->to, $mail->subject, $mail->message); } }

Slide 69

Slide 69 text

BOOTSTRAPPING Leverage your existing infrastructure as much as possible!

Slide 70

Slide 70 text

getServiceManager(); if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker \n\n"); exit(1); } $name = $argv[1]; try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (\Exception $e) { fprintf(STDERR, "%s\n", $msg); exit(1); } $consumer = null; echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0);

Slide 71

Slide 71 text

EVENTS

Slide 72

Slide 72 text

SERVICES TRIGGER EVENTS use Zend\EventManager\EventManagerAwareTrait; class UserService { use EventManagerAwareTrait; public function save(MyUser $user) { $this->db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } }

Slide 73

Slide 73 text

ATTACH EVENTS use Zend\ServiceManager\ServiceManager; $sm = new ServiceManager(); $service = $sm->get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish(json_encode[ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); });

Slide 74

Slide 74 text

HANDLING PROGRESS UPDATES • Keep track of item state using a cache server or database. • Utilize events on your processing to provide updates to the UI.

Slide 75

Slide 75 text

HOW MIGHT WE UPDATE? $message = json_decode($stomp->readFrame()->body); $proc = proc_open($command, $descriptorSpec, $pipes); stream_set_blocking($pipes[1], 0); while (!feof($pipes[1])) { $content = fgets($pipes[1], 1024); if (preg_match('/something-i-(care)-about/', $content, $results)) { // provide update to DB, cache, etc. } }

Slide 76

Slide 76 text

TOOLING

Slide 77

Slide 77 text

JENKINS • It's not just for your CI/CD projects.

Slide 78

Slide 78 text

JENKINS: AMAZON EC2 • Install the Amazon EC2 Plugin • Leverage images and use spot instances • Conﬁgure spot instance to deploy latest code on boot • Conﬁgure jenkins to boot up additional spot instances if jobs are waiting.

Slide 79

Slide 79 text

JENKINS: JOBS • Handlers • 1 job per handler with: • Execute concurrent builds if necessary • Restrict where this project can be run (spot instance label). • Scheduled job to watch queue lengths and available workers • queue length > available workers after x time = execute job • queue length < available workers after x time = kill job

Slide 80

Slide 80 text

SYSTEMD • Integrated into the OS. • Monitors and handles starting, restarting and dependencies of processes. • Integrated logging.

Slide 81

Slide 81 text

SERVICE FILE EXAMPLE # /etc/systemd/system/my_worker.service [Unit] Description=My Worker Service After=network.target [Service] Type=simple User=unix-user WorkingDirectory=/path/to/working/dir ExecStart=/path/to/worker --opt1 Restart=on-failure # or always, on-abort, etc [Install] WantedBy=multi-user.target

Slide 82

Slide 82 text

HOW TO USE IT? # controlling  systemctl start|stop|status|enable| disable my-worker.service    # logs  journalctl -f -u my-worker.service

Slide 83

Slide 83 text

SUPERVISOR • Daemon that runs on the server. • Monitors programs and keeps them running in case of failure. • Handles logging. • If systemd is unavailable or a bit too scary :)

Slide 84

Slide 84 text

PAINLESS INSTALLATION sudo easy_install supervisor sudo echo_supervisord_conf > /etc/supervisord.conf sudo service supervisor start

Slide 85

Slide 85 text

EXAMPLE PROGRAM CONFIGURATION [program:emailworker] command=/usr/bin/php /var/www/worker "MyProject\Queue\Email" process_name=%(program_name)s_%(process_num)d numprocs=2 numprocs_start=2 user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15

Slide 86

Slide 86 text

SUPERVISORD MULTI SERVER MONITORING TOOL https://github.com/mlazarov/supervisord-monitor

Slide 87

Slide 87 text

WHEN BAD THINGS HAPPEN

Slide 88

Slide 88 text

QUEUE BACKUP • Queue length continues to grow and you're not processing enough...