Tweet
Tweet

Slide 1

Slide 1 text

DO YOU QUEUE? ZendCon 2015

Slide 2

Slide 2 text

I AM MIKE WILLBANKS • Father, Husband, Developer • VP of Development at Packet Power • Twitter: @mwillbanks

Slide 3

Slide 3 text

Task Producer Consumer Messages Messages Messages WHAT IS A QUEUE? • Pub/Sub • FIFO buffer • Push / Pull • A way to communicate between applications / systems. • A way to decouple components. • A way to offload work.

Slide 4

Slide 4 text

WHY QUEUE?

Slide 5

Slide 5 text

THE LONG RUNNING CASE Request Long-process Send Response Response

Slide 6

Slide 6 text

THE INTEROP CASE Request Conditioning Service Response Response Web Service Call Send Response

Slide 7

Slide 7 text

NOT YOUR DATABASE Request Process Response Send Response DB

Slide 8

Slide 8 text

SO WHY QUEUE • User experience • System Security • Load Distribution • System Reliability

Slide 9

Slide 9 text

IN PRACTICE You’ve seen this before…

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

WHAT YOU MIGHT QUEUE

Slide 14

Slide 14 text

COMMUNICATIONS • Emails • SMS • Push Notifications

Slide 15

Slide 15 text

IMAGES • Conversions • Resize • Thumbnail • Watermark

Slide 16

Slide 16 text

VIDEO • Conversion • Resampling • Audio overlay

Slide 17

Slide 17 text

IOT • Receive messages from devices and process responses

Slide 18

Slide 18 text

PATTERNS

Slide 19

Slide 19 text

POINT TO POINT Point to Point Channel Receiver Sender

Slide 20

Slide 20 text

PUBLISH / SUBSCRIBE Publiser Subscribe Channel Subscriber Subscriber Subscriber

Slide 21

Slide 21 text

MESSAGE BUS Application Application Application Message Bus

Slide 22

Slide 22 text

PIPELINE Sender Receiver Point to Point Channel Receiver Point to Point Channel

Slide 23

Slide 23 text

INVALID MESSAGE Channel Receiver Sender X Invalid Message Channel

Slide 24

Slide 24 text

PROTOCOLS Or implementations for that matter.

Slide 25

Slide 25 text

AMQP • AMQP Working Group (Community and Vendor) • Platform agnostic protocol. • Completely open, interoperable and broadly applicable. • Many severs available and many client libraries.

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

STOMP • Simple protocol • Behaviors follow very simple commands. • Most message queues can communicate over STOMP.

Slide 28

Slide 28 text

Connect Send Disconnect /queue/ msg P H P S T O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack

Slide 29

Slide 29 text

SQS • Fairly simple protocol • Supports delays, timers, and multiple policies.

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

SPECIAL PURPOSE Many queue implementations exist that don’t necessarily sit under standards…

Slide 32

Slide 32 text

XMPP • Best for real-time data. • Leveraging pub/sub can turn it into more of a generic message system. • Multiple libraries available.

Slide 33

Slide 33 text

SOCKET.IO • New comer • Real-time bidirectional event-based communication • Largely leverages pub/sub

Slide 34

Slide 34 text

ZEROMQ • The ultimate in message queue flexibility. • Socket library that acts as a concurrency framework.

Slide 35

Slide 35 text

GEARMAN • Application framework for farming out work. • Job sever for asynchronous or synchronous messages.

Slide 36

Slide 36 text

BEANSTALKD • Asynchronous Job Queue • Supports delays • Many PHP clients exist

Slide 37

Slide 37 text

CONSIDERATIONS How do we evaluate our options…

Slide 38

Slide 38 text

PULL VS. PUSH • Always PULL, whenever possible. • Push eliminates several benefits.

Slide 39

Slide 39 text

DURABILITY • Memory residence • Persistence • Restart survival

Slide 40

Slide 40 text

SECURITY • Authentication • Queue permissions / restrictions

Slide 41

Slide 41 text

DELIVERY • Is the delivery guaranteed? • If a message cannot be delivered how it it handled?

Slide 42

Slide 42 text

ROUTING • Multiple routing scenarios • Fanout • Direct • Topic • Broadcast

Slide 43

Slide 43 text

BATCHING • Do it later but in bulk (credit card processing) • Can be done via scheduling (cron)

Slide 44

Slide 44 text

RECEIPT • Do you get an acknowledgement of receipt?

Slide 45

Slide 45 text

IMPLEMENTING QUEUES

Slide 46

Slide 46 text

STARTING POINTS

Slide 47

Slide 47 text

PUSHING MESSAGES db->save($user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', [ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); } }

Slide 48

Slide 48 text

HANDLING MESSAGES subscribe('/queue/email'); while (true) { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body); mail($email->to, $email->subject, $email->message, $email->headers); }

Slide 49

Slide 49 text

MESSAGES

Slide 50

Slide 50 text

MESSAGE CONSIDERATIONS • Message Format • Message Contents

Slide 51

Slide 51 text

SERIALIZE O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s: 3:"baz";}}}}

Slide 52

Slide 52 text

WORKERS

Slide 53

Slide 53 text

WORKER CONSIDERATIONS • Should do ONE thing and ONE thing well. • Should attempt to be as quick as possible in handling that type. • Should be able to be scaled horizontally.

Slide 54

Slide 54 text

HANDLING WORKERS • Prevent Memory Leaks • memory_get_usage • Handle Signals! • pcntl_signal

Slide 55

Slide 55 text

ABSTRACTIONS We want to make this easy…

Slide 56

Slide 56 text

Slide 57

Slide 57 text

stomp = $stomp; $this->queue = $queue; } protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); } if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } } protected function signal($signal) { $this->signal = $signal; }

Slide 58

Slide 58 text

public function dispatch() { $this->prepare(); while (true) { if ($this->signal) { break ; } if (!$this->stomp->hasFrame()) { $this->wait(); continue ; } $frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); } $this->stomp->ack($frame); } } protected function wait() { sleep(1); } protected function validate(StompFrame $message) { return false; } public function publish(array $message) { return $this->stomp->send($this->queue, json_encode($message)); }

Slide 59

Slide 59 text

to, $mail->subject, $mail->message); } }

Slide 60

Slide 60 text

BOOTSTRAPPING Leverage your existing infrastructure as much as possible!

Slide 61

Slide 61 text

getServiceManager(); if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker \n\n"); exit(1); } $name = $argv[1]; try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (\Exception $e) { fprintf(STDERR, "%s\n", $msg); exit(1); } $consumer = null; echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0);

Slide 62

Slide 62 text

EVENTS

Slide 63

Slide 63 text

SERVICES TRIGGER EVENTS db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } }

Slide 64

Slide 64 text

ATTACH EVENTS get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish([ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); });

Slide 65

Slide 65 text

HANDLING PROGRESS • Keep track by using a generic handler • Or, keep track via your database.

Slide 66

Slide 66 text

TOOLING

Slide 67

Slide 67 text

SUPERVISOR • Daemon that runs on the server. • Monitors programs and keeps them running in case of failure. • Handles logging.

Slide 68

Slide 68 text

PAINLESS INSTALLATION sudo easy_install supervisor sudo echo_supervisord_conf > /etc/supervisord.conf sudo service supervisor start

Slide 69

Slide 69 text

EXAMPLE PROGRAM CONFIGURATION [program:emailworker] command=/usr/bin/php /var/www/worker "MyProject\Queue\Email" process_name=%(program_name)s_%(process_num)d numprocs=2 numprocs_start=2 user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15

Slide 70

Slide 70 text

SUPERVISORD MULTI SERVER MONITORING TOOL https://github.com/mlazarov/supervisord-monitor

Slide 71

Slide 71 text

WHEN BAD THINGS HAPPEN

Slide 72

Slide 72 text

QUEUE BACKUP • Most of all issues with queues are that a queue has backed up.

Slide 73

Slide 73 text

WORKER EXCEPTIONS • Broken code causes hardships, but recoverable!

Slide 74

Slide 74 text

• https://pixabay.com/en/autobahn-accident-germany-car-road-837643/ • https://pixabay.com/en/traffic-rent-a-car-traffic-jam-637118/ • https://pixabay.com/en/airplanes-line-runway-military-713662/ • https://pixabay.com/en/leo-animal-savannah-lioness-safari-350690/ • https://pixabay.com/en/user-top-view-office-keyboard-154199/ • https://pixabay.com/en/mechanics-engine-springs-mechanic-424130/ • https://pixabay.com/en/laughter-fun-happiness-boy-child-449781/ • https://pixabay.com/en/umbrellas-red-blue-patterns-205386/ • https://pixabay.com/en/spot-runs-start-la-stadion-862274/ • https://pixabay.com/en/artistic-the-art-of-abstraction-948588/ • https://pixabay.com/en/boots-work-boots-shoes-647035/ • https://pixabay.com/en/meerkat-watch-guard-cute-676944/ • https://pixabay.com/en/broken-window-hole-glass-damage-960188/ • https://pixabay.com/en/police-security-safety-protection-869216/ • https://pixabay.com/en/parcel-package-packaging-box-575623/ • https://pixabay.com/en/directory-signposts-trail-direction-494457/ • https://pixabay.com/en/cookies-chocolate-chip-food-dessert-28423/ • https://pixabay.com/en/phone-communication-call-select-735060/ • https://pixabay.com/en/receipt-note-paper-bill-document-575750/ • https://pixabay.com/en/tools-construct-craft-repair-864983/ • https://pixabay.com/en/moore-oklahoma-tornado-disaster-112781/ • https://pixabay.com/en/network-iot-internet-of-things-782707/ • https://pixabay.com/en/mobile-phone-smartphone-app-426559/ • https://pixabay.com/en/mr-site-build-crane-baukran-462074/ • https://pixabay.com/en/film-projector-movie-projector-738806/ • https://pixabay.com/en/calves-legs-human-standing-on-540519/ • https://pixabay.com/en/notebook-pages-opened-paper-note-820078/ • https://pixabay.com/en/letters-penpal-cards-leave-stack-566810/ • https://pixabay.com/en/spray-household-surface-shine-315164/ • https://pixabay.com/en/industry-crafts-gears-mechanical-94448/ • https://pixabay.com/en/hornet-wasp-insect-sting-macro-11514/ • https://pixabay.com/en/honey-bees-bees-hive-bee-hive-401238/ • https://pixabay.com/en/temple-china-door-handle-840526/ • https://pixabay.com/en/no-button-push-sign-icon-symbol-685042/ Image Credits THANK YOU! http://joind.in/talk/view/15538