Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2015 ZendCon - Do you queue?

2015 ZendCon - Do you queue?

Presentation given during ZendCon 2015 on queuing.

753abf3aed0f744b24efda893b67ff43?s=128

Mike Willbanks

October 20, 2015
Tweet

Transcript

  1. DO YOU QUEUE? ZendCon 2015

  2. I AM MIKE WILLBANKS • Father, Husband, Developer • VP

    of Development at Packet Power • Twitter: @mwillbanks
  3. Task Producer Consumer Messages Messages Messages WHAT IS A QUEUE?

    • Pub/Sub • FIFO buffer • Push / Pull • A way to communicate between applications / systems. • A way to decouple components. • A way to offload work.
  4. WHY QUEUE?

  5. THE LONG RUNNING CASE Request Long-process Send Response Response

  6. THE INTEROP CASE Request Conditioning Service Response Response Web Service

    Call Send Response
  7. NOT YOUR DATABASE Request Process Response Send Response DB

  8. SO WHY QUEUE • User experience • System Security •

    Load Distribution • System Reliability
  9. IN PRACTICE You’ve seen this before…

  10. None
  11. None
  12. None
  13. WHAT YOU MIGHT QUEUE

  14. COMMUNICATIONS • Emails • SMS • Push Notifications

  15. IMAGES • Conversions • Resize • Thumbnail • Watermark

  16. VIDEO • Conversion • Resampling • Audio overlay

  17. IOT • Receive messages from devices and process responses

  18. PATTERNS

  19. POINT TO POINT Point to Point Channel Receiver Sender

  20. PUBLISH / SUBSCRIBE Publiser Subscribe Channel Subscriber Subscriber Subscriber

  21. MESSAGE BUS Application Application Application Message Bus

  22. PIPELINE Sender Receiver Point to Point Channel Receiver Point to

    Point Channel
  23. INVALID MESSAGE Channel Receiver Sender X Invalid Message Channel

  24. PROTOCOLS Or implementations for that matter.

  25. AMQP • AMQP Working Group (Community and Vendor) • Platform

    agnostic protocol. • Completely open, interoperable and broadly applicable. • Many severs available and many client libraries.
  26. None
  27. STOMP • Simple protocol • Behaviors follow very simple commands.

    • Most message queues can communicate over STOMP.
  28. Connect Send Disconnect /queue/ msg P H P S T

    O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack
  29. SQS • Fairly simple protocol • Supports delays, timers, and

    multiple policies.
  30. None
  31. SPECIAL PURPOSE Many queue implementations exist that don’t necessarily sit

    under standards…
  32. XMPP • Best for real-time data. • Leveraging pub/sub can

    turn it into more of a generic message system. • Multiple libraries available.
  33. SOCKET.IO • New comer • Real-time bidirectional event-based communication •

    Largely leverages pub/sub
  34. ZEROMQ • The ultimate in message queue flexibility. • Socket

    library that acts as a concurrency framework.
  35. GEARMAN • Application framework for farming out work. • Job

    sever for asynchronous or synchronous messages.
  36. BEANSTALKD • Asynchronous Job Queue • Supports delays • Many

    PHP clients exist
  37. CONSIDERATIONS How do we evaluate our options…

  38. PULL VS. PUSH • Always PULL, whenever possible. • Push

    eliminates several benefits.
  39. DURABILITY • Memory residence • Persistence • Restart survival

  40. SECURITY • Authentication • Queue permissions / restrictions

  41. DELIVERY • Is the delivery guaranteed? • If a message

    cannot be delivered how it it handled?
  42. ROUTING • Multiple routing scenarios • Fanout • Direct •

    Topic • Broadcast
  43. BATCHING • Do it later but in bulk (credit card

    processing) • Can be done via scheduling (cron)
  44. RECEIPT • Do you get an acknowledgement of receipt?

  45. IMPLEMENTING QUEUES

  46. STARTING POINTS

  47. PUSHING MESSAGES <?php class UserService { public function save($user) {

    $this->db->save($user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', [ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); } }
  48. HANDLING MESSAGES <?php $stomp = new Stomp('tcp://localhost:61613'); $stomp->subscribe('/queue/email'); while (true)

    { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body); mail($email->to, $email->subject, $email->message, $email->headers); }
  49. MESSAGES

  50. MESSAGE CONSIDERATIONS • Message Format • Message Contents

  51. SERIALIZE O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s: 3:"baz";}}}}

  52. WORKERS

  53. WORKER CONSIDERATIONS • Should do ONE thing and ONE thing

    well. • Should attempt to be as quick as possible in handling that type. • Should be able to be scaled horizontally.
  54. HANDLING WORKERS • Prevent Memory Leaks • memory_get_usage • Handle

    Signals! • pcntl_signal
  55. ABSTRACTIONS We want to make this easy…

  56. <?php interface QueueInterface { public function __construct(Stomp $stomp, $queue); public

    function dispatch(); public function publish(array $message); public function work(StompFrame $message); }
  57. <?php class AbstractQueue implements QueueInterface { protected $stomp; protected $queue;

    protected $signal; public function __construct(Stomp $stomp, $queue) { $this->stomp = $stomp; $this->queue = $queue; } protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); } if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } } protected function signal($signal) { $this->signal = $signal; }
  58. public function dispatch() { $this->prepare(); while (true) { if ($this->signal)

    { break ; } if (!$this->stomp->hasFrame()) { $this->wait(); continue ; } $frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); } $this->stomp->ack($frame); } } protected function wait() { sleep(1); } protected function validate(StompFrame $message) { return false; } public function publish(array $message) { return $this->stomp->send($this->queue, json_encode($message)); }
  59. <?php class EmailQueue extends AbstractQueue { public function validate(StompFrame $message)

    { if (!array_key_exists('to', $message)) { return false; } return true; } public function work(StompFrame $message) { $mail = json_decode($message); mail($mail->to, $mail->subject, $mail->message); } }
  60. BOOTSTRAPPING Leverage your existing infrastructure as much as possible!

  61. <?php declare(ticks=1); include 'vendor/autoload.php'; $app = Zend\Mvc\Application::init(include 'config/application.config.php'); $sm =

    $app->getServiceManager(); if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker <name>\n\n"); exit(1); } $name = $argv[1]; try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (\Exception $e) { fprintf(STDERR, "%s\n", $msg); exit(1); } $consumer = null; echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0);
  62. EVENTS

  63. SERVICES TRIGGER EVENTS <?php use Zend\EventManager\EventManagerAwareTrait; class UserService { use

    EventManagerAwareTrait; public function save($user) { $this->db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } }
  64. ATTACH EVENTS <?php use Zend\ServiceManager\ServiceManager; $sm = new ServiceManager(); $service

    = $sm->get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish([ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); });
  65. HANDLING PROGRESS • Keep track by using a generic handler

    • Or, keep track via your database.
  66. TOOLING

  67. SUPERVISOR • Daemon that runs on the server. • Monitors

    programs and keeps them running in case of failure. • Handles logging.
  68. PAINLESS INSTALLATION sudo easy_install supervisor sudo echo_supervisord_conf > /etc/supervisord.conf sudo

    service supervisor start
  69. EXAMPLE PROGRAM CONFIGURATION [program:emailworker] command=/usr/bin/php /var/www/worker "MyProject\Queue\Email" process_name=%(program_name)s_%(process_num)d numprocs=2 numprocs_start=2

    user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15
  70. SUPERVISORD MULTI SERVER MONITORING TOOL https://github.com/mlazarov/supervisord-monitor

  71. WHEN BAD THINGS HAPPEN

  72. QUEUE BACKUP • Most of all issues with queues are

    that a queue has backed up.
  73. WORKER EXCEPTIONS • Broken code causes hardships, but recoverable!

  74. • https://pixabay.com/en/autobahn-accident-germany-car-road-837643/ • https://pixabay.com/en/traffic-rent-a-car-traffic-jam-637118/ • https://pixabay.com/en/airplanes-line-runway-military-713662/ • https://pixabay.com/en/leo-animal-savannah-lioness-safari-350690/ • https://pixabay.com/en/user-top-view-office-keyboard-154199/

    • https://pixabay.com/en/mechanics-engine-springs-mechanic-424130/ • https://pixabay.com/en/laughter-fun-happiness-boy-child-449781/ • https://pixabay.com/en/umbrellas-red-blue-patterns-205386/ • https://pixabay.com/en/spot-runs-start-la-stadion-862274/ • https://pixabay.com/en/artistic-the-art-of-abstraction-948588/ • https://pixabay.com/en/boots-work-boots-shoes-647035/ • https://pixabay.com/en/meerkat-watch-guard-cute-676944/ • https://pixabay.com/en/broken-window-hole-glass-damage-960188/ • https://pixabay.com/en/police-security-safety-protection-869216/ • https://pixabay.com/en/parcel-package-packaging-box-575623/ • https://pixabay.com/en/directory-signposts-trail-direction-494457/ • https://pixabay.com/en/cookies-chocolate-chip-food-dessert-28423/ • https://pixabay.com/en/phone-communication-call-select-735060/ • https://pixabay.com/en/receipt-note-paper-bill-document-575750/ • https://pixabay.com/en/tools-construct-craft-repair-864983/ • https://pixabay.com/en/moore-oklahoma-tornado-disaster-112781/ • https://pixabay.com/en/network-iot-internet-of-things-782707/ • https://pixabay.com/en/mobile-phone-smartphone-app-426559/ • https://pixabay.com/en/mr-site-build-crane-baukran-462074/ • https://pixabay.com/en/film-projector-movie-projector-738806/ • https://pixabay.com/en/calves-legs-human-standing-on-540519/ • https://pixabay.com/en/notebook-pages-opened-paper-note-820078/ • https://pixabay.com/en/letters-penpal-cards-leave-stack-566810/ • https://pixabay.com/en/spray-household-surface-shine-315164/ • https://pixabay.com/en/industry-crafts-gears-mechanical-94448/ • https://pixabay.com/en/hornet-wasp-insect-sting-macro-11514/ • https://pixabay.com/en/honey-bees-bees-hive-bee-hive-401238/ • https://pixabay.com/en/temple-china-door-handle-840526/ • https://pixabay.com/en/no-button-push-sign-icon-symbol-685042/ Image Credits THANK YOU! http://joind.in/talk/view/15538