2015 ZendCon - Do you queue?

Presentation given during ZendCon 2015 on queuing.

Transcript

1. DO YOU QUEUE?
   ZendCon 2015
   

   View Slide

2. I AM MIKE WILLBANKS
   • Father, Husband, Developer
   • VP of Development at
   Packet Power
   • Twitter: @mwillbanks
   

   View Slide

3. Task
   Producer Consumer
   Messages
   Messages
   Messages
   WHAT IS A QUEUE?
   • Pub/Sub
   • FIFO buffer
   • Push / Pull
   • A way to communicate
   between applications / systems.
   • A way to decouple
   components.
   • A way to offload work.
   

   View Slide

4. WHY QUEUE?
   

   View Slide

5. THE LONG RUNNING CASE
   Request Long-process
   Send Response
   Response
   

   View Slide

6. THE INTEROP CASE
   Request Conditioning
   Service Response
   Response
   Web Service Call
   Send Response
   

   View Slide

7. NOT YOUR DATABASE
   Request Process
   Response Send Response
   DB
   

   View Slide

8. SO WHY QUEUE
   • User experience
   • System Security
   • Load Distribution
   • System Reliability
   

   View Slide

9. IN PRACTICE
   You’ve seen this before…
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. WHAT YOU MIGHT QUEUE
    

    View Slide

14. COMMUNICATIONS
    • Emails
    • SMS
    • Push Notifications
    

    View Slide

15. IMAGES
    • Conversions
    • Resize
    • Thumbnail
    • Watermark
    

    View Slide

16. VIDEO
    • Conversion
    • Resampling
    • Audio overlay
    

    View Slide

17. IOT
    • Receive messages from
    devices and process
    responses
    

    View Slide

18. PATTERNS
    

    View Slide

19. POINT TO POINT
    Point to Point
    Channel
    Receiver
    Sender
    

    View Slide

20. PUBLISH / SUBSCRIBE
    Publiser Subscribe Channel
    Subscriber
    Subscriber
    Subscriber
    

    View Slide

21. MESSAGE BUS
    Application
    Application
    Application
    Message Bus
    

    View Slide

22. PIPELINE
    Sender
    Receiver
    Point to Point
    Channel
    Receiver
    Point to Point
    Channel
    

    View Slide

23. INVALID MESSAGE
    Channel
    Receiver
    Sender
    X
    Invalid Message
    Channel
    

    View Slide

24. PROTOCOLS
    Or implementations for that matter.
    

    View Slide

25. AMQP
    • AMQP Working Group
    (Community and Vendor)
    • Platform agnostic protocol.
    • Completely open,
    interoperable and broadly
    applicable.
    • Many severs available and
    many client libraries.
    

    View Slide

26. View Slide

27. STOMP
    • Simple protocol
    • Behaviors follow very
    simple commands.
    • Most message queues can
    communicate over STOMP.
    

    View Slide

28. Connect Send Disconnect
    /queue/
    msg
    P
    H
    P
    S
    T
    O
    M
    P
    S
    E
    R
    V
    E
    R
    Connect Subscribe Disconnect
    /queue/
    msg
    Read
    Ack
    

    View Slide

29. SQS
    • Fairly simple protocol
    • Supports delays, timers, and
    multiple policies.
    

    View Slide

30. View Slide

31. SPECIAL
    PURPOSE
    Many queue implementations
    exist that don’t necessarily sit
    under standards…
    

    View Slide

32. XMPP
    • Best for real-time data.
    • Leveraging pub/sub can
    turn it into more of a
    generic message system.
    • Multiple libraries available.
    

    View Slide

33. SOCKET.IO
    • New comer
    • Real-time bidirectional
    event-based communication
    • Largely leverages pub/sub
    

    View Slide

34. ZEROMQ
    • The ultimate in message
    queue flexibility.
    • Socket library that acts as a
    concurrency framework.
    

    View Slide

35. GEARMAN
    • Application framework for
    farming out work.
    • Job sever for asynchronous
    or synchronous messages.
    

    View Slide

36. BEANSTALKD
    • Asynchronous Job Queue
    • Supports delays
    • Many PHP clients exist
    

    View Slide

37. CONSIDERATIONS
    How do we evaluate our options…
    

    View Slide

38. PULL VS. PUSH
    • Always PULL, whenever
    possible.
    • Push eliminates several
    benefits.
    

    View Slide

39. DURABILITY
    • Memory residence
    • Persistence
    • Restart survival
    

    View Slide

40. SECURITY
    • Authentication
    • Queue permissions /
    restrictions
    

    View Slide

41. DELIVERY
    • Is the delivery guaranteed?
    • If a message cannot be
    delivered how it it handled?
    

    View Slide

42. ROUTING
    • Multiple routing scenarios
    • Fanout
    • Direct
    • Topic
    • Broadcast
    

    View Slide

43. BATCHING
    • Do it later but in bulk
    (credit card processing)
    • Can be done via scheduling
    (cron)
    

    View Slide

44. RECEIPT
    • Do you get an
    acknowledgement of
    receipt?
    

    View Slide

45. IMPLEMENTING QUEUES
    

    View Slide

46. STARTING POINTS
    

    View Slide

47. PUSHING MESSAGES
    class UserService {
    public function save($user) {
    $this->db->save($user);
    $stomp = new Stomp('tcp://localhost:61613');
    $stomp->send('/queue/email', [
    'to' => $user->getEmail(),
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]);
    }
    }
    

    View Slide

48. HANDLING MESSAGES
    $stomp = new Stomp('tcp://localhost:61613');
    $stomp->subscribe('/queue/email');
    while (true) {
    if (!$stomp->hasFrame()) {
    sleep(2);
    continue ;
    }
    $stomp->readFrame();
    $email = json_decode($frame->body);
    mail($email->to, $email->subject, $email->message, $email->headers);
    }
    

    View Slide

49. MESSAGES
    

    View Slide

50. MESSAGE CONSIDERATIONS
    • Message Format
    • Message Contents
    

    View Slide

51. SERIALIZE
    O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s:
    3:"baz";}}}}
    

    View Slide

52. WORKERS
    

    View Slide

53. WORKER CONSIDERATIONS
    • Should do ONE thing and
    ONE thing well.
    • Should attempt to be as
    quick as possible in handling
    that type.
    • Should be able to be scaled
    horizontally.
    

    View Slide

54. HANDLING WORKERS
    • Prevent Memory Leaks
    • memory_get_usage
    • Handle Signals!
    • pcntl_signal
    

    View Slide

55. ABSTRACTIONS
    We want to make this easy…
    

    View Slide

56. interface QueueInterface {
    public function __construct(Stomp $stomp, $queue);
    public function dispatch();
    public function publish(array $message);
    public function work(StompFrame $message);
    }
    

    View Slide

57. class AbstractQueue implements QueueInterface {
    protected $stomp;
    protected $queue;
    protected $signal;
    public function __construct(Stomp $stomp, $queue) {
    $this->stomp = $stomp;
    $this->queue = $queue;
    }
    protected function prepare() {
    if (php_sapi_name() != 'cli') {
    throw new RuntimeException('You cannot dispatch outside of the CLI');
    }
    if (function_exists('pcntl_signal')) {
    pcntl_signal(SIGTERM, array($this, 'signal'));
    pcntl_signal(SIGINT, array($this, 'signal'));
    pcntl_signal(SIGHUP, array($this, 'signal'));
    }
    }
    protected function signal($signal) {
    $this->signal = $signal;
    }
    

    View Slide

58. public function dispatch() {
    $this->prepare();
    while (true) {
    if ($this->signal) {
    break ;
    }
    if (!$this->stomp->hasFrame()) {
    $this->wait();
    continue ;
    }
    $frame = $this->stomp->readFrame();
    if ($this->validate($frame)) {
    $this->work($frame);
    }
    $this->stomp->ack($frame);
    }
    }
    protected function wait() {
    sleep(1);
    }
    protected function validate(StompFrame $message) {
    return false;
    }
    public function publish(array $message) {
    return $this->stomp->send($this->queue, json_encode($message));
    }
    

    View Slide

59. class EmailQueue extends AbstractQueue {
    public function validate(StompFrame $message) {
    if (!array_key_exists('to', $message)) {
    return false;
    }
    return true;
    }
    public function work(StompFrame $message) {
    $mail = json_decode($message);
    mail($mail->to, $mail->subject, $mail->message);
    }
    }
    

    View Slide

60. BOOTSTRAPPING
    Leverage your existing infrastructure as much as possible!
    

    View Slide

61. declare(ticks=1);
    include 'vendor/autoload.php';
    $app = Zend\Mvc\Application::init(include 'config/application.config.php');
    $sm = $app->getServiceManager();
    if (!isset($argv[1])) {
    fprintf(STDERR, "Syntax: worker \n\n");
    exit(1);
    }
    $name = $argv[1];
    try {
    echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL;
    $consumer = $sm->get($name);
    $consumer->dispatch();
    } catch (\Exception $e) {
    fprintf(STDERR, "%s\n", $msg);
    exit(1);
    }
    $consumer = null;
    echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL;
    exit(0);
    

    View Slide

62. EVENTS
    

    View Slide

63. SERVICES TRIGGER EVENTS
    use Zend\EventManager\EventManagerAwareTrait;
    class UserService
    {
    use EventManagerAwareTrait;
    public function save($user) {
    $this->db->save($user);
    $this->getEventManager()->trigger('save', null, ['user' => $user]);
    }
    }
    

    View Slide

64. ATTACH EVENTS
    use Zend\ServiceManager\ServiceManager;
    $sm = new ServiceManager();
    $service = $sm->get('UserService');
    $queue = $sm->get('EmailQueue');
    $service->getEventManager()->attach('save', function($e) use ($queue) {
    $params = $e->getParams();
    $queue->publish([
    'to' => $params['user']['email'],
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]);
    });
    

    View Slide

65. HANDLING
    PROGRESS
    • Keep track by using a
    generic handler
    • Or, keep track via your
    database.
    

    View Slide

66. TOOLING
    

    View Slide

67. SUPERVISOR
    • Daemon that runs on the
    server.
    • Monitors programs and
    keeps them running in case
    of failure.
    • Handles logging.
    

    View Slide

68. PAINLESS INSTALLATION
    sudo easy_install supervisor
    sudo echo_supervisord_conf > /etc/supervisord.conf
    sudo service supervisor start
    

    View Slide

69. EXAMPLE PROGRAM
    CONFIGURATION
    [program:emailworker]
    command=/usr/bin/php /var/www/worker "MyProject\Queue\Email"
    process_name=%(program_name)s_%(process_num)d
    numprocs=2
    numprocs_start=2
    user=www-data
    autostart=true ; start at supervisord start (default: true)
    autorestart=true ; retstart at unexpected quit (default: true)
    startsecs=10 ; number of secs prog must stay running (def. 10)
    startretries=5 ; max # of serial start failures (default 3)
    log_stdout=true ; if true, log program stdout (default true)
    log_stderr=true ; if true, log program stderr (def false)
    redirect_stderr=true ; if true, redirect stderr to stdout
    stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log
    stdout_logfile_maxbytes=10MB
    stdout_logfile_backups=15
    

    View Slide

70. SUPERVISORD MULTI SERVER MONITORING
    TOOL
    https://github.com/mlazarov/supervisord-monitor
    

    View Slide

71. WHEN BAD THINGS HAPPEN
    

    View Slide

72. QUEUE BACKUP
    • Most of all issues with
    queues are that a queue has
    backed up.
    

    View Slide

73. WORKER EXCEPTIONS
    • Broken code causes
    hardships, but recoverable!
    

    View Slide

74. • https://pixabay.com/en/autobahn-accident-germany-car-road-837643/
    • https://pixabay.com/en/traffic-rent-a-car-traffic-jam-637118/
    • https://pixabay.com/en/airplanes-line-runway-military-713662/
    • https://pixabay.com/en/leo-animal-savannah-lioness-safari-350690/
    • https://pixabay.com/en/user-top-view-office-keyboard-154199/
    • https://pixabay.com/en/mechanics-engine-springs-mechanic-424130/
    • https://pixabay.com/en/laughter-fun-happiness-boy-child-449781/
    • https://pixabay.com/en/umbrellas-red-blue-patterns-205386/
    • https://pixabay.com/en/spot-runs-start-la-stadion-862274/
    • https://pixabay.com/en/artistic-the-art-of-abstraction-948588/
    • https://pixabay.com/en/boots-work-boots-shoes-647035/
    • https://pixabay.com/en/meerkat-watch-guard-cute-676944/
    • https://pixabay.com/en/broken-window-hole-glass-damage-960188/
    • https://pixabay.com/en/police-security-safety-protection-869216/
    • https://pixabay.com/en/parcel-package-packaging-box-575623/
    • https://pixabay.com/en/directory-signposts-trail-direction-494457/
    • https://pixabay.com/en/cookies-chocolate-chip-food-dessert-28423/
    • https://pixabay.com/en/phone-communication-call-select-735060/
    • https://pixabay.com/en/receipt-note-paper-bill-document-575750/
    • https://pixabay.com/en/tools-construct-craft-repair-864983/
    • https://pixabay.com/en/moore-oklahoma-tornado-disaster-112781/
    • https://pixabay.com/en/network-iot-internet-of-things-782707/
    • https://pixabay.com/en/mobile-phone-smartphone-app-426559/
    • https://pixabay.com/en/mr-site-build-crane-baukran-462074/
    • https://pixabay.com/en/film-projector-movie-projector-738806/
    • https://pixabay.com/en/calves-legs-human-standing-on-540519/
    • https://pixabay.com/en/notebook-pages-opened-paper-note-820078/
    • https://pixabay.com/en/letters-penpal-cards-leave-stack-566810/
    • https://pixabay.com/en/spray-household-surface-shine-315164/
    • https://pixabay.com/en/industry-crafts-gears-mechanical-94448/
    • https://pixabay.com/en/hornet-wasp-insect-sting-macro-11514/
    • https://pixabay.com/en/honey-bees-bees-hive-bee-hive-401238/
    • https://pixabay.com/en/temple-china-door-handle-840526/
    • https://pixabay.com/en/no-button-push-sign-icon-symbol-685042/
    Image Credits
    THANK YOU!
    http://joind.in/talk/view/15538
    

    View Slide