$30 off During Our Annual Pro Sale. View Details »

Queue It

Queue It

Message Queues and PHP

Mike Willbanks

August 16, 2018
Tweet

More Decks by Mike Willbanks

Other Decks in Technology

Transcript

  1. QUEUE IT!

    View Slide

  2. Task
    Producer Consumer
    Messages
    Messages
    Messages
    WHAT IS A QUEUE?
    • Pub/Sub
    • FIFO buffer
    • Push / Pull
    • A way to communicate
    between applications / systems.
    • A way to decouple
    components.
    • A way to offload work.

    View Slide

  3. WHY QUEUE?

    View Slide

  4. THE LONG RUNNING CASE
    Request Long-process
    Send Response
    Response

    View Slide

  5. THE INTEROP CASE
    Request Conditioning
    Service Response
    Response
    Web Service Call
    Send Response

    View Slide

  6. NOT YOUR DATABASE
    Request Process
    Response Send Response
    DB

    View Slide

  7. IOT
    Data S3
    Possible Action Process
    Event

    View Slide

  8. SO WHY QUEUE
    • User experience
    • System Security
    • Load Distribution
    • System Reliability

    View Slide

  9. IN PRACTICE
    You’ve seen this before…

    View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. WHAT YOU MIGHT QUEUE

    View Slide

  14. COMMUNICATIONS
    • Emails
    • SMS
    • Push Notifications

    View Slide

  15. IMAGES
    • Conversions
    • Resize
    • Thumbnail
    • Watermark

    View Slide

  16. VIDEO
    • Conversion
    • Resampling
    • Audio overlay

    View Slide

  17. IOT
    • Receive messages from
    devices and process
    responses

    View Slide

  18. PATTERNS

    View Slide

  19. POINT TO POINT
    Point to Point
    Channel
    Receiver
    Sender

    View Slide

  20. PUBLISH / SUBSCRIBE
    Publiser Subscribe Channel
    Subscriber
    Subscriber
    Subscriber

    View Slide

  21. MESSAGE BUS
    Application
    Application
    Application
    Message Bus

    View Slide

  22. PIPELINE
    Sender
    Receiver
    Point to Point
    Channel
    Receiver
    Point to Point
    Channel

    View Slide

  23. INVALID MESSAGE
    Channel
    Receiver
    Sender
    X
    Invalid Message
    Channel

    View Slide

  24. PROTOCOLS
    Or implementations for that matter.

    View Slide

  25. AMQP
    • AMQP Working Group
    (Community and Vendor)
    • Platform agnostic protocol.
    • Completely open, interoperable
    and broadly applicable.
    • Many severs available and many
    client libraries.
    • Best generally known in
    RabbitMQ.

    View Slide

  26. View Slide

  27. KAFKA
    • Works with "streams" of
    data.
    • Limited messaging patterns.

    View Slide

  28. View Slide

  29. STOMP
    • Simple protocol
    • Behaviors follow very
    simple commands.
    • Most message queues can
    communicate over STOMP.

    View Slide

  30. Connect Send Disconnect
    /queue/
    msg
    P
    H
    P
    S
    T
    O
    M
    P
    S
    E
    R
    V
    E
    R
    Connect Subscribe Disconnect
    /queue/
    msg
    Read
    Ack

    View Slide

  31. SQS
    • Simplistic protocol, HTTP-
    based.
    • Supports delays, timers, and
    multiple policies.
    • Combine with SNS to
    implement patterns.

    View Slide

  32. View Slide

  33. SPECIAL
    PURPOSE
    Many queue implementations
    exist that don’t necessarily sit
    under standards…

    View Slide

  34. ZEROMQ
    • The ultimate in message
    queue flexibility.
    • Socket library that acts as a
    concurrency framework.

    View Slide

  35. SOCKET.IO
    • Real-time bidirectional
    event-based communication
    • Largely leverages pub/sub

    View Slide

  36. XMPP
    • Best for real-time data.
    • Leveraging pub/sub can
    turn it into more of a
    generic message system.
    • Multiple libraries available.

    View Slide

  37. BEANSTALKD
    • Asynchronous Job Queue
    • Supports delays
    • Many PHP clients exist

    View Slide

  38. REDIS
    • Supports pub/sub
    • Supports atomic lists which
    people build queues off of

    View Slide

  39. YOUR DATABASE
    • The poor mans queue.
    • Generally leveraging
    memory tables.

    View Slide

  40. A list of 30+ message queue implementations.
    NOTE: NOT ALL ARE "REAL" MESSAGE QUEUES.
    queues.io

    View Slide

  41. CONSIDERATIONS
    How do we evaluate our options…

    View Slide

  42. PULL VS. PUSH
    • Always PULL, whenever
    possible in back-end
    applications.
    • Push eliminates several
    benefits, however, can be
    useful with web-workers
    and front-end applications.

    View Slide

  43. DURABILITY
    • Memory residence
    • Persistence
    • Restart survival

    View Slide

  44. SECURITY
    • Authentication
    • Queue permissions /
    restrictions

    View Slide

  45. DELIVERY
    • Is the delivery guaranteed?
    • If a message cannot be
    delivered how it it handled?

    View Slide

  46. ROUTING
    • Multiple routing scenarios
    • Fanout
    • Direct
    • Topic
    • Broadcast

    View Slide

  47. BATCHING
    • Do it later but in bulk
    (credit card processing)
    • Can be done via scheduling
    (jenkins)

    View Slide

  48. RECEIPT
    • Do you get an
    acknowledgement of
    receipt?

    View Slide

  49. IMPLEMENTING QUEUES

    View Slide

  50. STARTING POINTS

    View Slide

  51. PUSHING VIA STOMP
    class UserService {
    public function save(MyUser $user) {
    $this->db->save(MyUser $user);
    $stomp = new Stomp('tcp://localhost:61613');
    $stomp->send('/queue/email', json_encode([
    'to' => $user->getEmail(),
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]));
    }
    }

    View Slide

  52. FETCHING VIA STOMP
    $stomp = new Stomp('tcp://localhost:61613');
    $stomp->subscribe('/queue/email');
    while (true) {
    if (!$stomp->hasFrame()) {
    sleep(2);
    continue ;
    }
    $stomp->readFrame();
    $email = json_decode($frame->body);
    mail($email->to, $email->subject, $email->message, $email->headers);
    }

    View Slide

  53. PUSHING VIA AMQP
    class UserService {
    public function save(MyUser $user) {
    $this->db->save(MyUser $user);
    $amqp = new AMQPConnection();
    $amqp->connect();
    $ex = new AMQPExchange();
    $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT);
    $q = new AMQPQueue($amqp);
    $q->declare('email');
    $ex->bind('email', 'routing.key');
    $ex->publish(json_encode([
    'to' => $user->getEmail(),
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]), 'routing.key');
    }
    }

    View Slide

  54. FETCHING VIA AMQP
    $amqp = new AMQPConnection();
    $amqp->connect();
    $ex = new AMQPExchange();
    $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT);
    $q = new AMQPQueue($amqp);
    $q->declare('email');
    $ex->bind('email', 'routing.key');
    while ($messages = $q->consume()) {
    foreach ($messages as $message) {
    $email = json_decode($message['message_body']);
    mail($email->to, $email->subject, $email->message, $email->headers);
    }
    }

    View Slide

  55. MESSAGES

    View Slide

  56. MESSAGE CONSIDERATIONS
    • Message Format
    • Message Contents

    View Slide

  57. SERIALIZE
    O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s:
    3:"baz";}}}}

    View Slide

  58. WORKERS

    View Slide

  59. WORKER CONSIDERATIONS
    • Should do ONE thing and
    ONE thing well.
    • Should attempt to be as
    quick as possible in handling
    that type.
    • Should be able to be scaled
    horizontally.

    View Slide

  60. HANDLING WORKERS
    • Prevent Memory Leaks
    • memory_get_usage
    • Handle Signals!
    • pcntl_signal

    View Slide

  61. ABSTRACTIONS
    We want to make this easy…

    View Slide

  62. SYMFONY MESSENGER

    View Slide

  63. LARAVEL QUEUES

    View Slide

  64. BERNARD
    General queue implementation

    View Slide

  65. interface QueueInterface {
    public function __construct(Stomp $stomp, String $queue);
    public function dispatch();
    public function publish : bool(array $message);
    public function work(StompFrame $message);
    }

    View Slide

  66. class AbstractQueue implements QueueInterface {
    protected $stomp;
    protected $queue;
    protected $signal;
    public function __construct(Stomp $stomp, string $queue) {
    $this->stomp = $stomp;
    $this->queue = $queue;
    }
    protected function prepare() {
    if (php_sapi_name() != 'cli') {
    throw new RuntimeException('You cannot dispatch outside of the CLI');
    }
    if (function_exists('pcntl_signal')) {
    pcntl_signal(SIGTERM, array($this, 'signal'));
    pcntl_signal(SIGINT, array($this, 'signal'));
    pcntl_signal(SIGHUP, array($this, 'signal'));
    }
    }
    protected function signal(int $signal) {
    $this->signal = $signal;
    }

    View Slide

  67. public function dispatch() {
    $this->prepare();
    while (true) {
    if ($this->signal) {
    break ;
    }
    if (!$this->stomp->hasFrame()) {
    $this->wait();
    continue ;
    }
    $frame = $this->stomp->readFrame();
    if ($this->validate($frame)) {
    $this->work($frame);
    }
    $this->stomp->ack($frame);
    }
    }
    protected function wait() {
    sleep(1);
    }
    protected function validate(StompFrame $message): bool {
    return false;
    }
    public function publish(array $message): bool {
    return $this->stomp->send($this->queue, json_encode($message));
    }

    View Slide

  68. class EmailQueue extends AbstractQueue {
    public function validate(StompFrame $message): bool {
    if (!array_key_exists('to', $message)) {
    return false;
    }
    return true;
    }
    public function work(StompFrame $message) {
    $mail = json_decode($message);
    mail($mail->to, $mail->subject, $mail->message);
    }
    }

    View Slide

  69. BOOTSTRAPPING
    Leverage your existing infrastructure as much as possible!

    View Slide

  70. declare(ticks=1);
    include 'vendor/autoload.php';
    $app = Zend\Mvc\Application::init(include 'config/application.config.php');
    $sm = $app->getServiceManager();
    if (!isset($argv[1])) {
    fprintf(STDERR, "Syntax: worker \n\n");
    exit(1);
    }
    $name = $argv[1];
    try {
    echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL;
    $consumer = $sm->get($name);
    $consumer->dispatch();
    } catch (\Exception $e) {
    fprintf(STDERR, "%s\n", $msg);
    exit(1);
    }
    $consumer = null;
    echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL;
    exit(0);

    View Slide

  71. EVENTS

    View Slide

  72. SERVICES TRIGGER EVENTS
    use Zend\EventManager\EventManagerAwareTrait;
    class UserService
    {
    use EventManagerAwareTrait;
    public function save(MyUser $user) {
    $this->db->save($user);
    $this->getEventManager()->trigger('save', null, ['user' => $user]);
    }
    }

    View Slide

  73. ATTACH EVENTS
    use Zend\ServiceManager\ServiceManager;
    $sm = new ServiceManager();
    $service = $sm->get('UserService');
    $queue = $sm->get('EmailQueue');
    $service->getEventManager()->attach('save', function($e) use ($queue) {
    $params = $e->getParams();
    $queue->publish(json_encode[
    'to' => $params['user']['email'],
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]);
    });

    View Slide

  74. HANDLING
    PROGRESS
    UPDATES
    • Keep track of item state
    using a cache server or
    database.
    • Utilize events on your
    processing to provide
    updates to the UI.

    View Slide

  75. HOW MIGHT WE UPDATE?
    $message = json_decode($stomp->readFrame()->body);
    $proc = proc_open($command, $descriptorSpec, $pipes);
    stream_set_blocking($pipes[1], 0);
    while (!feof($pipes[1])) {
    $content = fgets($pipes[1], 1024);
    if (preg_match('/something-i-(care)-about/', $content, $results)) {
    // provide update to DB, cache, etc.
    }
    }

    View Slide

  76. TOOLING

    View Slide

  77. JENKINS
    • It's not just for your CI/CD
    projects.

    View Slide

  78. JENKINS: AMAZON EC2
    • Install the Amazon EC2 Plugin
    • Leverage images and use spot instances
    • Configure spot instance to deploy latest code on
    boot
    • Configure jenkins to boot up additional spot
    instances if jobs are waiting.

    View Slide

  79. JENKINS: JOBS
    • Handlers
    • 1 job per handler with:
    • Execute concurrent builds if necessary
    • Restrict where this project can be run (spot instance label).
    • Scheduled job to watch queue lengths and available workers
    • queue length > available workers after x time = execute job
    • queue length < available workers after x time = kill job

    View Slide

  80. SYSTEMD
    • Integrated into the OS.
    • Monitors and handles
    starting, restarting and
    dependencies of processes.
    • Integrated logging.

    View Slide

  81. SERVICE FILE EXAMPLE
    # /etc/systemd/system/my_worker.service
    [Unit]
    Description=My Worker Service
    After=network.target
    [Service]
    Type=simple
    User=unix-user
    WorkingDirectory=/path/to/working/dir
    ExecStart=/path/to/worker --opt1
    Restart=on-failure # or always, on-abort, etc
    [Install]
    WantedBy=multi-user.target

    View Slide

  82. HOW TO USE IT?
    # controlling

    systemctl start|stop|status|enable|
    disable my-worker.service


    # logs

    journalctl -f -u my-worker.service

    View Slide

  83. SUPERVISOR
    • Daemon that runs on the
    server.
    • Monitors programs and
    keeps them running in case
    of failure.
    • Handles logging.
    • If systemd is unavailable or a
    bit too scary :)

    View Slide

  84. PAINLESS INSTALLATION
    sudo easy_install supervisor
    sudo echo_supervisord_conf > /etc/supervisord.conf
    sudo service supervisor start

    View Slide

  85. EXAMPLE PROGRAM
    CONFIGURATION
    [program:emailworker]
    command=/usr/bin/php /var/www/worker "MyProject\Queue\Email"
    process_name=%(program_name)s_%(process_num)d
    numprocs=2
    numprocs_start=2
    user=www-data
    autostart=true ; start at supervisord start (default: true)
    autorestart=true ; retstart at unexpected quit (default: true)
    startsecs=10 ; number of secs prog must stay running (def. 10)
    startretries=5 ; max # of serial start failures (default 3)
    log_stdout=true ; if true, log program stdout (default true)
    log_stderr=true ; if true, log program stderr (def false)
    redirect_stderr=true ; if true, redirect stderr to stdout
    stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log
    stdout_logfile_maxbytes=10MB
    stdout_logfile_backups=15

    View Slide

  86. SUPERVISORD MULTI SERVER MONITORING
    TOOL
    https://github.com/mlazarov/supervisord-monitor

    View Slide

  87. WHEN BAD THINGS HAPPEN

    View Slide

  88. QUEUE BACKUP
    • Queue length continues to
    grow and you're not
    processing enough...

    View Slide

  89. WORKER EXCEPTIONS
    • Broken code, causes
    blowups.

    View Slide

  90. THANK YOU
    • @mwillbanks
    • Provide your feedback!
    joind.in/talk/a6dfa

    View Slide