Queue It

Message Queues and PHP

Transcript

1. QUEUE IT!
   

   View Slide

2. Task
   Producer Consumer
   Messages
   Messages
   Messages
   WHAT IS A QUEUE?
   • Pub/Sub
   • FIFO buffer
   • Push / Pull
   • A way to communicate
   between applications / systems.
   • A way to decouple
   components.
   • A way to offload work.
   

   View Slide

3. WHY QUEUE?
   

   View Slide

4. THE LONG RUNNING CASE
   Request Long-process
   Send Response
   Response
   

   View Slide

5. THE INTEROP CASE
   Request Conditioning
   Service Response
   Response
   Web Service Call
   Send Response
   

   View Slide

6. NOT YOUR DATABASE
   Request Process
   Response Send Response
   DB
   

   View Slide

7. IOT
   Data S3
   Possible Action Process
   Event
   

   View Slide

8. SO WHY QUEUE
   • User experience
   • System Security
   • Load Distribution
   • System Reliability
   

   View Slide

9. IN PRACTICE
   You’ve seen this before…
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. WHAT YOU MIGHT QUEUE
    

    View Slide

14. COMMUNICATIONS
    • Emails
    • SMS
    • Push Notifications
    

    View Slide

15. IMAGES
    • Conversions
    • Resize
    • Thumbnail
    • Watermark
    

    View Slide

16. VIDEO
    • Conversion
    • Resampling
    • Audio overlay
    

    View Slide

17. IOT
    • Receive messages from
    devices and process
    responses
    

    View Slide

18. PATTERNS
    

    View Slide

19. POINT TO POINT
    Point to Point
    Channel
    Receiver
    Sender
    

    View Slide

20. PUBLISH / SUBSCRIBE
    Publiser Subscribe Channel
    Subscriber
    Subscriber
    Subscriber
    

    View Slide

21. MESSAGE BUS
    Application
    Application
    Application
    Message Bus
    

    View Slide

22. PIPELINE
    Sender
    Receiver
    Point to Point
    Channel
    Receiver
    Point to Point
    Channel
    

    View Slide

23. INVALID MESSAGE
    Channel
    Receiver
    Sender
    X
    Invalid Message
    Channel
    

    View Slide

24. PROTOCOLS
    Or implementations for that matter.
    

    View Slide

25. AMQP
    • AMQP Working Group
    (Community and Vendor)
    • Platform agnostic protocol.
    • Completely open, interoperable
    and broadly applicable.
    • Many severs available and many
    client libraries.
    • Best generally known in
    RabbitMQ.
    

    View Slide

26. View Slide

27. KAFKA
    • Works with "streams" of
    data.
    • Limited messaging patterns.
    

    View Slide

28. View Slide

29. STOMP
    • Simple protocol
    • Behaviors follow very
    simple commands.
    • Most message queues can
    communicate over STOMP.
    

    View Slide

30. Connect Send Disconnect
    /queue/
    msg
    P
    H
    P
    S
    T
    O
    M
    P
    S
    E
    R
    V
    E
    R
    Connect Subscribe Disconnect
    /queue/
    msg
    Read
    Ack
    

    View Slide

31. SQS
    • Simplistic protocol, HTTP-
    based.
    • Supports delays, timers, and
    multiple policies.
    • Combine with SNS to
    implement patterns.
    

    View Slide

32. View Slide

33. SPECIAL
    PURPOSE
    Many queue implementations
    exist that don’t necessarily sit
    under standards…
    

    View Slide

34. ZEROMQ
    • The ultimate in message
    queue flexibility.
    • Socket library that acts as a
    concurrency framework.
    

    View Slide

35. SOCKET.IO
    • Real-time bidirectional
    event-based communication
    • Largely leverages pub/sub
    

    View Slide

36. XMPP
    • Best for real-time data.
    • Leveraging pub/sub can
    turn it into more of a
    generic message system.
    • Multiple libraries available.
    

    View Slide

37. BEANSTALKD
    • Asynchronous Job Queue
    • Supports delays
    • Many PHP clients exist
    

    View Slide

38. REDIS
    • Supports pub/sub
    • Supports atomic lists which
    people build queues off of
    

    View Slide

39. YOUR DATABASE
    • The poor mans queue.
    • Generally leveraging
    memory tables.
    

    View Slide

40. A list of 30+ message queue implementations.
    NOTE: NOT ALL ARE "REAL" MESSAGE QUEUES.
    queues.io
    

    View Slide

41. CONSIDERATIONS
    How do we evaluate our options…
    

    View Slide

42. PULL VS. PUSH
    • Always PULL, whenever
    possible in back-end
    applications.
    • Push eliminates several
    benefits, however, can be
    useful with web-workers
    and front-end applications.
    

    View Slide

43. DURABILITY
    • Memory residence
    • Persistence
    • Restart survival
    

    View Slide

44. SECURITY
    • Authentication
    • Queue permissions /
    restrictions
    

    View Slide

45. DELIVERY
    • Is the delivery guaranteed?
    • If a message cannot be
    delivered how it it handled?
    

    View Slide

46. ROUTING
    • Multiple routing scenarios
    • Fanout
    • Direct
    • Topic
    • Broadcast
    

    View Slide

47. BATCHING
    • Do it later but in bulk
    (credit card processing)
    • Can be done via scheduling
    (jenkins)
    

    View Slide

48. RECEIPT
    • Do you get an
    acknowledgement of
    receipt?
    

    View Slide

49. IMPLEMENTING QUEUES
    

    View Slide

50. STARTING POINTS
    

    View Slide

51. PUSHING VIA STOMP
    class UserService {
    public function save(MyUser $user) {
    $this->db->save(MyUser $user);
    $stomp = new Stomp('tcp://localhost:61613');
    $stomp->send('/queue/email', json_encode([
    'to' => $user->getEmail(),
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]));
    }
    }
    

    View Slide

52. FETCHING VIA STOMP
    $stomp = new Stomp('tcp://localhost:61613');
    $stomp->subscribe('/queue/email');
    while (true) {
    if (!$stomp->hasFrame()) {
    sleep(2);
    continue ;
    }
    $stomp->readFrame();
    $email = json_decode($frame->body);
    mail($email->to, $email->subject, $email->message, $email->headers);
    }
    

    View Slide

53. PUSHING VIA AMQP
    class UserService {
    public function save(MyUser $user) {
    $this->db->save(MyUser $user);
    $amqp = new AMQPConnection();
    $amqp->connect();
    $ex = new AMQPExchange();
    $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT);
    $q = new AMQPQueue($amqp);
    $q->declare('email');
    $ex->bind('email', 'routing.key');
    $ex->publish(json_encode([
    'to' => $user->getEmail(),
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]), 'routing.key');
    }
    }
    

    View Slide

54. FETCHING VIA AMQP
    $amqp = new AMQPConnection();
    $amqp->connect();
    $ex = new AMQPExchange();
    $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT);
    $q = new AMQPQueue($amqp);
    $q->declare('email');
    $ex->bind('email', 'routing.key');
    while ($messages = $q->consume()) {
    foreach ($messages as $message) {
    $email = json_decode($message['message_body']);
    mail($email->to, $email->subject, $email->message, $email->headers);
    }
    }
    

    View Slide

55. MESSAGES
    

    View Slide

56. MESSAGE CONSIDERATIONS
    • Message Format
    • Message Contents
    

    View Slide

57. SERIALIZE
    O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s:
    3:"baz";}}}}
    

    View Slide

58. WORKERS
    

    View Slide

59. WORKER CONSIDERATIONS
    • Should do ONE thing and
    ONE thing well.
    • Should attempt to be as
    quick as possible in handling
    that type.
    • Should be able to be scaled
    horizontally.
    

    View Slide

60. HANDLING WORKERS
    • Prevent Memory Leaks
    • memory_get_usage
    • Handle Signals!
    • pcntl_signal
    

    View Slide

61. ABSTRACTIONS
    We want to make this easy…
    

    View Slide

62. SYMFONY MESSENGER
    

    View Slide

63. LARAVEL QUEUES
    

    View Slide

64. BERNARD
    General queue implementation
    

    View Slide

65. interface QueueInterface {
    public function __construct(Stomp $stomp, String $queue);
    public function dispatch();
    public function publish : bool(array $message);
    public function work(StompFrame $message);
    }
    

    View Slide

66. class AbstractQueue implements QueueInterface {
    protected $stomp;
    protected $queue;
    protected $signal;
    public function __construct(Stomp $stomp, string $queue) {
    $this->stomp = $stomp;
    $this->queue = $queue;
    }
    protected function prepare() {
    if (php_sapi_name() != 'cli') {
    throw new RuntimeException('You cannot dispatch outside of the CLI');
    }
    if (function_exists('pcntl_signal')) {
    pcntl_signal(SIGTERM, array($this, 'signal'));
    pcntl_signal(SIGINT, array($this, 'signal'));
    pcntl_signal(SIGHUP, array($this, 'signal'));
    }
    }
    protected function signal(int $signal) {
    $this->signal = $signal;
    }
    

    View Slide

67. public function dispatch() {
    $this->prepare();
    while (true) {
    if ($this->signal) {
    break ;
    }
    if (!$this->stomp->hasFrame()) {
    $this->wait();
    continue ;
    }
    $frame = $this->stomp->readFrame();
    if ($this->validate($frame)) {
    $this->work($frame);
    }
    $this->stomp->ack($frame);
    }
    }
    protected function wait() {
    sleep(1);
    }
    protected function validate(StompFrame $message): bool {
    return false;
    }
    public function publish(array $message): bool {
    return $this->stomp->send($this->queue, json_encode($message));
    }
    

    View Slide

68. class EmailQueue extends AbstractQueue {
    public function validate(StompFrame $message): bool {
    if (!array_key_exists('to', $message)) {
    return false;
    }
    return true;
    }
    public function work(StompFrame $message) {
    $mail = json_decode($message);
    mail($mail->to, $mail->subject, $mail->message);
    }
    }
    

    View Slide

69. BOOTSTRAPPING
    Leverage your existing infrastructure as much as possible!
    

    View Slide

70. declare(ticks=1);
    include 'vendor/autoload.php';
    $app = Zend\Mvc\Application::init(include 'config/application.config.php');
    $sm = $app->getServiceManager();
    if (!isset($argv[1])) {
    fprintf(STDERR, "Syntax: worker \n\n");
    exit(1);
    }
    $name = $argv[1];
    try {
    echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL;
    $consumer = $sm->get($name);
    $consumer->dispatch();
    } catch (\Exception $e) {
    fprintf(STDERR, "%s\n", $msg);
    exit(1);
    }
    $consumer = null;
    echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL;
    exit(0);
    

    View Slide

71. EVENTS
    

    View Slide

72. SERVICES TRIGGER EVENTS
    use Zend\EventManager\EventManagerAwareTrait;
    class UserService
    {
    use EventManagerAwareTrait;
    public function save(MyUser $user) {
    $this->db->save($user);
    $this->getEventManager()->trigger('save', null, ['user' => $user]);
    }
    }
    

    View Slide

73. ATTACH EVENTS
    use Zend\ServiceManager\ServiceManager;
    $sm = new ServiceManager();
    $service = $sm->get('UserService');
    $queue = $sm->get('EmailQueue');
    $service->getEventManager()->attach('save', function($e) use ($queue) {
    $params = $e->getParams();
    $queue->publish(json_encode[
    'to' => $params['user']['email'],
    'subject' => 'Welcome',
    'message' => 'Welcome',
    'headers' => [],
    ]);
    });
    

    View Slide

74. HANDLING
    PROGRESS
    UPDATES
    • Keep track of item state
    using a cache server or
    database.
    • Utilize events on your
    processing to provide
    updates to the UI.
    

    View Slide

75. HOW MIGHT WE UPDATE?
    $message = json_decode($stomp->readFrame()->body);
    $proc = proc_open($command, $descriptorSpec, $pipes);
    stream_set_blocking($pipes[1], 0);
    while (!feof($pipes[1])) {
    $content = fgets($pipes[1], 1024);
    if (preg_match('/something-i-(care)-about/', $content, $results)) {
    // provide update to DB, cache, etc.
    }
    }
    

    View Slide

76. TOOLING
    

    View Slide

77. JENKINS
    • It's not just for your CI/CD
    projects.
    

    View Slide

78. JENKINS: AMAZON EC2
    • Install the Amazon EC2 Plugin
    • Leverage images and use spot instances
    • Configure spot instance to deploy latest code on
    boot
    • Configure jenkins to boot up additional spot
    instances if jobs are waiting.
    

    View Slide

79. JENKINS: JOBS
    • Handlers
    • 1 job per handler with:
    • Execute concurrent builds if necessary
    • Restrict where this project can be run (spot instance label).
    • Scheduled job to watch queue lengths and available workers
    • queue length > available workers after x time = execute job
    • queue length < available workers after x time = kill job
    

    View Slide

80. SYSTEMD
    • Integrated into the OS.
    • Monitors and handles
    starting, restarting and
    dependencies of processes.
    • Integrated logging.
    

    View Slide

81. SERVICE FILE EXAMPLE
    # /etc/systemd/system/my_worker.service
    [Unit]
    Description=My Worker Service
    After=network.target
    [Service]
    Type=simple
    User=unix-user
    WorkingDirectory=/path/to/working/dir
    ExecStart=/path/to/worker --opt1
    Restart=on-failure # or always, on-abort, etc
    [Install]
    WantedBy=multi-user.target
    

    View Slide

82. HOW TO USE IT?
    # controlling

    systemctl start|stop|status|enable|
    disable my-worker.service

    

    # logs

    journalctl -f -u my-worker.service
    

    View Slide

83. SUPERVISOR
    • Daemon that runs on the
    server.
    • Monitors programs and
    keeps them running in case
    of failure.
    • Handles logging.
    • If systemd is unavailable or a
    bit too scary :)
    

    View Slide

84. PAINLESS INSTALLATION
    sudo easy_install supervisor
    sudo echo_supervisord_conf > /etc/supervisord.conf
    sudo service supervisor start
    

    View Slide

85. EXAMPLE PROGRAM
    CONFIGURATION
    [program:emailworker]
    command=/usr/bin/php /var/www/worker "MyProject\Queue\Email"
    process_name=%(program_name)s_%(process_num)d
    numprocs=2
    numprocs_start=2
    user=www-data
    autostart=true ; start at supervisord start (default: true)
    autorestart=true ; retstart at unexpected quit (default: true)
    startsecs=10 ; number of secs prog must stay running (def. 10)
    startretries=5 ; max # of serial start failures (default 3)
    log_stdout=true ; if true, log program stdout (default true)
    log_stderr=true ; if true, log program stderr (def false)
    redirect_stderr=true ; if true, redirect stderr to stdout
    stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log
    stdout_logfile_maxbytes=10MB
    stdout_logfile_backups=15
    

    View Slide

86. SUPERVISORD MULTI SERVER MONITORING
    TOOL
    https://github.com/mlazarov/supervisord-monitor
    

    View Slide

87. WHEN BAD THINGS HAPPEN
    

    View Slide

88. QUEUE BACKUP
    • Queue length continues to
    grow and you're not
    processing enough...
    

    View Slide

89. WORKER EXCEPTIONS
    • Broken code, causes
    blowups.
    

    View Slide

90. THANK YOU
    • @mwillbanks
    • Provide your feedback!
    joind.in/talk/a6dfa
    

    View Slide