Queue It

Transcript

1. QUEUE IT!

2. Task Producer Consumer Messages Messages Messages WHAT IS A QUEUE?

   • Pub/Sub • FIFO buffer • Push / Pull • A way to communicate between applications / systems. • A way to decouple components. • A way to offload work.

3. WHY QUEUE?

4. THE LONG RUNNING CASE Request Long-process Send Response Response

5. THE INTEROP CASE Request Conditioning Service Response Response Web Service

   Call Send Response

6. NOT YOUR DATABASE Request Process Response Send Response DB

7. IOT Data S3 Possible Action Process Event

8. SO WHY QUEUE • User experience • System Security •

   Load Distribution • System Reliability

9. IN PRACTICE You’ve seen this before…

10. None
11. None
12. None

13. WHAT YOU MIGHT QUEUE

14. COMMUNICATIONS • Emails • SMS • Push Notifications

15. IMAGES • Conversions • Resize • Thumbnail • Watermark

16. VIDEO • Conversion • Resampling • Audio overlay

17. IOT • Receive messages from devices and process responses

18. PATTERNS

19. POINT TO POINT Point to Point Channel Receiver Sender

20. PUBLISH / SUBSCRIBE Publiser Subscribe Channel Subscriber Subscriber Subscriber

21. MESSAGE BUS Application Application Application Message Bus

22. PIPELINE Sender Receiver Point to Point Channel Receiver Point to

    Point Channel

23. INVALID MESSAGE Channel Receiver Sender X Invalid Message Channel

24. PROTOCOLS Or implementations for that matter.

25. AMQP • AMQP Working Group (Community and Vendor) • Platform

    agnostic protocol. • Completely open, interoperable and broadly applicable. • Many severs available and many client libraries. • Best generally known in RabbitMQ.
26. None

27. KAFKA • Works with "streams" of data. • Limited messaging

    patterns.
28. None

29. STOMP • Simple protocol • Behaviors follow very simple commands.

    • Most message queues can communicate over STOMP.

30. Connect Send Disconnect /queue/ msg P H P S T

    O M P S E R V E R Connect Subscribe Disconnect /queue/ msg Read Ack

31. SQS • Simplistic protocol, HTTP- based. • Supports delays, timers,

    and multiple policies. • Combine with SNS to implement patterns.
32. None

33. SPECIAL PURPOSE Many queue implementations exist that don’t necessarily sit

    under standards…

34. ZEROMQ • The ultimate in message queue flexibility. • Socket

    library that acts as a concurrency framework.

35. SOCKET.IO • Real-time bidirectional event-based communication • Largely leverages pub/sub

36. XMPP • Best for real-time data. • Leveraging pub/sub can

    turn it into more of a generic message system. • Multiple libraries available.

37. BEANSTALKD • Asynchronous Job Queue • Supports delays • Many

    PHP clients exist

38. REDIS • Supports pub/sub • Supports atomic lists which people

    build queues off of

39. YOUR DATABASE • The poor mans queue. • Generally leveraging

    memory tables.

40. A list of 30+ message queue implementations. NOTE: NOT ALL

    ARE "REAL" MESSAGE QUEUES. queues.io

41. CONSIDERATIONS How do we evaluate our options…

42. PULL VS. PUSH • Always PULL, whenever possible in back-end

    applications. • Push eliminates several benefits, however, can be useful with web-workers and front-end applications.

43. DURABILITY • Memory residence • Persistence • Restart survival

44. SECURITY • Authentication • Queue permissions / restrictions

45. DELIVERY • Is the delivery guaranteed? • If a message

    cannot be delivered how it it handled?

46. ROUTING • Multiple routing scenarios • Fanout • Direct •

    Topic • Broadcast

47. BATCHING • Do it later but in bulk (credit card

    processing) • Can be done via scheduling (jenkins)

48. RECEIPT • Do you get an acknowledgement of receipt?

49. IMPLEMENTING QUEUES

50. STARTING POINTS

51. PUSHING VIA STOMP <?php class UserService { public function save(MyUser

    $user) { $this->db->save(MyUser $user); $stomp = new Stomp('tcp://localhost:61613'); $stomp->send('/queue/email', json_encode([ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ])); } }

52. FETCHING VIA STOMP <?php $stomp = new Stomp('tcp://localhost:61613'); $stomp->subscribe('/queue/email'); while

    (true) { if (!$stomp->hasFrame()) { sleep(2); continue ; } $stomp->readFrame(); $email = json_decode($frame->body); mail($email->to, $email->subject, $email->message, $email->headers); }

53. PUSHING VIA AMQP <?php class UserService { public function save(MyUser

    $user) { $this->db->save(MyUser $user); $amqp = new AMQPConnection(); $amqp->connect(); $ex = new AMQPExchange(); $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT); $q = new AMQPQueue($amqp); $q->declare('email'); $ex->bind('email', 'routing.key'); $ex->publish(json_encode([ 'to' => $user->getEmail(), 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]), 'routing.key'); } }

54. FETCHING VIA AMQP <?php $amqp = new AMQPConnection(); $amqp->connect(); $ex

    = new AMQPExchange(); $ex->declare('email-exchange', AMQP_EX_TYPE_FANOUT); $q = new AMQPQueue($amqp); $q->declare('email'); $ex->bind('email', 'routing.key'); while ($messages = $q->consume()) { foreach ($messages as $message) { $email = json_decode($message['message_body']); mail($email->to, $email->subject, $email->message, $email->headers); } }

55. MESSAGES

56. MESSAGE CONSIDERATIONS • Message Format • Message Contents

57. SERIALIZE O:7:"Message":1:{s:7:"content";a:1:{s:3:"foo";a:1:{s:3:"bar";a:1:{i:0;s: 3:"baz";}}}}

58. WORKERS

59. WORKER CONSIDERATIONS • Should do ONE thing and ONE thing

    well. • Should attempt to be as quick as possible in handling that type. • Should be able to be scaled horizontally.

60. HANDLING WORKERS • Prevent Memory Leaks • memory_get_usage • Handle

    Signals! • pcntl_signal

61. ABSTRACTIONS We want to make this easy…

62. SYMFONY MESSENGER

63. LARAVEL QUEUES

64. BERNARD General queue implementation

65. interface QueueInterface { public function __construct(Stomp $stomp, String $queue); public

    function dispatch(); public function publish : bool(array $message); public function work(StompFrame $message); }

66. class AbstractQueue implements QueueInterface { protected $stomp; protected $queue; protected

    $signal; public function __construct(Stomp $stomp, string $queue) { $this->stomp = $stomp; $this->queue = $queue; } protected function prepare() { if (php_sapi_name() != 'cli') { throw new RuntimeException('You cannot dispatch outside of the CLI'); } if (function_exists('pcntl_signal')) { pcntl_signal(SIGTERM, array($this, 'signal')); pcntl_signal(SIGINT, array($this, 'signal')); pcntl_signal(SIGHUP, array($this, 'signal')); } } protected function signal(int $signal) { $this->signal = $signal; }

67. public function dispatch() { $this->prepare(); while (true) { if ($this->signal)

    { break ; } if (!$this->stomp->hasFrame()) { $this->wait(); continue ; } $frame = $this->stomp->readFrame(); if ($this->validate($frame)) { $this->work($frame); } $this->stomp->ack($frame); } } protected function wait() { sleep(1); } protected function validate(StompFrame $message): bool { return false; } public function publish(array $message): bool { return $this->stomp->send($this->queue, json_encode($message)); }

68. class EmailQueue extends AbstractQueue { public function validate(StompFrame $message): bool

    { if (!array_key_exists('to', $message)) { return false; } return true; } public function work(StompFrame $message) { $mail = json_decode($message); mail($mail->to, $mail->subject, $mail->message); } }

69. BOOTSTRAPPING Leverage your existing infrastructure as much as possible!

70. <?php declare(ticks=1); include 'vendor/autoload.php'; $app = Zend\Mvc\Application::init(include 'config/application.config.php'); $sm =

    $app->getServiceManager(); if (!isset($argv[1])) { fprintf(STDERR, "Syntax: worker <name>\n\n"); exit(1); } $name = $argv[1]; try { echo "Starting worker: " . $name . ' as ' . get_current_user() . PHP_EOL; $consumer = $sm->get($name); $consumer->dispatch(); } catch (\Exception $e) { fprintf(STDERR, "%s\n", $msg); exit(1); } $consumer = null; echo 'Shutdown ' . $name . ' worker gracefully.' . PHP_EOL; exit(0);

71. EVENTS

72. SERVICES TRIGGER EVENTS use Zend\EventManager\EventManagerAwareTrait; class UserService { use EventManagerAwareTrait;

    public function save(MyUser $user) { $this->db->save($user); $this->getEventManager()->trigger('save', null, ['user' => $user]); } }

73. ATTACH EVENTS use Zend\ServiceManager\ServiceManager; $sm = new ServiceManager(); $service =

    $sm->get('UserService'); $queue = $sm->get('EmailQueue'); $service->getEventManager()->attach('save', function($e) use ($queue) { $params = $e->getParams(); $queue->publish(json_encode[ 'to' => $params['user']['email'], 'subject' => 'Welcome', 'message' => 'Welcome', 'headers' => [], ]); });

74. HANDLING PROGRESS UPDATES • Keep track of item state using

    a cache server or database. • Utilize events on your processing to provide updates to the UI.

75. HOW MIGHT WE UPDATE? $message = json_decode($stomp->readFrame()->body); $proc = proc_open($command,

    $descriptorSpec, $pipes); stream_set_blocking($pipes[1], 0); while (!feof($pipes[1])) { $content = fgets($pipes[1], 1024); if (preg_match('/something-i-(care)-about/', $content, $results)) { // provide update to DB, cache, etc. } }

76. TOOLING

77. JENKINS • It's not just for your CI/CD projects.

78. JENKINS: AMAZON EC2 • Install the Amazon EC2 Plugin •

    Leverage images and use spot instances • Configure spot instance to deploy latest code on boot • Configure jenkins to boot up additional spot instances if jobs are waiting.

79. JENKINS: JOBS • Handlers • 1 job per handler with:

    • Execute concurrent builds if necessary • Restrict where this project can be run (spot instance label). • Scheduled job to watch queue lengths and available workers • queue length > available workers after x time = execute job • queue length < available workers after x time = kill job

80. SYSTEMD • Integrated into the OS. • Monitors and handles

    starting, restarting and dependencies of processes. • Integrated logging.

81. SERVICE FILE EXAMPLE # /etc/systemd/system/my_worker.service [Unit] Description=My Worker Service After=network.target

    [Service] Type=simple User=unix-user WorkingDirectory=/path/to/working/dir ExecStart=/path/to/worker --opt1 Restart=on-failure # or always, on-abort, etc [Install] WantedBy=multi-user.target

82. HOW TO USE IT? # controlling
 systemctl start|stop|status|enable| disable my-worker.service


    
 # logs
 journalctl -f -u my-worker.service

83. SUPERVISOR • Daemon that runs on the server. • Monitors

    programs and keeps them running in case of failure. • Handles logging. • If systemd is unavailable or a bit too scary :)

84. PAINLESS INSTALLATION sudo easy_install supervisor sudo echo_supervisord_conf > /etc/supervisord.conf sudo

    service supervisor start

85. EXAMPLE PROGRAM CONFIGURATION [program:emailworker] command=/usr/bin/php /var/www/worker "MyProject\Queue\Email" process_name=%(program_name)s_%(process_num)d numprocs=2 numprocs_start=2

    user=www-data autostart=true ; start at supervisord start (default: true) autorestart=true ; retstart at unexpected quit (default: true) startsecs=10 ; number of secs prog must stay running (def. 10) startretries=5 ; max # of serial start failures (default 3) log_stdout=true ; if true, log program stdout (default true) log_stderr=true ; if true, log program stderr (def false) redirect_stderr=true ; if true, redirect stderr to stdout stdout_logfile=/var/www/logs/worker-panoramaqueuekrpano.log stdout_logfile_maxbytes=10MB stdout_logfile_backups=15

86. SUPERVISORD MULTI SERVER MONITORING TOOL https://github.com/mlazarov/supervisord-monitor

87. WHEN BAD THINGS HAPPEN

88. QUEUE BACKUP • Queue length continues to grow and you're

    not processing enough...

89. WORKER EXCEPTIONS • Broken code, causes blowups.

90. THANK YOU • @mwillbanks • Provide your feedback! joind.in/talk/a6dfa