Asynchronous data processing (v1)

Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating information behind the scenes.

There are different message queuing systems designed to manage and process data asynchronously. Using simple messages, it becomes possible to define many types of queue patterns, from the straightforward solution to the more complex one like routing, publisher/subscriber and topic.

The purpose of the talk is to show how to approach the different scenarios in php, adding value to your application.

Transcript

1. A S Y N C H R O N O U S D ATA P R O C E S S I N G A N D R E A G I U L I A N O @bit_shark # C A K E F E S T
2. T H E P R O B L E M
3. P R O D U C T C ATA L O G G E N E R AT O R • 30k+ records in db representing products • dispose records in pages (with a certain order) • generate a 300+ pages PDF catalog T H E P R O B L E M
4. H E AV Y O P E R AT I O N S T O D O H E R E T H E P R O B L E M
5. U S E R S D O N ’ T WA N T T O WA I T T H E P R O B L E M
6. U S E R S D O N ’ T WA N T T O WA I T T H E P R O B L E M A N D I F T H E Y H AV E T O T H E Y D O N ’ T WA N T T O G E T S T U C K !
7. T H E P R O B L E M S Y N C H R O N Y time process A process B Request Response blocked
8. T H E N E E D S
9. P R E V E N T T I M E O U T T H E N E E D S
10. E V E N T U A L LY D E L I V E RY T H E N E E D S “sooner or later your job will be processed.”
11. “ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ” N O T I F I C AT I O N T H E N E E D S
12. T H E N E E D S A S Y N C H R O N Y time process A process B Request
13. I N T E R O P E R A B I L I T Y T H E N E E D S
14. A D VA N C E D M E S S A G E Q U E U I N G P R O T O C O L T H E S O L U T I O N
15. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Consumes B R O K E R
16. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Consumes B R O K E R
17. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Routes Consumes B R O K E R
18. H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Queue Routes Consumes B R O K E R
19. T H R O U G H C O M P O S E R I N S TA L L A M Q P L I B R A RY {          "require":  {                  "videlalvaro/php-­‐amqplib":  "@stable",                    ...          }   }   $ composer.phar install
20. D I F F E R E N T S C E N A R I O S
21. P R O D U C E R / C O N S U M E R S C E N A R I O
22. S C E N A R I O P R O D U C E R / C O N S U M E R P R O D U C E R C O N S U M E R Q U E U E
23. S C E N A R I O P R O D U C E R / C O N S U M E R use  PhpAmqpLib\Connection\AMQPConnection;   use  PhpAmqpLib\Message\AMQPMessage;       $connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);   $channel  =  $connection-­‐>channel(); S E T T I N G U P C O N N E C T I O N C R E AT E A C H A N N E L
24. S C E N A R I O P R O D U C E R / C O N S U M E R P R O D U C E R Q U E U E                                                                        passive    durable  exclusive  auto-­‐delete   $channel-­‐>queue_declare('catalog',  false,    false,    false,        false);         foreach  ($catalog-­‐>getPages()  as  $page)  {          $message  =  new  AMQPMessage($page);          $channel-­‐>basic_publish($message,  '',  'catalog');   }       $channel-­‐>close();   $connection-­‐>close();
25. S C E N A R I O P R O D U C E R / C O N S U M E R C O N S U M E R Q U E U E $connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);   $channel  =  $connection-­‐>channel();       $channel-­‐>queue_declare('catalog',  false,  false,  false,  false);       $callback  =  function($msg)  {          $msg-­‐>body-­‐>generatePdf();   };       $channel-­‐>basic_consume('catalog',  '',  false,  true,  false,  false,  $callback);       while(count($channel-­‐>callbacks))  {          $channel-­‐>wait();   }       $channel-­‐>close();   $connection-­‐>close();  
26. S C E N A R I O M U LT I P L E C O N S U M E R S P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R …
27. S C E N A R I O M U LT I P L E C O N S U M E R S P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R … parallelize work easy scalability
28. C O N S U M E R S C A N D I E
29. M E S S A G E A K N O W L E D G E M E N T
30. S A F E T Y M E S S A G E A C K S $callback  =  function($msg){          $msg-­‐>body-­‐>generatePdf();          $msg-­‐>delivery_info['channel']-­‐>basic_ack($msg-­‐>delivery_info['delivery_tag']);   };   ! $channel-­‐>basic_consume('catalog',  '',  false,  false,  false,  false,  $callback); S W I T C H A C K O N
31. D O N ’ T F O R G E T T O S E N D A C K S
32. T H E B R O K E R C A N D I E
33. D U R A B I L I T Y
34. D U R A B I L I T Y M A R K T H E Q U E U E A N D T H E C H A N N E L $channel-­‐>queue_declare('catalog',  false,  true,  false,  false); In order to achieve durability $message  =  new  AMQPMessage($data,                #the  message  is  now  persistent                array('delivery_mode'  =>  2)                  ); the queue must be declared durable the message must be marked as persistent
35. Q O S P O L I T I C S
36. Q O S P O L I T I C A B E FA I R C O N S U M E R C O N S U M E R for certain instances of the Round Robin dispatching B R O K E R #1 #3 #5 #2 #4 #6
37. Q O S P O L I T I C S B E FA I R $channel-­‐>basic_qos(null,  1,  null); C O N S U M E R C O N S U M E R the broker sends messages only when it receives acks B R O K E R #1 #3 #3 #2 #4
38. P U B L I S H / S U B S C R I B E S C E N A R I O
39. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom FA N O U T E X C H A N G E $connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);   $channel  =  $connection-­‐>channel();       $channel-­‐>exchange_declare('chatRoom',  'fanout',  false,  false,  false); Exchange … Setting up the connection and declare the fanout exchange
40. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Exchange … $data  =  getAMessageToSendInTheRoom();   $msg  =  new  AMQPMessage($data);       $channel-­‐>basic_publish($msg,  'chatRoom');       $channel-­‐>close();   $connection-­‐>close();   P R O D U C E R Produce Publishing a message to the exchange
41. N O W I T ’ S T H E S U B S C R I B E R ’ S T U R N
42. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Q U E U E B I N D I N G Exchange … //connection  setted  up   ! list($queue_name,  ,)  =  $channel-­‐>queue_declare("",  false,  false,  true,  false);       $channel-­‐>queue_bind($queue_name,  'chatRoom'); Bind the Queue on the Exchange amq.gen-A7d bind bind amq.gen-Sb4
43. S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Exchange … amq.gen-A7d $channel-­‐>basic_consume($queue_name,                    '',                    false,                    true,                    false,                    false,                    'readMessage');       $channel-­‐>close();   $connection-­‐>close();   C O N S U M E R Consumes amq.gen-Sb4
44. R O U T I N G S C E N A R I O
45. S C E N A R I O R O U T I N G chatRoom Exchange type=direct … amq.gen-A7d amq.gen-Sb4 Consumer1 ConsumerN P R O D U C E R Produce routing_keys = friends routing_keys = friends, colleagues $channel-­‐>exchange_declare('chatRoom',  'direct',  false,  false,  false);
46. S C E N A R I O R O U T I N G chatRoom Exchange type=direct P R O D U C E R Produce Producing messages //connection  setted  up   ! $channel-­‐>exchange_declare('chatRoom',                      'direct',                      false,                      false,                      false);         $data  =  getAMessageToSendInTheRoom('friends');   $msg  =  new  AMQPMessage($data);       $channel-­‐>basic_publish($msg,  'chatRoom',  'friends');       $channel-­‐>close();   $connection-­‐>close();  
47. S C E N A R I O R O U T I N G //connection  setted  up   //exchange  setted  up   //$queue_name  is  a  system  generated  queue  name   ! $rooms  =  array('friends',  'colleagues');       foreach($rooms  as  $room)  {          $channel-­‐>queue_bind($queue_name,  'chatRoom',  $room);   }       $channel-­‐>basic_consume($queue_name,  '',  false,  true,  false,  false,  'readMessage');       $channel-­‐>close();   $connection-­‐>close();   Bind a Consumer on different routing keys and consuming messages chatRoom amq.gen-Sb4 Consumer routing_keys = friends, colleagues
48. T O P I C S C E N A R I O
49. G O F U R T H E R T O P I C $channel-­‐>exchange_declare('vehicle',  'topic',  false,  false,  false); vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <type>.<vehicle>.<colour> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
50. A Y E L L O W S P O R T C A R
51. T O P I C A Y E L L O W S P O R T C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <sport>.<car>.<yellow> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
52. T O P I C A Y E L L O W S P O R T C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <sport>.<car>.<yellow> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
53. A R E D R A C E M O T O R B I K E
54. T O P I C A R E D R A C E M O T O R B I K E vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<motorbike>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
55. T O P I C A R E D R A C E M O T O R B I K E vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<motorbike>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
56. A R E D R A C E C A R
57. T O P I C A R E D R A C E C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<car>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
58. T O P I C A R E D R A C E C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <race>.<car>.<red> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
59. A B L U E C I T Y VA N
60. T O P I C A B L U E C I T Y VA N vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# <city>.<van>.<blue> routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y
61. D I S C A R D E D
63. R E M E M B E R
64. P R O C E S S I N G Y O U R D ATA A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY
65. E V E N I N P H P
66. C A N M A K E Y O U R ( D E V ) L I F E H A P P I E R
67. Y O U R A P P L I C AT I O N S T R O N G E R
68. Andrea Giuliano @bit_shark andreagiuliano.it
69. joind.in/11608 Please rate the talk!
70. C R E D I T S • https://www.flickr.com/photos/rayofsun/9401226342 • https://www.flickr.com/photos/thereeljames/11376085194 • https://www.flickr.com/photos/ollily/359817111 • https://www.flickr.com/photos/legofenris/4004170937 • https://www.flickr.com/photos/bsabarnowl/10993445723 • https://www.flickr.com/photos/jpott/2984914512 • https://www.flickr.com/photos/kalexanderson/6231391820 • https://www.flickr.com/photos/a2gemma/1448178195 • https://www.flickr.com/photos/dubpics/5619966355 • https://www.flickr.com/photos/umbertofistarol/5747425870 • https://www.flickr.com/photos/streetcarl/6888965017 • https://www.flickr.com/photos/infomastern/12407730413 • https://www.flickr.com/photos/giuseppemilo/11817936944 • https://www.flickr.com/photos/avardwoolaver/7137096221