Tweet
Tweet

Slide 1

Slide 1 text

A S Y N C H R O N O U S D ATA P R O C E S S I N G A N D R E A G I U L I A N O @bit_shark # C A K E F E S T

Slide 2

Slide 2 text

T H E P R O B L E M

Slide 3

Slide 3 text

P R O D U C T C ATA L O G G E N E R AT O R • 30k+ records in db representing products • dispose records in pages (with a certain order) • generate a 300+ pages PDF catalog T H E P R O B L E M

Slide 4

Slide 4 text

H E AV Y O P E R AT I O N S T O D O H E R E T H E P R O B L E M

Slide 5

Slide 5 text

U S E R S D O N ’ T WA N T T O WA I T T H E P R O B L E M

Slide 6

Slide 6 text

U S E R S D O N ’ T WA N T T O WA I T T H E P R O B L E M A N D I F T H E Y H AV E T O T H E Y D O N ’ T WA N T T O G E T S T U C K !

Slide 7

Slide 7 text

T H E P R O B L E M S Y N C H R O N Y time process A process B Request Response blocked

Slide 8

Slide 8 text

T H E N E E D S

Slide 9

Slide 9 text

P R E V E N T T I M E O U T T H E N E E D S

Slide 10

Slide 10 text

E V E N T U A L LY D E L I V E RY T H E N E E D S “sooner or later your job will be processed.”

Slide 11

Slide 11 text

“ E V E R I T H I N G ’ S G O N N A B E A L R I G H T ” N O T I F I C AT I O N T H E N E E D S

Slide 12

Slide 12 text

T H E N E E D S A S Y N C H R O N Y time process A process B Request

Slide 13

Slide 13 text

I N T E R O P E R A B I L I T Y T H E N E E D S

Slide 14

Slide 14 text

A D VA N C E D M E S S A G E Q U E U I N G P R O T O C O L T H E S O L U T I O N

Slide 15

Slide 15 text

H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Consumes B R O K E R

Slide 16

Slide 16 text

H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Consumes B R O K E R

Slide 17

Slide 17 text

H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Routes Consumes B R O K E R

Slide 18

Slide 18 text

H O W D O E S I T W O R K A M Q P P R O D U C E R C O N S U M E R Produce Exchange Queue Routes Consumes B R O K E R

Slide 19

Slide 19 text

T H R O U G H C O M P O S E R I N S TA L L A M Q P L I B R A RY {          "require":  {                  "videlalvaro/php-­‐amqplib":  "@stable",                    ...          }   }   $ composer.phar install

Slide 20

Slide 20 text

D I F F E R E N T S C E N A R I O S

Slide 21

Slide 21 text

P R O D U C E R / C O N S U M E R S C E N A R I O

Slide 22

Slide 22 text

S C E N A R I O P R O D U C E R / C O N S U M E R P R O D U C E R C O N S U M E R Q U E U E

Slide 23

Slide 23 text

S C E N A R I O P R O D U C E R / C O N S U M E R use  PhpAmqpLib\Connection\AMQPConnection;   use  PhpAmqpLib\Message\AMQPMessage;       $connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);   $channel  =  $connection-­‐>channel(); S E T T I N G U P C O N N E C T I O N C R E AT E A C H A N N E L

Slide 24

Slide 24 text

S C E N A R I O P R O D U C E R / C O N S U M E R P R O D U C E R Q U E U E                                                                        passive    durable  exclusive  auto-­‐delete   $channel-­‐>queue_declare('catalog',  false,    false,    false,        false);         foreach  ($catalog-­‐>getPages()  as  $page)  {          $message  =  new  AMQPMessage($page);          $channel-­‐>basic_publish($message,  '',  'catalog');   }       $channel-­‐>close();   $connection-­‐>close();

Slide 25

Slide 25 text

S C E N A R I O P R O D U C E R / C O N S U M E R C O N S U M E R Q U E U E $connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);   $channel  =  $connection-­‐>channel();       $channel-­‐>queue_declare('catalog',  false,  false,  false,  false);       $callback  =  function($msg)  {          $msg-­‐>body-­‐>generatePdf();   };       $channel-­‐>basic_consume('catalog',  '',  false,  true,  false,  false,  $callback);       while(count($channel-­‐>callbacks))  {          $channel-­‐>wait();   }       $channel-­‐>close();   $connection-­‐>close();  

Slide 26

Slide 26 text

S C E N A R I O M U LT I P L E C O N S U M E R S P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R …

Slide 27

Slide 27 text

S C E N A R I O M U LT I P L E C O N S U M E R S P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R … parallelize work easy scalability

Slide 28

Slide 28 text

C O N S U M E R S C A N D I E

Slide 29

Slide 29 text

M E S S A G E A K N O W L E D G E M E N T

Slide 30

Slide 30 text

S A F E T Y M E S S A G E A C K S $callback  =  function($msg){          $msg-­‐>body-­‐>generatePdf();          $msg-­‐>delivery_info['channel']-­‐>basic_ack($msg-­‐>delivery_info['delivery_tag']);   };   ! $channel-­‐>basic_consume('catalog',  '',  false,  false,  false,  false,  $callback); S W I T C H A C K O N

Slide 31

Slide 31 text

D O N ’ T F O R G E T T O S E N D A C K S

Slide 32

Slide 32 text

T H E B R O K E R C A N D I E

Slide 33

Slide 33 text

D U R A B I L I T Y

Slide 34

Slide 34 text

D U R A B I L I T Y M A R K T H E Q U E U E A N D T H E C H A N N E L $channel-­‐>queue_declare('catalog',  false,  true,  false,  false); In order to achieve durability $message  =  new  AMQPMessage($data,                #the  message  is  now  persistent                array('delivery_mode'  =>  2)                  ); the queue must be declared durable the message must be marked as persistent

Slide 35

Slide 35 text

Q O S P O L I T I C S

Slide 36

Slide 36 text

Q O S P O L I T I C A B E FA I R C O N S U M E R C O N S U M E R for certain instances of the Round Robin dispatching B R O K E R #1 #3 #5 #2 #4 #6

Slide 37

Slide 37 text

Q O S P O L I T I C S B E FA I R $channel-­‐>basic_qos(null,  1,  null); C O N S U M E R C O N S U M E R the broker sends messages only when it receives acks B R O K E R #1 #3 #3 #2 #4

Slide 38

Slide 38 text

P U B L I S H / S U B S C R I B E S C E N A R I O

Slide 39

Slide 39 text

S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom FA N O U T E X C H A N G E $connection  =  new  AMQPConnection(HOST,  PORT,  USER,  PASSWORD);   $channel  =  $connection-­‐>channel();       $channel-­‐>exchange_declare('chatRoom',  'fanout',  false,  false,  false); Exchange … Setting up the connection and declare the fanout exchange

Slide 40

Slide 40 text

S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Exchange … $data  =  getAMessageToSendInTheRoom();   $msg  =  new  AMQPMessage($data);       $channel-­‐>basic_publish($msg,  'chatRoom');       $channel-­‐>close();   $connection-­‐>close();   P R O D U C E R Produce Publishing a message to the exchange

Slide 41

Slide 41 text

N O W I T ’ S T H E S U B S C R I B E R ’ S T U R N

Slide 42

Slide 42 text

S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Q U E U E B I N D I N G Exchange … //connection  setted  up   ! list($queue_name,  ,)  =  $channel-­‐>queue_declare("",  false,  false,  true,  false);       $channel-­‐>queue_bind($queue_name,  'chatRoom'); Bind the Queue on the Exchange amq.gen-A7d bind bind amq.gen-Sb4

Slide 43

Slide 43 text

S C E N A R I O P U B L I S H / S U B S C R I B E chatRoom Exchange … amq.gen-A7d $channel-­‐>basic_consume($queue_name,                    '',                    false,                    true,                    false,                    false,                    'readMessage');       $channel-­‐>close();   $connection-­‐>close();   C O N S U M E R Consumes amq.gen-Sb4

Slide 44

Slide 44 text

R O U T I N G S C E N A R I O

Slide 45

Slide 45 text

S C E N A R I O R O U T I N G chatRoom Exchange type=direct … amq.gen-A7d amq.gen-Sb4 Consumer1 ConsumerN P R O D U C E R Produce routing_keys = friends routing_keys = friends, colleagues $channel-­‐>exchange_declare('chatRoom',  'direct',  false,  false,  false);

Slide 46

Slide 46 text

S C E N A R I O R O U T I N G chatRoom Exchange type=direct P R O D U C E R Produce Producing messages //connection  setted  up   ! $channel-­‐>exchange_declare('chatRoom',                      'direct',                      false,                      false,                      false);         $data  =  getAMessageToSendInTheRoom('friends');   $msg  =  new  AMQPMessage($data);       $channel-­‐>basic_publish($msg,  'chatRoom',  'friends');       $channel-­‐>close();   $connection-­‐>close();  

Slide 47

Slide 47 text

S C E N A R I O R O U T I N G //connection  setted  up   //exchange  setted  up   //$queue_name  is  a  system  generated  queue  name   ! $rooms  =  array('friends',  'colleagues');       foreach($rooms  as  $room)  {          $channel-­‐>queue_bind($queue_name,  'chatRoom',  $room);   }       $channel-­‐>basic_consume($queue_name,  '',  false,  true,  false,  false,  'readMessage');       $channel-­‐>close();   $connection-­‐>close();   Bind a Consumer on different routing keys and consuming messages chatRoom amq.gen-Sb4 Consumer routing_keys = friends, colleagues

Slide 48

Slide 48 text

T O P I C S C E N A R I O

Slide 49

Slide 49 text

G O F U R T H E R T O P I C $channel-­‐>exchange_declare('vehicle',  'topic',  false,  false,  false); vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 50

Slide 50 text

A Y E L L O W S P O R T C A R

Slide 51

Slide 51 text

T O P I C A Y E L L O W S P O R T C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 52

Slide 52 text

T O P I C A Y E L L O W S P O R T C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 53

Slide 53 text

A R E D R A C E M O T O R B I K E

Slide 54

Slide 54 text

T O P I C A R E D R A C E M O T O R B I K E vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 55

Slide 55 text

T O P I C A R E D R A C E M O T O R B I K E vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 56

Slide 56 text

A R E D R A C E C A R

Slide 57

Slide 57 text

T O P I C A R E D R A C E C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 58

Slide 58 text

T O P I C A R E D R A C E C A R vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 59

Slide 59 text

A B L U E C I T Y VA N

Slide 60

Slide 60 text

T O P I C A B L U E C I T Y VA N vehicle Exchange type=topic amq.gen-A7d amq.gen-Sb4 Consumer1 Consumer2 P R O D U C E R Produce routing_keys = *.car.* routing_keys = race.# .. routing_keys = *.*.red D O T S D E L I M I T E D R O U T I N G K E Y

Slide 61

Slide 61 text

D I S C A R D E D

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

R E M E M B E R

Slide 64

Slide 64 text

P R O C E S S I N G Y O U R D ATA A S Y N C H R O N O U S LY C A N B E D O N E E A S I LY

Slide 65

Slide 65 text

E V E N I N P H P

Slide 66

Slide 66 text

C A N M A K E Y O U R ( D E V ) L I F E H A P P I E R

Slide 67

Slide 67 text

Y O U R A P P L I C AT I O N S T R O N G E R

Slide 68

Slide 68 text

Andrea Giuliano @bit_shark andreagiuliano.it

Slide 69

Slide 69 text

joind.in/11608 Please rate the talk!

Slide 70

Slide 70 text

C R E D I T S • https://www.flickr.com/photos/rayofsun/9401226342 • https://www.flickr.com/photos/thereeljames/11376085194 • https://www.flickr.com/photos/ollily/359817111 • https://www.flickr.com/photos/legofenris/4004170937 • https://www.flickr.com/photos/bsabarnowl/10993445723 • https://www.flickr.com/photos/jpott/2984914512 • https://www.flickr.com/photos/kalexanderson/6231391820 • https://www.flickr.com/photos/a2gemma/1448178195 • https://www.flickr.com/photos/dubpics/5619966355 • https://www.flickr.com/photos/umbertofistarol/5747425870 • https://www.flickr.com/photos/streetcarl/6888965017 • https://www.flickr.com/photos/infomastern/12407730413 • https://www.flickr.com/photos/giuseppemilo/11817936944 • https://www.flickr.com/photos/avardwoolaver/7137096221