Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Asynchronous Data Processing

Asynchronous Data Processing

Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating informations behind the scenes. A solution could be to use a message queuing system for processing data asyncronously. Using AMQP it becomes possible to define different types of queue patterns in order to solve different types of problems.
The purpose of the talk is to show how to approach the diffentent scenarios in php adding a great value to your application.

21935bc100d813458e925f4a0bef80db?s=128

Andrea Giuliano

September 27, 2015
Tweet

Transcript

  1. A S Y N C H R O N O

    U S D ATA P R O C E S S I N G Andrea Giuliano @bit_shark B u l g a r i a P H P C O N F E R E N C E September 27th, 2015
  2. Andrea Giuliano @bit_shark $WHOAMI

  3. COMMUNICATION

  4. SYNCHRONOUS COMMUNICATION time system A system B Request Response blocked

  5. Pros Cons • easy to reason about • immediate feedback

    • client can get stuck • impractical in low latency env SYNCHRONOUS COMMUNICATION
  6. ASYNCHRONOUS COMMUNICATION time system A system B Request

  7. • useful for long-running jobs • systems decoupled • scales

    well (event-based) • additional system requirements ASYNCHRONOUS COMMUNICATION Pros Cons
  8. INTEROPERABILITY

  9. ADVANCED MESSAGE QUEUING PROTOCOL • message orientation • queuing •

    routing • reliability • security
  10. P R O D U C E R C O

    N S U M E R Produce Consumes B R O K E R ENTITIES INVOLVED
  11. P R O D U C E R C O

    N S U M E R Produce Exchange Consumes B R O K E R THE BROKER
  12. P R O D U C E R C O

    N S U M E R Produce Exchange Routes Consumes B R O K E R THE BROKER
  13. P R O D U C E R C O

    N S U M E R Produce Exchange Queue Routes Consumes B R O K E R THE BROKER
  14. MESSAGING PATTERNS

  15. { "require": { "videlalvaro/php-amqplib": "@stable", ... } } $ composer.phar

    install
  16. PRODUCER/CONSUMER

  17. P R O D U C E R C O

    N S U M E R Q U E U E PRODUCER/CONSUMER
  18. PRODUCING MESSAGES

  19. PRODUCING MESSAGES

  20. PRODUCING MESSAGES

  21. PRODUCING MESSAGES

  22. CONSUMING MESSAGES

  23. CONSUMING MESSAGES

  24. CONSUMING MESSAGES

  25. SCALING UP P R O D U C E R

    C O N S U M E R Q U E U E C O N S U M E R …
  26. CONSUMERS CAN DIE

  27. MESSAGE ACKNOWLEDGEMENT

  28. MESSAGE ACKS switch on acks send ack

  29. QOS POLICY

  30. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING
  31. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING
  32. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING
  33. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING !
  34. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  35. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  36. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  37. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R ack
  38. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  39. FAIR DISPATCHING

  40. BROKER CAN DIE

  41. DURABILITY

  42. DURABILITY declare the queue persistent

  43. PUBLISH/SUBSCRIBE

  44. PUBLISH/SUBSCRIBE P U B L I S H E R

    S U B S C R I B E R Q U E U E S U B S C R I B E R S U B S C R I B E R
  45. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R tell me when a user is created tell me when a user is created tell me when a user is created
  46. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E U S E R S E R V I C E B R O K E R username: agiuliano password: ****** mail: ***@gmail.com UserCreated
  47. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R UserCreated UserCreated UserCreated
  48. PUBLISH/SUBSCRIBE U S E R S E R V I

    C E B R O K E R UserCreated
  49. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E B R O K E R tell me when a user is created
  50. PUBLISH/SUBSCRIBE UserCreated M A I L S E R V

    I C E B R O K E R
  51. ROUTING

  52. ROUTING M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [UserCreated, UserUpdated] UserCreated UserCreated
  53. ROUTING U S E R S E R V I

    C E B R O K E R UserUpdated routing key: updated
  54. ROUTING M A I L S E R V I

    C E B R O K E R routing keys: [created, updated]
  55. ROUTING M A I L S E R V I

    C E B R O K E R routing keys: [created, updated]
  56. TOPIC

  57. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased]
  58. TOPIC

  59. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
  60. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
  61. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
  62. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
  63. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
  64. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
  65. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
  66. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
  67. SUMMARY

  68. PERFORMANCE

  69. DECOUPLED SYSTEM

  70. INTEROPERABILITY

  71. SCALABILITY

  72. joind.in/14878 Please rate the talk

  73. joind.in/14878

  74. CREDITS • www.flickr.com/photos/rayofsun/9401226342 • www.flickr.com/photos/kalexanderson/6231391820 • www.flickr.com/photos/dhammza/2891991931 • www.flickr.com/photos/kevlewis/15372562472 •

    www.flickr.com/photos/hohumhobo/6349729535 • www.flickr.com/photos/107346206 • www.flickr.com/photos/berlin_steve/8414515260 • www.flickr.com/photos/s3a/4816966416 • www.flickr.com/photos/thatguyfromcchs08/2300190277 • www.flickr.com/photos/toniblay/52445415 • www.flickr.com/photos/bobsfever/8226802321 • www.flickr.com/photos/activars/6590201161 • www.flickr.com/photos/michaeljzealot/6485009571