Asynchronous data processing (Confoo)

Asynchronous data processing (Confoo)

Choosing the right way to process data is a strategic decision. Especially in applications where an high percentage of the time is spent elaborating informations behind the scenes.
A solution is to use a message queuing system and run tasks asynchronously.
The purpose of the talk is to show how to use the AMQP in order to strengthen your application.

21935bc100d813458e925f4a0bef80db?s=128

Andrea Giuliano

February 24, 2016
Tweet

Transcript

  1. A S Y N C H R O N O

    U S D ATA P R O C E S S I N G Andrea Giuliano @bit_shark February 24th, 2016
  2. Andrea Giuliano @bit_shark $WHOAMI

  3. COMMUNICATION

  4. SYNCHRONOUS COMMUNICATION time system A system B Request Response blocked

  5. Pros Cons • easy to reason about • immediate feedback

    • client can get stuck • impractical in low latency env SYNCHRONOUS COMMUNICATION
  6. ASYNCHRONOUS COMMUNICATION time system A system B Request

  7. • useful for long-running jobs • systems decoupled • scales

    well (event-based) • additional system requirements ASYNCHRONOUS COMMUNICATION Pros Cons
  8. INTEROPERABILITY

  9. ADVANCED MESSAGE QUEUING PROTOCOL • message orientation • queuing •

    routing • reliability • security
  10. P R O D U C E R C O

    N S U M E R Produce Consumes B R O K E R ENTITIES INVOLVED
  11. P R O D U C E R C O

    N S U M E R Produce Exchange Consumes B R O K E R THE BROKER
  12. P R O D U C E R C O

    N S U M E R Produce Exchange Routes Consumes B R O K E R THE BROKER
  13. P R O D U C E R C O

    N S U M E R Produce Exchange Queue Routes Consumes B R O K E R THE BROKER
  14. MESSAGING PATTERNS

  15. { "require": { "php-amqplib/php-amqplib": "@stable", ... } } $ composer.phar

    install
  16. PRODUCER/CONSUMER

  17. P R O D U C E R C O

    N S U M E R Q U E U E PRODUCER/CONSUMER
  18. PRODUCING MESSAGES

  19. PRODUCING MESSAGES

  20. PRODUCING MESSAGES

  21. PRODUCING MESSAGES

  22. CONSUMING MESSAGES

  23. CONSUMING MESSAGES

  24. CONSUMING MESSAGES

  25. SCALING UP P R O D U C E R

    C O N S U M E R Q U E U E C O N S U M E R …
  26. CONSUMERS CAN DIE

  27. MESSAGE ACKNOWLEDGEMENT

  28. MESSAGE ACKS switch on acks send ack

  29. QOS POLICY

  30. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING
  31. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING
  32. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING
  33. C O N S U M E R C O

    N S U M E R ROUND ROBIN DISPATCHING !
  34. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  35. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  36. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  37. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R ack
  38. FAIR DISPATCHING C O N S U M E R

    C O N S U M E R
  39. FAIR DISPATCHING

  40. BROKER CAN DIE

  41. DURABILITY

  42. DURABILITY declare the queue persistent

  43. PUBLISH/SUBSCRIBE

  44. PUBLISH/SUBSCRIBE P U B L I S H E R

    S U B S C R I B E R Q U E U E S U B S C R I B E R S U B S C R I B E R
  45. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R tell me when a user is created tell me when a user is created tell me when a user is created
  46. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E U S E R S E R V I C E B R O K E R username: agiuliano password: ****** mail: ***@gmail.com UserCreated
  47. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R UserCreated UserCreated UserCreated
  48. PUBLISH/SUBSCRIBE U S E R S E R V I

    C E B R O K E R UserCreated
  49. PUBLISH/SUBSCRIBE M A I L S E R V I

    C E B R O K E R tell me when a user is created
  50. PUBLISH/SUBSCRIBE UserCreated M A I L S E R V

    I C E B R O K E R
  51. ROUTING

  52. ROUTING M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [UserCreated, UserUpdated] UserCreated UserCreated
  53. ROUTING U S E R S E R V I

    C E B R O K E R UserUpdated routing key: updated
  54. ROUTING M A I L S E R V I

    C E B R O K E R routing keys: [created, updated]
  55. ROUTING M A I L S E R V I

    C E B R O K E R routing keys: [created, updated]
  56. TOPIC

  57. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased]
  58. TOPIC

  59. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
  60. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
  61. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
  62. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
  63. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
  64. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
  65. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
  66. TOPIC M A I L S E R V I

    C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
  67. SUMMARY

  68. PERFORMANCE

  69. DECOUPLED SYSTEM

  70. INTEROPERABILITY

  71. SCALABILITY

  72. None
  73. None
  74. CREDITS • www.flickr.com/photos/rayofsun/9401226342 • www.flickr.com/photos/kalexanderson/6231391820 • www.flickr.com/photos/dhammza/2891991931 • www.flickr.com/photos/kevlewis/15372562472 •

    www.flickr.com/photos/hohumhobo/6349729535 • www.flickr.com/photos/107346206 • www.flickr.com/photos/berlin_steve/8414515260 • www.flickr.com/photos/s3a/4816966416 • www.flickr.com/photos/thatguyfromcchs08/2300190277 • www.flickr.com/photos/toniblay/52445415 • www.flickr.com/photos/bobsfever/8226802321 • www.flickr.com/photos/activars/6590201161 • www.flickr.com/photos/michaeljzealot/6485009571