Asynchronous data processing (Confoo)

Choosing the right way to process data is a strategic decision. Especially in applications where an high percentage of the time is spent elaborating informations behind the scenes.
A solution is to use a message queuing system and run tasks asynchronously.
The purpose of the talk is to show how to use the AMQP in order to strengthen your application.

Transcript

1. A S Y N C H R O N O U S D ATA P R O C E S S I N G Andrea Giuliano @bit_shark February 24th, 2016
2. Andrea Giuliano @bit_shark $WHOAMI
3. COMMUNICATION
4. SYNCHRONOUS COMMUNICATION time system A system B Request Response blocked
5. Pros Cons • easy to reason about • immediate feedback • client can get stuck • impractical in low latency env SYNCHRONOUS COMMUNICATION
6. ASYNCHRONOUS COMMUNICATION time system A system B Request
7. • useful for long-running jobs • systems decoupled • scales well (event-based) • additional system requirements ASYNCHRONOUS COMMUNICATION Pros Cons
8. INTEROPERABILITY
9. ADVANCED MESSAGE QUEUING PROTOCOL • message orientation • queuing • routing • reliability • security
10. P R O D U C E R C O N S U M E R Produce Consumes B R O K E R ENTITIES INVOLVED
11. P R O D U C E R C O N S U M E R Produce Exchange Consumes B R O K E R THE BROKER
12. P R O D U C E R C O N S U M E R Produce Exchange Routes Consumes B R O K E R THE BROKER
13. P R O D U C E R C O N S U M E R Produce Exchange Queue Routes Consumes B R O K E R THE BROKER
14. MESSAGING PATTERNS
15. { "require": { "php-amqplib/php-amqplib": "@stable", ... } } $ composer.phar install
16. PRODUCER/CONSUMER
17. P R O D U C E R C O N S U M E R Q U E U E PRODUCER/CONSUMER
18. PRODUCING MESSAGES
19. PRODUCING MESSAGES
20. PRODUCING MESSAGES
21. PRODUCING MESSAGES
22. CONSUMING MESSAGES
23. CONSUMING MESSAGES
24. CONSUMING MESSAGES
25. SCALING UP P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R …
26. CONSUMERS CAN DIE
27. MESSAGE ACKNOWLEDGEMENT
28. MESSAGE ACKS switch on acks send ack
29. QOS POLICY
30. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING
31. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING
32. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING
33. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING !
34. FAIR DISPATCHING C O N S U M E R C O N S U M E R
35. FAIR DISPATCHING C O N S U M E R C O N S U M E R
36. FAIR DISPATCHING C O N S U M E R C O N S U M E R
37. FAIR DISPATCHING C O N S U M E R C O N S U M E R ack
38. FAIR DISPATCHING C O N S U M E R C O N S U M E R
39. FAIR DISPATCHING
40. BROKER CAN DIE
41. DURABILITY
42. DURABILITY declare the queue persistent
43. PUBLISH/SUBSCRIBE
44. PUBLISH/SUBSCRIBE P U B L I S H E R S U B S C R I B E R Q U E U E S U B S C R I B E R S U B S C R I B E R
45. PUBLISH/SUBSCRIBE M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R tell me when a user is created tell me when a user is created tell me when a user is created
46. PUBLISH/SUBSCRIBE M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E U S E R S E R V I C E B R O K E R username: agiuliano password: ****** mail: ***@gmail.com UserCreated
47. PUBLISH/SUBSCRIBE M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R UserCreated UserCreated UserCreated
48. PUBLISH/SUBSCRIBE U S E R S E R V I C E B R O K E R UserCreated
49. PUBLISH/SUBSCRIBE M A I L S E R V I C E B R O K E R tell me when a user is created
50. PUBLISH/SUBSCRIBE UserCreated M A I L S E R V I C E B R O K E R
51. ROUTING
52. ROUTING M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [UserCreated, UserUpdated] UserCreated UserCreated
53. ROUTING U S E R S E R V I C E B R O K E R UserUpdated routing key: updated
54. ROUTING M A I L S E R V I C E B R O K E R routing keys: [created, updated]
55. ROUTING M A I L S E R V I C E B R O K E R routing keys: [created, updated]
56. TOPIC
57. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased]
58. TOPIC
59. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
60. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
61. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
62. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
63. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
64. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
65. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
66. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
67. SUMMARY
68. PERFORMANCE
69. DECOUPLED SYSTEM
70. INTEROPERABILITY
71. SCALABILITY
74. CREDITS • www.flickr.com/photos/rayofsun/9401226342 • www.flickr.com/photos/kalexanderson/6231391820 • www.flickr.com/photos/dhammza/2891991931 • www.flickr.com/photos/kevlewis/15372562472 • www.flickr.com/photos/hohumhobo/6349729535 • www.flickr.com/photos/107346206 • www.flickr.com/photos/berlin_steve/8414515260 • www.flickr.com/photos/s3a/4816966416 • www.flickr.com/photos/thatguyfromcchs08/2300190277 • www.flickr.com/photos/toniblay/52445415 • www.flickr.com/photos/bobsfever/8226802321 • www.flickr.com/photos/activars/6590201161 • www.flickr.com/photos/michaeljzealot/6485009571