Asynchronous Data Processing

Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating informations behind the scenes. A solution could be to use a message queuing system for processing data asyncronously. Using AMQP it becomes possible to define different types of queue patterns in order to solve different types of problems.
The purpose of the talk is to show how to approach the diffentent scenarios in php adding a great value to your application.

Transcript

1. A S Y N C H R O N O U S D ATA P R O C E S S I N G Andrea Giuliano @bit_shark B u l g a r i a P H P C O N F E R E N C E September 27th, 2015
2. Andrea Giuliano @bit_shark $WHOAMI
3. COMMUNICATION
4. SYNCHRONOUS COMMUNICATION time system A system B Request Response blocked
5. Pros Cons • easy to reason about • immediate feedback • client can get stuck • impractical in low latency env SYNCHRONOUS COMMUNICATION
6. ASYNCHRONOUS COMMUNICATION time system A system B Request
7. • useful for long-running jobs • systems decoupled • scales well (event-based) • additional system requirements ASYNCHRONOUS COMMUNICATION Pros Cons
8. INTEROPERABILITY
9. ADVANCED MESSAGE QUEUING PROTOCOL • message orientation • queuing • routing • reliability • security
10. P R O D U C E R C O N S U M E R Produce Consumes B R O K E R ENTITIES INVOLVED
11. P R O D U C E R C O N S U M E R Produce Exchange Consumes B R O K E R THE BROKER
12. P R O D U C E R C O N S U M E R Produce Exchange Routes Consumes B R O K E R THE BROKER
13. P R O D U C E R C O N S U M E R Produce Exchange Queue Routes Consumes B R O K E R THE BROKER
14. MESSAGING PATTERNS
15. { "require": { "videlalvaro/php-amqplib": "@stable", ... } } $ composer.phar install
16. PRODUCER/CONSUMER
17. P R O D U C E R C O N S U M E R Q U E U E PRODUCER/CONSUMER
18. PRODUCING MESSAGES
19. PRODUCING MESSAGES
20. PRODUCING MESSAGES
21. PRODUCING MESSAGES
22. CONSUMING MESSAGES
23. CONSUMING MESSAGES
24. CONSUMING MESSAGES
25. SCALING UP P R O D U C E R C O N S U M E R Q U E U E C O N S U M E R …
26. CONSUMERS CAN DIE
27. MESSAGE ACKNOWLEDGEMENT
28. MESSAGE ACKS switch on acks send ack
29. QOS POLICY
30. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING
31. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING
32. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING
33. C O N S U M E R C O N S U M E R ROUND ROBIN DISPATCHING !
34. FAIR DISPATCHING C O N S U M E R C O N S U M E R
35. FAIR DISPATCHING C O N S U M E R C O N S U M E R
36. FAIR DISPATCHING C O N S U M E R C O N S U M E R
37. FAIR DISPATCHING C O N S U M E R C O N S U M E R ack
38. FAIR DISPATCHING C O N S U M E R C O N S U M E R
39. FAIR DISPATCHING
40. BROKER CAN DIE
41. DURABILITY
42. DURABILITY declare the queue persistent
43. PUBLISH/SUBSCRIBE
44. PUBLISH/SUBSCRIBE P U B L I S H E R S U B S C R I B E R Q U E U E S U B S C R I B E R S U B S C R I B E R
45. PUBLISH/SUBSCRIBE M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R tell me when a user is created tell me when a user is created tell me when a user is created
46. PUBLISH/SUBSCRIBE M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E U S E R S E R V I C E B R O K E R username: agiuliano password: ****** mail: ***@gmail.com UserCreated
47. PUBLISH/SUBSCRIBE M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R UserCreated UserCreated UserCreated
48. PUBLISH/SUBSCRIBE U S E R S E R V I C E B R O K E R UserCreated
49. PUBLISH/SUBSCRIBE M A I L S E R V I C E B R O K E R tell me when a user is created
50. PUBLISH/SUBSCRIBE UserCreated M A I L S E R V I C E B R O K E R
51. ROUTING
52. ROUTING M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [UserCreated, UserUpdated] UserCreated UserCreated
53. ROUTING U S E R S E R V I C E B R O K E R UserUpdated routing key: updated
54. ROUTING M A I L S E R V I C E B R O K E R routing keys: [created, updated]
55. ROUTING M A I L S E R V I C E B R O K E R routing keys: [created, updated]
56. TOPIC
57. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased]
58. TOPIC
59. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
60. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. created>
61. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
62. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] U S E R S E R V I C E <user. updated>
63. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
64. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.purchased>
65. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
66. TOPIC M A I L S E R V I C E A U T H S E R V I C E C O U P O N S E R V I C E B R O K E R [user. *] [user. created] [user. created, *.purchased] S T O R E S E R V I C E <item.created>
67. SUMMARY
68. PERFORMANCE
69. DECOUPLED SYSTEM
70. INTEROPERABILITY
71. SCALABILITY
72. joind.in/14878 Please rate the talk
73. joind.in/14878
74. CREDITS • www.flickr.com/photos/rayofsun/9401226342 • www.flickr.com/photos/kalexanderson/6231391820 • www.flickr.com/photos/dhammza/2891991931 • www.flickr.com/photos/kevlewis/15372562472 • www.flickr.com/photos/hohumhobo/6349729535 • www.flickr.com/photos/107346206 • www.flickr.com/photos/berlin_steve/8414515260 • www.flickr.com/photos/s3a/4816966416 • www.flickr.com/photos/thatguyfromcchs08/2300190277 • www.flickr.com/photos/toniblay/52445415 • www.flickr.com/photos/bobsfever/8226802321 • www.flickr.com/photos/activars/6590201161 • www.flickr.com/photos/michaeljzealot/6485009571