$30 off During Our Annual Pro Sale. View Details »

Asynchronous Data Processing

Asynchronous Data Processing

Choosing the right way to process data might become a strategic and non trivial decision for many kind of applications. Especially in applications where an high percentage of the time is spent elaborating informations behind the scenes. A solution could be to use a message queuing system for processing data asyncronously. Using AMQP it becomes possible to define different types of queue patterns in order to solve different types of problems.
The purpose of the talk is to show how to approach the diffentent scenarios in php adding a great value to your application.

Andrea Giuliano

September 27, 2015
Tweet

More Decks by Andrea Giuliano

Other Decks in Technology

Transcript

  1. A S Y N C H R O N O U S D ATA P R O C E S S I N G
    Andrea Giuliano
    @bit_shark
    B u l g a r i a P H P
    C O N F E R E N C E
    September 27th, 2015

    View Slide

  2. Andrea Giuliano
    @bit_shark
    $WHOAMI

    View Slide

  3. COMMUNICATION

    View Slide

  4. SYNCHRONOUS
    COMMUNICATION
    time
    system A
    system B
    Request
    Response
    blocked

    View Slide

  5. Pros Cons
    • easy to reason about
    • immediate feedback
    • client can get stuck
    • impractical in low latency env
    SYNCHRONOUS
    COMMUNICATION

    View Slide

  6. ASYNCHRONOUS
    COMMUNICATION
    time
    system A
    system B
    Request

    View Slide

  7. • useful for long-running jobs
    • systems decoupled
    • scales well (event-based)
    • additional system
    requirements
    ASYNCHRONOUS
    COMMUNICATION
    Pros Cons

    View Slide

  8. INTEROPERABILITY

    View Slide

  9. ADVANCED MESSAGE
    QUEUING PROTOCOL
    • message orientation
    • queuing
    • routing
    • reliability
    • security

    View Slide

  10. P R O D U C E R C O N S U M E R
    Produce Consumes
    B R O K E R
    ENTITIES INVOLVED

    View Slide

  11. P R O D U C E R C O N S U M E R
    Produce
    Exchange
    Consumes
    B R O K E R
    THE BROKER

    View Slide

  12. P R O D U C E R C O N S U M E R
    Produce
    Exchange
    Routes
    Consumes
    B R O K E R
    THE BROKER

    View Slide

  13. P R O D U C E R C O N S U M E R
    Produce
    Exchange Queue
    Routes
    Consumes
    B R O K E R
    THE BROKER

    View Slide

  14. MESSAGING PATTERNS

    View Slide

  15. {
    "require": {
    "videlalvaro/php-amqplib": "@stable",
    ...
    }
    }
    $ composer.phar install

    View Slide

  16. PRODUCER/CONSUMER

    View Slide

  17. P R O D U C E R C O N S U M E R
    Q U E U E
    PRODUCER/CONSUMER

    View Slide

  18. PRODUCING MESSAGES

    View Slide

  19. PRODUCING MESSAGES

    View Slide

  20. PRODUCING MESSAGES

    View Slide

  21. PRODUCING MESSAGES

    View Slide

  22. CONSUMING MESSAGES

    View Slide

  23. CONSUMING MESSAGES

    View Slide

  24. CONSUMING MESSAGES

    View Slide

  25. SCALING UP
    P R O D U C E R
    C O N S U M E R
    Q U E U E
    C O N S U M E R

    View Slide

  26. CONSUMERS CAN DIE

    View Slide

  27. MESSAGE
    ACKNOWLEDGEMENT

    View Slide

  28. MESSAGE ACKS
    switch on acks
    send ack

    View Slide

  29. QOS POLICY

    View Slide

  30. C O N S U M E R
    C O N S U M E R
    ROUND ROBIN DISPATCHING

    View Slide

  31. C O N S U M E R
    C O N S U M E R
    ROUND ROBIN DISPATCHING

    View Slide

  32. C O N S U M E R
    C O N S U M E R
    ROUND ROBIN DISPATCHING

    View Slide

  33. C O N S U M E R
    C O N S U M E R
    ROUND ROBIN DISPATCHING
    !

    View Slide

  34. FAIR DISPATCHING
    C O N S U M E R
    C O N S U M E R

    View Slide

  35. FAIR DISPATCHING
    C O N S U M E R
    C O N S U M E R

    View Slide

  36. FAIR DISPATCHING
    C O N S U M E R
    C O N S U M E R

    View Slide

  37. FAIR DISPATCHING
    C O N S U M E R
    C O N S U M E R
    ack

    View Slide

  38. FAIR DISPATCHING
    C O N S U M E R
    C O N S U M E R

    View Slide

  39. FAIR DISPATCHING

    View Slide

  40. BROKER CAN DIE

    View Slide

  41. DURABILITY

    View Slide

  42. DURABILITY
    declare the queue
    persistent

    View Slide

  43. PUBLISH/SUBSCRIBE

    View Slide

  44. PUBLISH/SUBSCRIBE
    P U B L I S H E R
    S U B S C R I B E R
    Q U E U E
    S U B S C R I B E R S U B S C R I B E R

    View Slide

  45. PUBLISH/SUBSCRIBE
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    tell me when a user is created
    tell me when a user is created
    tell me when a user is created

    View Slide

  46. PUBLISH/SUBSCRIBE
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    U S E R
    S E R V I C E
    B R O K E R
    username: agiuliano
    password: ******
    mail: ***@gmail.com
    UserCreated

    View Slide

  47. PUBLISH/SUBSCRIBE
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    UserCreated
    UserCreated
    UserCreated

    View Slide

  48. PUBLISH/SUBSCRIBE
    U S E R
    S E R V I C E
    B R O K E R
    UserCreated

    View Slide

  49. PUBLISH/SUBSCRIBE
    M A I L
    S E R V I C E
    B R O K E R
    tell me when a user is created

    View Slide

  50. PUBLISH/SUBSCRIBE
    UserCreated
    M A I L
    S E R V I C E
    B R O K E R

    View Slide

  51. ROUTING

    View Slide

  52. ROUTING
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [UserCreated, UserUpdated]
    UserCreated
    UserCreated

    View Slide

  53. ROUTING
    U S E R
    S E R V I C E
    B R O K E R
    UserUpdated
    routing key: updated

    View Slide

  54. ROUTING
    M A I L
    S E R V I C E
    B R O K E R
    routing keys:
    [created, updated]

    View Slide

  55. ROUTING
    M A I L
    S E R V I C E
    B R O K E R
    routing keys:
    [created, updated]

    View Slide

  56. TOPIC

    View Slide

  57. TOPIC
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]

    View Slide

  58. TOPIC

    View Slide

  59. TOPIC
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    U S E R
    S E R V I C E

    View Slide

  60. TOPIC
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    U S E R
    S E R V I C E

    View Slide

  61. TOPIC
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    U S E R
    S E R V I C E

    View Slide

  62. TOPIC
    M A I L
    S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    U S E R
    S E R V I C E

    View Slide

  63. TOPIC
    M A I L S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    S T O R E
    S E R V I C E

    View Slide

  64. TOPIC
    M A I L S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    S T O R E
    S E R V I C E

    View Slide

  65. TOPIC
    M A I L S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    S T O R E
    S E R V I C E

    View Slide

  66. TOPIC
    M A I L S E R V I C E
    A U T H
    S E R V I C E
    C O U P O N
    S E R V I C E
    B R O K E R
    [user. *]
    [user. created]
    [user. created, *.purchased]
    S T O R E
    S E R V I C E

    View Slide

  67. SUMMARY

    View Slide

  68. PERFORMANCE

    View Slide

  69. DECOUPLED SYSTEM

    View Slide

  70. INTEROPERABILITY

    View Slide

  71. SCALABILITY

    View Slide

  72. joind.in/14878
    Please rate the talk

    View Slide

  73. joind.in/14878

    View Slide

  74. CREDITS
    • www.flickr.com/photos/rayofsun/9401226342
    • www.flickr.com/photos/kalexanderson/6231391820
    • www.flickr.com/photos/dhammza/2891991931
    • www.flickr.com/photos/kevlewis/15372562472
    • www.flickr.com/photos/hohumhobo/6349729535
    • www.flickr.com/photos/107346206
    • www.flickr.com/photos/berlin_steve/8414515260
    • www.flickr.com/photos/s3a/4816966416
    • www.flickr.com/photos/thatguyfromcchs08/2300190277
    • www.flickr.com/photos/toniblay/52445415
    • www.flickr.com/photos/bobsfever/8226802321
    • www.flickr.com/photos/activars/6590201161
    • www.flickr.com/photos/michaeljzealot/6485009571

    View Slide