Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From HTTP to Kafka-based Microservices (extended)

wrzasa
November 25, 2019

From HTTP to Kafka-based Microservices (extended)

How we switched microservices from HTTP to Kafka in FLYR Inc. (http://flyrlabs.com), presented on SegFault confenerce in Kraków (https://segfault.events/).

wrzasa

November 25, 2019
Tweet

More Decks by wrzasa

Other Decks in Programming

Transcript

  1. FROM HTTP TO KAFKA-BASED
    FROM HTTP TO KAFKA-BASED
    MICROSERVICES
    MICROSERVICES
    (EXTENDED)
    (EXTENDED)
    Wojciech Rząsa, FLYR Poland
    @wrzasa
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  2. Informatics specialist by passion and by
    profession
    15 years of academic work
    PhD but primarily an engineer
    FLYR Inc.
    Distributed systems
    Rzeszow Ruby User Group
    ABOUT ME
    ABOUT ME
    http://flyrlabs.com
    http://rrug.pl
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  3. FLYR
    FLYR
    Revenue management system for airlines
    Offices in
    San Francisco, USA (PST)
    Kraków, Poland (CEST)
    Machine Learning
    Microservices
    Python
    GCloud
    Kubernetes
    Big Query
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  4. FLYR DEVS ON PYCON CZ
    FLYR DEVS ON PYCON CZ
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  5. IN FLYR MICROSERVICES
    IN FLYR MICROSERVICES
    IPC based on HTTP
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  6. IN FLYR MICROSERVICES
    IN FLYR MICROSERVICES
    IPC based on HTTP
    New requirements for eCommerce use case
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  7. FAN-OUT REQUESTS
    FAN-OUT REQUESTS
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  8. PARTIAL RESPONSES
    PARTIAL RESPONSES
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  9. PARTIAL RESPONSES
    PARTIAL RESPONSES
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  10. IN FLYR MICROSERVICES
    IN FLYR MICROSERVICES
    IPC based on HTTP
    New requirements for eCommerce use case
    partial responses
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  11. IN FLYR MICROSERVICES
    IN FLYR MICROSERVICES
    IPC based on HTTP
    New requirements for eCommerce use case
    partial responses
    performance
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  12. IN FLYR MICROSERVICES
    IN FLYR MICROSERVICES
    IPC based on HTTP
    New requirements for eCommerce use case
    partial responses
    performance
    We will need MQs anyway in the future
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  13. OK, LET'S SWITCH FROM HTTP
    OK, LET'S SWITCH FROM HTTP
    TO... A... MQ?
    TO... A... MQ?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  14. BUT...
    BUT...
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  15. BUT...
    BUT...
    We have HTTP-based infrastructure
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  16. BUT...
    BUT...
    We have HTTP-based infrastructure
    We have HTTP developers experience and habits
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  17. BUT...
    BUT...
    We have HTTP-based infrastructure
    We have HTTP developers experience and habits
    We lack experience with MQ-based IPC
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  18. BUT...
    BUT...
    We have HTTP-based infrastructure
    We have HTTP developers experience and habits
    We lack experience with MQ-based IPC
    We need to do it well ;-)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  19. WITH MQ WE GET
    WITH MQ WE GET
    Flexibility
    Reliability
    Scalability
    Robustness
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  20. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES...
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  21. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  22. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    Concurrency issues
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  23. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    Concurrency issues
    Race conditions
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  24. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    Concurrency issues
    Race conditions
    Incorrect broker choice
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  25. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    Concurrency issues
    Race conditions
    Incorrect broker choice
    Incorrect driver for the correct broker
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  26. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    Concurrency issues
    Race conditions
    Incorrect broker choice
    Incorrect driver for the correct broker
    Incorrect usage patterns for the correct driver
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  27. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    Concurrency issues
    Race conditions
    Incorrect broker choice
    Incorrect driver for the correct broker
    Incorrect usage patterns for the correct driver
    Incorrect usage patterns for the correct broker
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  28. NEW OPPORTUNITIES...
    NEW OPPORTUNITIES... TO MAKE
    TO MAKE
    NEW MISTAKES
    NEW MISTAKES
    Concurrency issues
    Race conditions
    Incorrect broker choice
    Incorrect driver for the correct broker
    Incorrect usage patterns for the correct driver
    Incorrect usage patterns for the correct broker
    . . .
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  29. LET'S CONTAIN THE RISKS IN
    LET'S CONTAIN THE RISKS IN
    ONE PLACE
    ONE PLACE
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  30. LET'S CONTAIN THE RISKS IN
    LET'S CONTAIN THE RISKS IN
    ONE PLACE
    ONE PLACE
    (A LIBRARY)
    (A LIBRARY)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  31. AND CALL THIS PLACE
    AND CALL THIS PLACE
    async_calls
    async_calls
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  32. AND CALL THIS PLACE
    AND CALL THIS PLACE
    async_calls
    async_calls
    (FOR THE LACK OF BETTER CONCEPT)
    (FOR THE LACK OF BETTER CONCEPT)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  33. A LIBRARY THAT
    A LIBRARY THAT
    meets functional requirements
    for developers, resembles HTTP where possible
    uses a MQ broker for communication
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  34. FOR MAINTAINERS
    FOR MAINTAINERS
    THE SAURON ADVANTAGE :)
    THE SAURON ADVANTAGE :)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  35. FOR MAINTAINERS
    FOR MAINTAINERS
    THE SAURON ADVANTAGE :)
    THE SAURON ADVANTAGE :)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  36. FOR MAINTAINERS
    FOR MAINTAINERS
    THE SAURON ADVANTAGE :)
    THE SAURON ADVANTAGE :)
    One place to fix them all (bugs)
    One place to change them all
    (decisions about broker, drivers, ...)
    One place to apply them all
    (correct usage patters)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  37. FOR DEVELOPERS
    FOR DEVELOPERS
    New complexity is hidden
    Lower entry barrier
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  38. DECISIONS
    DECISIONS
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  39. DECISIONS
    DECISIONS
    Message Broker – Kafka
    performance
    persistence
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  40. DECISIONS
    DECISIONS
    Message Broker – Kafka
    performance
    persistence
    Kafka driver – confluent-kafka
    performance
    supported by Confluent
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  41. ASSUMPTIONS
    ASSUMPTIONS
    Make it simple – provide just IPC
    Library, not framework approach
    Make it testable
    manually (curl-like tool)
    automatically (reasonable mocks)
    Make it resemble Flask?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  42. TALK IS CHEAP!
    TALK IS CHEAP!
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  43. From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  44. CREATE AN APPLICATION OBJECT
    CREATE AN APPLICATION OBJECT
    from async_calls import AsyncCalls
    async_calls = AsyncCalls('a-money-broker')
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  45. CREATE AN APPLICATION OBJECT
    CREATE AN APPLICATION OBJECT
    CREATE A BASIC ENDPOINT
    CREATE A BASIC ENDPOINT
    from async_calls import AsyncCalls
    async_calls = AsyncCalls('a-money-broker')
    @async_calls.server.callback_for('/show-me-the-money')
    def show_me_the_money(request):
    for i in range(1,5):
    payload = f"Response {i} for call: {request.id}"
    response = request.create_response(payload)
    async_calls.server.send(response)
    time.sleep(1)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  46. CREATE AN APPLICATION OBJECT
    CREATE AN APPLICATION OBJECT
    CREATE A BASIC ENDPOINT
    CREATE A BASIC ENDPOINT
    from async_calls import AsyncCalls
    async_calls = AsyncCalls('a-money-broker') # a service ID
    @async_calls.server.callback_for('/show-me-the-money')
    def show_me_the_money(request):
    for i in range(1,5):
    payload = f"Response {i} for call: {request.id}"
    response = request.create_response(payload)
    async_calls.server.send(response)
    time.sleep(1)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  47. CREATE AN APPLICATION OBJECT
    CREATE AN APPLICATION OBJECT
    CREATE A BASIC ENDPOINT
    CREATE A BASIC ENDPOINT
    from async_calls import AsyncCalls
    async_calls = AsyncCalls('a-money-broker') # a service ID
    @async_calls.server.callback_for('/show-me-the-money')
    def show_me_the_money(request): # ^^^^^^^^ an endpoint name
    for i in range(1,5):
    payload = f"Response {i} for call: {request.id}"
    response = request.create_response(payload)
    async_calls.server.send(response)
    time.sleep(1)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  48. CREATE A BASIC CLIENT
    CREATE A BASIC CLIENT
    request = async_calls.client.new_message(
    destination_service_id: 'a-money-broker',
    target_endpoint: '/show-me-the-money',
    request_payload
    )
    async_calls.client.send(request)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  49. CREATE A BASIC CLIENT
    CREATE A BASIC CLIENT
    @async_calls.client.callback_for(
    'a-money-broker', '/show-me-the-money')
    def the_money_handler(response):
    logger.info(
    f"Got: {response.id} for: {response.referenced_id}"
    )
    request = async_calls.client.new_message(
    destination_service_id: 'a-money-broker',
    target_endpoint: '/show-me-the-money',
    request_payload
    )
    async_calls.client.send(request)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  50. TO START LISTENING
    TO START LISTENING
    (CLIENT AND SERVER)
    (CLIENT AND SERVER)
    async_calls.listen()
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  51. WHAT WE HAVE THEN?
    WHAT WE HAVE THEN?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  52. WHAT WE HAVE THEN?
    WHAT WE HAVE THEN?
    Server — event-driven (like HTTP)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  53. WHAT WE HAVE THEN?
    WHAT WE HAVE THEN?
    Server — event-driven (like HTTP)
    Client — non-blocking, event-driven (unlike HTTP)
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  54. WHAT WE HAVE THEN?
    WHAT WE HAVE THEN?
    Server — event-driven (like HTTP)
    Client — non-blocking, event-driven (unlike HTTP)
    One request — any number of responses
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  55. WHAT WE HAVE THEN?
    WHAT WE HAVE THEN?
    Server — event-driven (like HTTP)
    Client — non-blocking, event-driven (unlike HTTP)
    One request — any number of responses
    A single process can be a server and a client
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  56. HOW DO WE TEST THIS!?
    HOW DO WE TEST THIS!?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  57. async_calls
    async_calls
    HAS A TESTING MODE
    HAS A TESTING MODE
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  58. TESTING A SERVER
    TESTING A SERVER
    DOES IT RESPOND CORRECTLY?
    DOES IT RESPOND CORRECTLY?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  59. TESTING A CLIENT
    TESTING A CLIENT
    DOES IT SEND EXPECTED REQUESTS?
    DOES IT SEND EXPECTED REQUESTS?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  60. TESTING CLIENT
    TESTING CLIENT
    FAKING SERVER'S RESPONSE
    FAKING SERVER'S RESPONSE
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  61. TESTING SUMMARY
    TESTING SUMMARY
    Tools out-of-the-box
    Calls made on stack, deterministic tests
    No MQ broker required for unittests
    No need to think about IPC details when
    implementing tests
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  62. MANY MORE FEATURES
    MANY MORE FEATURES
    before send and before receive hooks
    (e.g. for validations)
    endpoint context managers
    (e.g. for performance measurements)
    endpoint error handlers
    Kubernetes healthcheck
    CLI curl-like client
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  63. ANY DRAWBACKS?
    ANY DRAWBACKS?
    Hiding complexity we hide opportunities...
    ...not only to make new errors
    e.g. no Kafka Streams via async_calls
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  64. HOW DID IT SAVE US?
    HOW DID IT SAVE US?
    Concurrency issues
    Race conditions
    Incorrect broker choice
    Incorrect driver for the correct broker
    Incorrect usage patterns for the correct driver
    Incorrect usage patterns for the correct broker
    . . .
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  65. SUMMARY
    SUMMARY
    Switching from HTTP to async_calls
    Server is straightforward
    Client is not complicated
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  66. SUMMARY
    SUMMARY
    Switching from HTTP to async_calls
    Server is straightforward
    Client is not complicated
    Support for one-way communication
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  67. SUMMARY
    SUMMARY
    Switching from HTTP to async_calls
    Server is straightforward
    Client is not complicated
    Support for one-way communication
    More complex use cases require more attention
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  68. SUMMARY
    SUMMARY
    Switching from HTTP to async_calls
    Server is straightforward
    Client is not complicated
    Support for one-way communication
    More complex use cases require more attention
    Services are easily testable
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  69. SUMMARY
    SUMMARY
    Switching from HTTP to async_calls
    Server is straightforward
    Client is not complicated
    Support for one-way communication
    More complex use cases require more attention
    Services are easily testable
    Standard project-wide layer for asynchronous IPC
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  70. SUMMARY
    SUMMARY
    Switching from HTTP to async_calls
    Server is straightforward
    Client is not complicated
    Support for one-way communication
    More complex use cases require more attention
    Services are easily testable
    Standard project-wide layer for asynchronous IPC
    A number of small but useful bonuses
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  71. ASYNCHRONOUS WAR STORIES
    ASYNCHRONOUS WAR STORIES
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  72. SERVICES
    SERVICES
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  73. FAILURES AND ERRORS
    FAILURES AND ERRORS
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  74. SERVICES – HAPPY PATH
    SERVICES – HAPPY PATH
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  75. COMMUNICATION ASSUMPTIONS
    COMMUNICATION ASSUMPTIONS
    Deduplication (using version on each record)
    Version checking
    on >= version
    not on == version + 1
    Error in code causes exception
    Exception prevents committing of Kafka message
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  76. SERVICES – FAILURE 1
    SERVICES – FAILURE 1
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  77. SERVICES – FAILURE 2
    SERVICES – FAILURE 2
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  78. SERVICES – FAILURE 3
    SERVICES – FAILURE 3
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  79. MESSAGE (RE)ORDERING
    MESSAGE (RE)ORDERING
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  80. MESSAGE ORDER OK
    MESSAGE ORDER OK
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  81. MESSAGE ORDER CHANGED
    MESSAGE ORDER CHANGED
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  82. ASYNCHRONOUS (REST-LIKE) CRUD
    ASYNCHRONOUS (REST-LIKE) CRUD
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  83. ASYNCHRONOUS (REST-LIKE)
    ASYNCHRONOUS (REST-LIKE)
    CRUD
    CRUD
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  84. DELETE CANNOT DELETE?
    DELETE CANNOT DELETE?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  85. DELETE SHOULD CREATE?
    DELETE SHOULD CREATE?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  86. PATCH?
    PATCH?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide

  87. SUMMARY
    SUMMARY
    A library hiding MQ from devs works well
    Cannot hide all async problems behind a facade
    Consequences in DB design
    Decisions dependent on business logic
    HTTP to MQ is not just a change of transport
    Procedures of restoring data consistency
    Do I really need all this mess?
    From HTTP to Kafka-based Microservices – [email protected] – @wrzasa

    View Slide