Pro Yearly is on sale from $80 to $50! »

From HTTP to Kafka-based Microservices (extended)

68414aba518bb5898c6ae29d9ae7dd7d?s=47 wrzasa
November 25, 2019

From HTTP to Kafka-based Microservices (extended)

How we switched microservices from HTTP to Kafka in FLYR Inc. (http://flyrlabs.com), presented on SegFault confenerce in Kraków (https://segfault.events/).

68414aba518bb5898c6ae29d9ae7dd7d?s=128

wrzasa

November 25, 2019
Tweet

Transcript

  1. FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES

    (EXTENDED) (EXTENDED) Wojciech Rząsa, FLYR Poland @wrzasa From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  2. Informatics specialist by passion and by profession 15 years of

    academic work PhD but primarily an engineer FLYR Inc. Distributed systems Rzeszow Ruby User Group ABOUT ME ABOUT ME http://flyrlabs.com http://rrug.pl From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  3. FLYR FLYR Revenue management system for airlines Offices in San

    Francisco, USA (PST) Kraków, Poland (CEST) Machine Learning Microservices Python GCloud Kubernetes Big Query From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  4. FLYR DEVS ON PYCON CZ FLYR DEVS ON PYCON CZ

    From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  5. IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP

    From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  6. IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP

    New requirements for eCommerce use case From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  7. FAN-OUT REQUESTS FAN-OUT REQUESTS From HTTP to Kafka-based Microservices –

    Wojciech.Rzasa@flyrlabs.com – @wrzasa
  8. PARTIAL RESPONSES PARTIAL RESPONSES From HTTP to Kafka-based Microservices –

    Wojciech.Rzasa@flyrlabs.com – @wrzasa
  9. PARTIAL RESPONSES PARTIAL RESPONSES From HTTP to Kafka-based Microservices –

    Wojciech.Rzasa@flyrlabs.com – @wrzasa
  10. IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP

    New requirements for eCommerce use case partial responses From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  11. IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP

    New requirements for eCommerce use case partial responses performance From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  12. IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP

    New requirements for eCommerce use case partial responses performance We will need MQs anyway in the future From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  13. OK, LET'S SWITCH FROM HTTP OK, LET'S SWITCH FROM HTTP

    TO... A... MQ? TO... A... MQ? From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  14. BUT... BUT... From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com –

    @wrzasa
  15. BUT... BUT... We have HTTP-based infrastructure From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  16. BUT... BUT... We have HTTP-based infrastructure We have HTTP developers

    experience and habits From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  17. BUT... BUT... We have HTTP-based infrastructure We have HTTP developers

    experience and habits We lack experience with MQ-based IPC From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  18. BUT... BUT... We have HTTP-based infrastructure We have HTTP developers

    experience and habits We lack experience with MQ-based IPC We need to do it well ;-) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  19. WITH MQ WE GET WITH MQ WE GET Flexibility Reliability

    Scalability Robustness From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  20. NEW OPPORTUNITIES... NEW OPPORTUNITIES... From HTTP to Kafka-based Microservices –

    Wojciech.Rzasa@flyrlabs.com – @wrzasa
  21. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  22. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES Concurrency issues From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  23. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES Concurrency issues Race conditions From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  24. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  25. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  26. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  27. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  28. NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES

    NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  29. LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN

    ONE PLACE ONE PLACE From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  30. LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN

    ONE PLACE ONE PLACE (A LIBRARY) (A LIBRARY) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  31. AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls

    From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  32. AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls

    (FOR THE LACK OF BETTER CONCEPT) (FOR THE LACK OF BETTER CONCEPT) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  33. A LIBRARY THAT A LIBRARY THAT meets functional requirements for

    developers, resembles HTTP where possible uses a MQ broker for communication From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  34. FOR MAINTAINERS FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON

    ADVANTAGE :) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  35. FOR MAINTAINERS FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON

    ADVANTAGE :) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  36. FOR MAINTAINERS FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON

    ADVANTAGE :) One place to fix them all (bugs) One place to change them all (decisions about broker, drivers, ...) One place to apply them all (correct usage patters) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  37. FOR DEVELOPERS FOR DEVELOPERS New complexity is hidden Lower entry

    barrier From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  38. DECISIONS DECISIONS From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com –

    @wrzasa
  39. DECISIONS DECISIONS Message Broker – Kafka performance persistence From HTTP

    to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  40. DECISIONS DECISIONS Message Broker – Kafka performance persistence Kafka driver

    – confluent-kafka performance supported by Confluent From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  41. ASSUMPTIONS ASSUMPTIONS Make it simple – provide just IPC Library,

    not framework approach Make it testable manually (curl-like tool) automatically (reasonable mocks) Make it resemble Flask? From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  42. TALK IS CHEAP! TALK IS CHEAP! From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  43. From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  44. CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT from async_calls

    import AsyncCalls async_calls = AsyncCalls('a-money-broker') From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  45. CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A

    BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  46. CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A

    BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') # a service ID @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  47. CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A

    BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') # a service ID @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): # ^^^^^^^^ an endpoint name for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  48. CREATE A BASIC CLIENT CREATE A BASIC CLIENT request =

    async_calls.client.new_message( destination_service_id: 'a-money-broker', target_endpoint: '/show-me-the-money', request_payload ) async_calls.client.send(request) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  49. CREATE A BASIC CLIENT CREATE A BASIC CLIENT @async_calls.client.callback_for( 'a-money-broker',

    '/show-me-the-money') def the_money_handler(response): logger.info( f"Got: {response.id} for: {response.referenced_id}" ) request = async_calls.client.new_message( destination_service_id: 'a-money-broker', target_endpoint: '/show-me-the-money', request_payload ) async_calls.client.send(request) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  50. TO START LISTENING TO START LISTENING (CLIENT AND SERVER) (CLIENT

    AND SERVER) async_calls.listen() From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  51. WHAT WE HAVE THEN? WHAT WE HAVE THEN? From HTTP

    to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  52. WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server —

    event-driven (like HTTP) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  53. WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server —

    event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  54. WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server —

    event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) One request — any number of responses From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  55. WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server —

    event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) One request — any number of responses A single process can be a server and a client From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  56. HOW DO WE TEST THIS!? HOW DO WE TEST THIS!?

    From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  57. async_calls async_calls HAS A TESTING MODE HAS A TESTING MODE

    From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  58. TESTING A SERVER TESTING A SERVER DOES IT RESPOND CORRECTLY?

    DOES IT RESPOND CORRECTLY? From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  59. TESTING A CLIENT TESTING A CLIENT DOES IT SEND EXPECTED

    REQUESTS? DOES IT SEND EXPECTED REQUESTS? From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  60. TESTING CLIENT TESTING CLIENT FAKING SERVER'S RESPONSE FAKING SERVER'S RESPONSE

    From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  61. TESTING SUMMARY TESTING SUMMARY Tools out-of-the-box Calls made on stack,

    deterministic tests No MQ broker required for unittests No need to think about IPC details when implementing tests From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  62. MANY MORE FEATURES MANY MORE FEATURES before send and before

    receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) endpoint error handlers Kubernetes healthcheck CLI curl-like client From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  63. ANY DRAWBACKS? ANY DRAWBACKS? Hiding complexity we hide opportunities... ...not

    only to make new errors e.g. no Kafka Streams via async_calls From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  64. HOW DID IT SAVE US? HOW DID IT SAVE US?

    Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  65. SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward

    Client is not complicated From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  66. SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward

    Client is not complicated Support for one-way communication From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  67. SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward

    Client is not complicated Support for one-way communication More complex use cases require more attention From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  68. SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward

    Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  69. SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward

    Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable Standard project-wide layer for asynchronous IPC From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  70. SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward

    Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable Standard project-wide layer for asynchronous IPC A number of small but useful bonuses From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  71. ASYNCHRONOUS WAR STORIES ASYNCHRONOUS WAR STORIES From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  72. SERVICES SERVICES From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com –

    @wrzasa
  73. FAILURES AND ERRORS FAILURES AND ERRORS From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  74. SERVICES – HAPPY PATH SERVICES – HAPPY PATH From HTTP

    to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  75. COMMUNICATION ASSUMPTIONS COMMUNICATION ASSUMPTIONS Deduplication (using version on each record)

    Version checking on >= version not on == version + 1 Error in code causes exception Exception prevents committing of Kafka message From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  76. SERVICES – FAILURE 1 SERVICES – FAILURE 1 From HTTP

    to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  77. SERVICES – FAILURE 2 SERVICES – FAILURE 2 From HTTP

    to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  78. SERVICES – FAILURE 3 SERVICES – FAILURE 3 From HTTP

    to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  79. MESSAGE (RE)ORDERING MESSAGE (RE)ORDERING From HTTP to Kafka-based Microservices –

    Wojciech.Rzasa@flyrlabs.com – @wrzasa
  80. MESSAGE ORDER OK MESSAGE ORDER OK From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  81. MESSAGE ORDER CHANGED MESSAGE ORDER CHANGED From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  82. ASYNCHRONOUS (REST-LIKE) CRUD ASYNCHRONOUS (REST-LIKE) CRUD From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  83. ASYNCHRONOUS (REST-LIKE) ASYNCHRONOUS (REST-LIKE) CRUD CRUD From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  84. DELETE CANNOT DELETE? DELETE CANNOT DELETE? From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  85. DELETE SHOULD CREATE? DELETE SHOULD CREATE? From HTTP to Kafka-based

    Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  86. PATCH? PATCH? From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com –

    @wrzasa
  87. SUMMARY SUMMARY A library hiding MQ from devs works well

    Cannot hide all async problems behind a facade Consequences in DB design Decisions dependent on business logic HTTP to MQ is not just a change of transport Procedures of restoring data consistency Do I really need all this mess? From HTTP to Kafka-based Microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa