From HTTP to Kafka-based Microservices

From HTTP to Kafka-based Microservices

68414aba518bb5898c6ae29d9ae7dd7d?s=128

wrzasa

July 12, 2019
Tweet

Transcript

  1. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 1/83 FROM HTTP

    TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rząsa, FLYR Poland @wrzasa From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  2. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 2/83 Informatics specialist

    by passion and by profession 15 years of academic work PhD but primarily an engineer FLYR Inc. Distributed systems Rzeszow Ruby User Group ABOUT ME ABOUT ME http://flyrlabs.com http://rrug.pl From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  3. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 3/83 FLYR FLYR

    Revenue management system for airlines Offices in San Francisco, USA (PST) Kraków, Poland (CEST) Machine Learning Microservices Python GCloud Kubernetes From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  4. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 4/83 FLYR DEVS

    ON PYCON CZ FLYR DEVS ON PYCON CZ From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  5. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 5/83 IN FLYR

    MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  6. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 6/83 IN FLYR

    MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  7. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 7/83 FAN-OUT REQUESTS

    FAN-OUT REQUESTS From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  8. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 8/83 PARTIAL RESPONSES

    PARTIAL RESPONSES From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  9. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 9/83 PARTIAL RESPONSES

    PARTIAL RESPONSES From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  10. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 10/83 IN FLYR

    MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  11. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 11/83 IN FLYR

    MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses performance From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  12. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 12/83 OK, LET'S

    SWITCH FROM HTTP OK, LET'S SWITCH FROM HTTP TO... A... MQ? TO... A... MQ? From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  13. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 13/83 BUT... BUT...

    From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  14. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 14/83 BUT... BUT...

    We have HTTP-based infrastructure From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  15. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 15/83 BUT... BUT...

    We have HTTP-based infrastructure We have HTTP developers experience and habits From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  16. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 16/83 BUT... BUT...

    We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  17. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 17/83 BUT... BUT...

    We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC We need to do it well ;-) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  18. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 18/83 WITH MQ

    WE GET WITH MQ WE GET Flexibility Reliability Scalability Robustness From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  19. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 19/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  20. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 20/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  21. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 21/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  22. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 22/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  23. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 23/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  24. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 24/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  25. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 25/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  26. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 26/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  27. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 27/83 NEW OPPORTUNITIES...

    NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  28. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 28/83 LET'S CONTAIN

    THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  29. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 29/83 LET'S CONTAIN

    THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE (A LIBRARY) (A LIBRARY) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  30. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 30/83 AND CALL

    THIS PLACE AND CALL THIS PLACE async_calls async_calls From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  31. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 31/83 AND CALL

    THIS PLACE AND CALL THIS PLACE async_calls async_calls (FOR THE LACK OF BETTER CONCEPT) (FOR THE LACK OF BETTER CONCEPT) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  32. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 32/83 A LIBRARY

    THAT A LIBRARY THAT meets functional requirements for developers, resembles HTTP where possible uses a MQ broker for communication From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  33. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 33/83 FOR MAINTAINERS

    FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON ADVANTAGE :) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  34. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 34/83 FOR MAINTAINERS

    FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON ADVANTAGE :) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  35. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 35/83 FOR MAINTAINERS

    FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON ADVANTAGE :) One place to fix them all (bugs) One place to change them all (decisions about broker, drivers, ...) One place to apply them all (correct usage patters) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  36. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 36/83 FOR DEVELOPERS

    FOR DEVELOPERS New complexity is hidden Lower entry barrier From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  37. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 37/83 DECISIONS DECISIONS

    From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  38. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 38/83 DECISIONS DECISIONS

    Message Broker – Kafka performance persistence From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  39. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 39/83 DECISIONS DECISIONS

    Message Broker – Kafka performance persistence Kafka driver – confluent-kafka performance supported by Confluent From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  40. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 40/83 ASSUMPTIONS ASSUMPTIONS

    Make it simple – provide just IPC Library, not framework approach Make it testable manually (curl-like tool) automatically (reasonable mocks) Make it resemble Flask? From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  41. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 41/83 TALK IS

    CHEAP! TALK IS CHEAP! From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  42. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 42/83 From HTTP

    to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  43. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 43/83 CREATE AN

    APPLICATION OBJECT CREATE AN APPLICATION OBJECT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  44. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 44/83 CREATE AN

    APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  45. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 45/83 CREATE AN

    APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') # a service ID @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  46. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 46/83 CREATE AN

    APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') # a service ID @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): # ^^^^^^^^ an endpoint name for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  47. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 47/83 CREATE A

    BASIC CLIENT CREATE A BASIC CLIENT request = async_calls.client.new_message( destination_service_id: 'a-money-broker', target_endpoint: '/show-me-the-money', request_payload ) async_calls.client.send(request) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  48. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 48/83 CREATE A

    BASIC CLIENT CREATE A BASIC CLIENT @async_calls.client.callback_for( 'a-money-broker', '/show-me-the-money') def the_money_handler(response): logger.info( f"Got: {response.id} for: {response.referenced_id}" ) request = async_calls.client.new_message( destination_service_id: 'a-money-broker', target_endpoint: '/show-me-the-money', request_payload ) async_calls.client.send(request) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  49. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 49/83 TO START

    LISTENING TO START LISTENING (CLIENT AND SERVER) (CLIENT AND SERVER) async_calls.listen() From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  50. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 50/83 WHAT WE

    HAVE THEN? WHAT WE HAVE THEN? From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  51. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 51/83 WHAT WE

    HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  52. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 52/83 WHAT WE

    HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  53. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 53/83 WHAT WE

    HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) One request — any number of responses From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  54. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 54/83 WHAT WE

    HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) One request — any number of responses A single process can be a server and a client From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  55. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 55/83 HOW DO

    WE TEST THIS!? HOW DO WE TEST THIS!? From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  56. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 56/83 async_calls async_calls

    HAS A TESTING MODE HAS A TESTING MODE # setup testing mode from async_calls import AsyncCalls AsyncCalls.testing = True From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  57. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 57/83 async_calls async_calls

    HAS A TESTING MODE HAS A TESTING MODE # setup testing mode from async_calls import AsyncCalls AsyncCalls.testing = True # import your app from ..async_endpoint import async_calls From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  58. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 58/83 async_calls async_calls

    HAS A TESTING MODE HAS A TESTING MODE # setup testing mode from async_calls import AsyncCalls AsyncCalls.testing = True # import your app from ..async_endpoint import async_calls # ensure you reset state between tests @pytest.fixture(autouse=True) def reset_async_calls_test_mode(): async_calls.test_mode_reset() From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  59. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 59/83 TESTING A

    SERVER TESTING A SERVER DOES IT RESPOND CORRECTLY? DOES IT RESPOND CORRECTLY? # Send a test request (using test_client) request = async_calls.test_client.new_message( 'a-money-broker', '/show-me-the-money', 'A money request' ) async_calls.test_client.send(request) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  60. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 60/83 TESTING A

    SERVER TESTING A SERVER DOES IT RESPOND CORRECTLY? DOES IT RESPOND CORRECTLY? # Send a test request (using test_client) request = async_calls.test_client.new_message( 'a-money-broker', '/show-me-the-money', 'A money request' ) async_calls.test_client.send(request) # Check what response were received (by test_client) responses = async_calls.test_client.received_responses() From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  61. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 61/83 TESTING A

    SERVER TESTING A SERVER DOES IT RESPOND CORRECTLY? DOES IT RESPOND CORRECTLY? # Send a test request (using test_client) request = async_calls.test_client.new_message( 'a-money-broker', '/show-me-the-money', 'A money request' ) async_calls.test_client.send(request) # Check what response were received (by test_client) responses = async_calls.test_client.received_responses() # Set your expectations expected_payloads = ["the money", "you expect"] received_payloads = [m.payload for m in responses] assert sorted(expected_payloads) == sorted(received_payloads) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  62. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 62/83 TESTING A

    CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  63. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 63/83 TESTING A

    CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? # Unit tests for 'a money broker' service # register endpoint in test_server async_calls.test_server.register_endpoint( 'notifications_receiver', '/notifiy' ) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  64. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 64/83 TESTING A

    CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? # Unit tests for 'a money broker' service # register endpoint in test_server async_calls.test_server.register_endpoint( 'notifications_receiver', '/notifiy' ) # some testing actions that should trigger messages to the # '/notify' endpoint of 'notifications_receiver' service From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  65. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 65/83 TESTING A

    CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? # Unit tests for 'a money broker' service # register endpoint in test_server async_calls.test_server.register_endpoint( 'notifications_receiver', '/notifiy' ) # some testing actions that should trigger messages to the # '/notify' endpoint of 'notifications_receiver' service rcved = async_calls.test_server.received_requests('/notify') # here you can assert that all required messages were # received by the '/notify' endpoint From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  66. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 66/83 TESTING CLIENT

    TESTING CLIENT FAKING SERVER'S RESPONSE FAKING SERVER'S RESPONSE From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  67. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 67/83 TESTING CLIENT

    TESTING CLIENT FAKING SERVER'S RESPONSE FAKING SERVER'S RESPONSE # Unit tests for 'a money broker' service # Create a fake random number service def fake_responses(request): for i in [4, 7, 8, 10, 12]: yield request.create_response(i) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  68. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 68/83 TESTING CLIENT

    TESTING CLIENT FAKING SERVER'S RESPONSE FAKING SERVER'S RESPONSE # Unit tests for 'a money broker' service # Create a fake random number service def fake_responses(request): for i in [4, 7, 8, 10, 12]: yield request.create_response(i) # register endpoint with generator in test_server async_calls.test_server.register_endpoint( 'random', '/get_values', fake_responses ) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  69. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 69/83 TESTING SUMMARY

    TESTING SUMMARY Tools out-of-the-box Calls made on stack, deterministic tests No MQ broker required for unittests No need to think about IPC details when implementing tests From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  70. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 70/83 MANY MORE

    FEATURES MANY MORE FEATURES From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  71. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 71/83 MANY MORE

    FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  72. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 72/83 MANY MORE

    FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  73. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 73/83 MANY MORE

    FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) endpoint error handlers From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  74. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 74/83 MANY MORE

    FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) endpoint error handlers Kubernetes healthcheck From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  75. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 75/83 MANY MORE

    FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) endpoint error handlers Kubernetes healthcheck CLI curl-like client From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  76. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 76/83 ANY DRAWBACKS?

    ANY DRAWBACKS? Hiding complexity we hide opportunities... ...not only to make new errors e.g. no Kafka Streams via async_calls From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  77. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 77/83 HOW DID

    IT SAVE US? HOW DID IT SAVE US? Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  78. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 78/83 SUMMARY SUMMARY

    Switching from HTTP to async_calls Server is straightforward Client is not complicated From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  79. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 79/83 SUMMARY SUMMARY

    Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  80. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 80/83 SUMMARY SUMMARY

    Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  81. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 81/83 SUMMARY SUMMARY

    Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  82. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 82/83 SUMMARY SUMMARY

    Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable Standard project-wide layer for asynchronous IPC From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa
  83. 7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 83/83 SUMMARY SUMMARY

    Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable Standard project-wide layer for asynchronous IPC A number of small but useful bonuses From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa