Tweet
Tweet

Slide 1

Slide 1 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 1/83 FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rząsa, FLYR Poland @wrzasa From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 2

Slide 2 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 2/83 Informatics specialist by passion and by profession 15 years of academic work PhD but primarily an engineer FLYR Inc. Distributed systems Rzeszow Ruby User Group ABOUT ME ABOUT ME http://flyrlabs.com http://rrug.pl From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 3

Slide 3 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 3/83 FLYR FLYR Revenue management system for airlines Offices in San Francisco, USA (PST) Kraków, Poland (CEST) Machine Learning Microservices Python GCloud Kubernetes From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 4

Slide 4 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 4/83 FLYR DEVS ON PYCON CZ FLYR DEVS ON PYCON CZ From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 5

Slide 5 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 5/83 IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 6

Slide 6 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 6/83 IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 7

Slide 7 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 7/83 FAN-OUT REQUESTS FAN-OUT REQUESTS From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 8

Slide 8 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 8/83 PARTIAL RESPONSES PARTIAL RESPONSES From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 9

Slide 9 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 9/83 PARTIAL RESPONSES PARTIAL RESPONSES From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 10

Slide 10 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 10/83 IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 11

Slide 11 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 11/83 IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses performance From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 12

Slide 12 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 12/83 OK, LET'S SWITCH FROM HTTP OK, LET'S SWITCH FROM HTTP TO... A... MQ? TO... A... MQ? From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 13

Slide 13 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 13/83 BUT... BUT... From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 14

Slide 14 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 14/83 BUT... BUT... We have HTTP-based infrastructure From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 15

Slide 15 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 15/83 BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 16

Slide 16 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 16/83 BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 17

Slide 17 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 17/83 BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC We need to do it well ;-) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 18

Slide 18 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 18/83 WITH MQ WE GET WITH MQ WE GET Flexibility Reliability Scalability Robustness From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 19

Slide 19 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 19/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 20

Slide 20 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 20/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 21

Slide 21 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 21/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 22

Slide 22 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 22/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 23

Slide 23 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 23/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 24

Slide 24 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 24/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 25

Slide 25 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 25/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 26

Slide 26 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 26/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 27

Slide 27 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 27/83 NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 28

Slide 28 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 28/83 LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 29

Slide 29 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 29/83 LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE (A LIBRARY) (A LIBRARY) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 30

Slide 30 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 30/83 AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 31

Slide 31 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 31/83 AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls (FOR THE LACK OF BETTER CONCEPT) (FOR THE LACK OF BETTER CONCEPT) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 32

Slide 32 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 32/83 A LIBRARY THAT A LIBRARY THAT meets functional requirements for developers, resembles HTTP where possible uses a MQ broker for communication From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 33

Slide 33 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 33/83 FOR MAINTAINERS FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON ADVANTAGE :) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 34

Slide 34 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 34/83 FOR MAINTAINERS FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON ADVANTAGE :) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 35

Slide 35 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 35/83 FOR MAINTAINERS FOR MAINTAINERS THE SAURON ADVANTAGE :) THE SAURON ADVANTAGE :) One place to fix them all (bugs) One place to change them all (decisions about broker, drivers, ...) One place to apply them all (correct usage patters) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 36

Slide 36 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 36/83 FOR DEVELOPERS FOR DEVELOPERS New complexity is hidden Lower entry barrier From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 37

Slide 37 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 37/83 DECISIONS DECISIONS From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 38

Slide 38 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 38/83 DECISIONS DECISIONS Message Broker – Kafka performance persistence From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 39

Slide 39 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 39/83 DECISIONS DECISIONS Message Broker – Kafka performance persistence Kafka driver – confluent-kafka performance supported by Confluent From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 40

Slide 40 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 40/83 ASSUMPTIONS ASSUMPTIONS Make it simple – provide just IPC Library, not framework approach Make it testable manually (curl-like tool) automatically (reasonable mocks) Make it resemble Flask? From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 41

Slide 41 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 41/83 TALK IS CHEAP! TALK IS CHEAP! From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 42

Slide 42 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 42/83 From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 43

Slide 43 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 43/83 CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 44

Slide 44 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 44/83 CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 45

Slide 45 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 45/83 CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') # a service ID @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 46

Slide 46 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 46/83 CREATE AN APPLICATION OBJECT CREATE AN APPLICATION OBJECT CREATE A BASIC ENDPOINT CREATE A BASIC ENDPOINT from async_calls import AsyncCalls async_calls = AsyncCalls('a-money-broker') # a service ID @async_calls.server.callback_for('/show-me-the-money') def show_me_the_money(request): # ^^^^^^^^ an endpoint name for i in range(1,5): payload = f"Response {i} for call: {request.id}" response = request.create_response(payload) async_calls.server.send(response) time.sleep(1) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 47

Slide 47 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 47/83 CREATE A BASIC CLIENT CREATE A BASIC CLIENT request = async_calls.client.new_message( destination_service_id: 'a-money-broker', target_endpoint: '/show-me-the-money', request_payload ) async_calls.client.send(request) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 48

Slide 48 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 48/83 CREATE A BASIC CLIENT CREATE A BASIC CLIENT @async_calls.client.callback_for( 'a-money-broker', '/show-me-the-money') def the_money_handler(response): logger.info( f"Got: {response.id} for: {response.referenced_id}" ) request = async_calls.client.new_message( destination_service_id: 'a-money-broker', target_endpoint: '/show-me-the-money', request_payload ) async_calls.client.send(request) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 49

Slide 49 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 49/83 TO START LISTENING TO START LISTENING (CLIENT AND SERVER) (CLIENT AND SERVER) async_calls.listen() From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 50

Slide 50 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 50/83 WHAT WE HAVE THEN? WHAT WE HAVE THEN? From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 51

Slide 51 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 51/83 WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 52

Slide 52 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 52/83 WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 53

Slide 53 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 53/83 WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) One request — any number of responses From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 54

Slide 54 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 54/83 WHAT WE HAVE THEN? WHAT WE HAVE THEN? Server — event-driven (like HTTP) Client — non-blocking, event-driven (unlike HTTP) One request — any number of responses A single process can be a server and a client From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 55

Slide 55 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 55/83 HOW DO WE TEST THIS!? HOW DO WE TEST THIS!? From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 56

Slide 56 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 56/83 async_calls async_calls HAS A TESTING MODE HAS A TESTING MODE # setup testing mode from async_calls import AsyncCalls AsyncCalls.testing = True From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 57

Slide 57 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 57/83 async_calls async_calls HAS A TESTING MODE HAS A TESTING MODE # setup testing mode from async_calls import AsyncCalls AsyncCalls.testing = True # import your app from ..async_endpoint import async_calls From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 58

Slide 58 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 58/83 async_calls async_calls HAS A TESTING MODE HAS A TESTING MODE # setup testing mode from async_calls import AsyncCalls AsyncCalls.testing = True # import your app from ..async_endpoint import async_calls # ensure you reset state between tests @pytest.fixture(autouse=True) def reset_async_calls_test_mode(): async_calls.test_mode_reset() From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 59

Slide 59 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 59/83 TESTING A SERVER TESTING A SERVER DOES IT RESPOND CORRECTLY? DOES IT RESPOND CORRECTLY? # Send a test request (using test_client) request = async_calls.test_client.new_message( 'a-money-broker', '/show-me-the-money', 'A money request' ) async_calls.test_client.send(request) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 60

Slide 60 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 60/83 TESTING A SERVER TESTING A SERVER DOES IT RESPOND CORRECTLY? DOES IT RESPOND CORRECTLY? # Send a test request (using test_client) request = async_calls.test_client.new_message( 'a-money-broker', '/show-me-the-money', 'A money request' ) async_calls.test_client.send(request) # Check what response were received (by test_client) responses = async_calls.test_client.received_responses() From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 61

Slide 61 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 61/83 TESTING A SERVER TESTING A SERVER DOES IT RESPOND CORRECTLY? DOES IT RESPOND CORRECTLY? # Send a test request (using test_client) request = async_calls.test_client.new_message( 'a-money-broker', '/show-me-the-money', 'A money request' ) async_calls.test_client.send(request) # Check what response were received (by test_client) responses = async_calls.test_client.received_responses() # Set your expectations expected_payloads = ["the money", "you expect"] received_payloads = [m.payload for m in responses] assert sorted(expected_payloads) == sorted(received_payloads) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 62

Slide 62 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 62/83 TESTING A CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 63

Slide 63 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 63/83 TESTING A CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? # Unit tests for 'a money broker' service # register endpoint in test_server async_calls.test_server.register_endpoint( 'notifications_receiver', '/notifiy' ) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 64

Slide 64 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 64/83 TESTING A CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? # Unit tests for 'a money broker' service # register endpoint in test_server async_calls.test_server.register_endpoint( 'notifications_receiver', '/notifiy' ) # some testing actions that should trigger messages to the # '/notify' endpoint of 'notifications_receiver' service From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 65

Slide 65 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 65/83 TESTING A CLIENT TESTING A CLIENT DOES IT SEND EXPECTED REQUESTS? DOES IT SEND EXPECTED REQUESTS? # Unit tests for 'a money broker' service # register endpoint in test_server async_calls.test_server.register_endpoint( 'notifications_receiver', '/notifiy' ) # some testing actions that should trigger messages to the # '/notify' endpoint of 'notifications_receiver' service rcved = async_calls.test_server.received_requests('/notify') # here you can assert that all required messages were # received by the '/notify' endpoint From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 66

Slide 66 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 66/83 TESTING CLIENT TESTING CLIENT FAKING SERVER'S RESPONSE FAKING SERVER'S RESPONSE From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 67

Slide 67 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 67/83 TESTING CLIENT TESTING CLIENT FAKING SERVER'S RESPONSE FAKING SERVER'S RESPONSE # Unit tests for 'a money broker' service # Create a fake random number service def fake_responses(request): for i in [4, 7, 8, 10, 12]: yield request.create_response(i) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 68

Slide 68 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 68/83 TESTING CLIENT TESTING CLIENT FAKING SERVER'S RESPONSE FAKING SERVER'S RESPONSE # Unit tests for 'a money broker' service # Create a fake random number service def fake_responses(request): for i in [4, 7, 8, 10, 12]: yield request.create_response(i) # register endpoint with generator in test_server async_calls.test_server.register_endpoint( 'random', '/get_values', fake_responses ) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 69

Slide 69 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 69/83 TESTING SUMMARY TESTING SUMMARY Tools out-of-the-box Calls made on stack, deterministic tests No MQ broker required for unittests No need to think about IPC details when implementing tests From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 70

Slide 70 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 70/83 MANY MORE FEATURES MANY MORE FEATURES From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 71

Slide 71 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 71/83 MANY MORE FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 72

Slide 72 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 72/83 MANY MORE FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 73

Slide 73 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 73/83 MANY MORE FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) endpoint error handlers From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 74

Slide 74 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 74/83 MANY MORE FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) endpoint error handlers Kubernetes healthcheck From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 75

Slide 75 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 75/83 MANY MORE FEATURES MANY MORE FEATURES before send and before receive hooks (e.g. for validations) endpoint context managers (e.g. for performance measurements) endpoint error handlers Kubernetes healthcheck CLI curl-like client From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 76

Slide 76 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 76/83 ANY DRAWBACKS? ANY DRAWBACKS? Hiding complexity we hide opportunities... ...not only to make new errors e.g. no Kafka Streams via async_calls From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 77

Slide 77 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 77/83 HOW DID IT SAVE US? HOW DID IT SAVE US? Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 78

Slide 78 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 78/83 SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward Client is not complicated From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 79

Slide 79 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 79/83 SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 80

Slide 80 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 80/83 SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 81

Slide 81 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 81/83 SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 82

Slide 82 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 82/83 SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable Standard project-wide layer for asynchronous IPC From HTTP to Kafka-based microservices – [email protected] – @wrzasa

Slide 83

Slide 83 text

7/17/2019 From HTTP to Kafka-based microservices localhost:4567/index.html?print-pdf#/ 83/83 SUMMARY SUMMARY Switching from HTTP to async_calls Server is straightforward Client is not complicated Support for one-way communication More complex use cases require more attention Services are easily testable Standard project-wide layer for asynchronous IPC A number of small but useful bonuses From HTTP to Kafka-based microservices – [email protected] – @wrzasa