Bulletproof communication in distributed systems

Transcript

1. 1.

   @jkubrynski / kubrynski.com @jkubrynski / kubrynski.com BULLETPROOF BULLETPROOF COMMUNICATION COMMUNICATION

   JAKUB KUBRYNSKI JAKUB KUBRYNSKI jk@devskiller.com / @jkubrynski / kubrynski.blog
2. 2.

   $ WHOAMI $ WHOAMI DEVSKILLER CO-FOUNDER DEVSKILLER CO-FOUNDER BOTTEGA TRAINER

   BOTTEGA TRAINER DEVOXX.PL PROGRAM COMMITTEE MEMBER DEVOXX.PL PROGRAM COMMITTEE MEMBER SPRING CLOUD CONTRACT CO-AUTHOR SPRING CLOUD CONTRACT CO-AUTHOR
3. 3.

   DEVSKILLER IS LOOKING FOR DEVSKILLER IS LOOKING FOR HELLO-WORLD DEVELOPERS

   HELLO-WORLD DEVELOPERS
4. 4.

   WHY DISTRIBUTED SYSTEMS? WHY DISTRIBUTED SYSTEMS? SCALABILITY SCALABILITY POLYGLOT POLYGLOT

   RESILIENCE RESILIENCE
5. 5.

   DISTRIBUTED SYSTEMS FALLACIES DISTRIBUTED SYSTEMS FALLACIES THE NETWORK IS RELIABLE

   THE NETWORK IS RELIABLE LATENCY IS ZERO LATENCY IS ZERO THE NETWORK IS SECURE THE NETWORK IS SECURE TOPOLOGY DOESN'T CHANGE TOPOLOGY DOESN'T CHANGE
6. 6.

   IF YOU WON'T RESPECT FALLACIES IF YOU WON'T RESPECT FALLACIES

   YOU'LL BUILD DISTRIBUTED MONOLITH YOU'LL BUILD DISTRIBUTED MONOLITH
7. 7.

   BRACE YOURSELF BRACE YOURSELF FAILURE IS COMING FAILURE IS COMING

8. 8.

   IN MONOLITH ALL COLLABORATORS ARE IN MONOLITH ALL COLLABORATORS ARE

   ALWAYS AVAILABLE ALWAYS AVAILABLE
9. 9.

   IN DISTRIBUTED SYSTEMS THEY ARE NOT IN DISTRIBUTED SYSTEMS THEY

   ARE NOT NETWORK ISSUES NETWORK ISSUES REDEPLOYS REDEPLOYS OUTAGES OUTAGES
10. 10.

    AVAILABILITY VS DOWNTIME AVAILABILITY VS DOWNTIME 9 9 . 9

    9 9 % 35d 4d 8h 50m 5m per year 2.5h 14m 1.5m 8s 0.8s per day
11. 11.

    99.9 (1.5 MIN/DAY) 99.9 (1.5 MIN/DAY) 5 * 99.9 =>

    99.5 (7 MIN/DAY) 5 * 99.9 => 99.5 (7 MIN/DAY)
12. 12.

    TEMPORAL COUPLING TEMPORAL COUPLING

13. 13.

    DESIGN FOR FAILURE DESIGN FOR FAILURE

14. 14.

    PREFER ASYNCHRONOUS PREFER ASYNCHRONOUS OVER SYNCHRONOUS OVER SYNCHRONOUS

15. 15.

    ASYNCHRONOUS ACTORS ASYNCHRONOUS ACTORS COMMANDS (QUEUES) COMMANDS (QUEUES) EVENTS (TOPICS)

    EVENTS (TOPICS)
16. 16.

    ASYNCHRONOUS PRINCIPLES ASYNCHRONOUS PRINCIPLES PERSISTENT PERSISTENT AT-LEAST-ONCE DELIVERY AT-LEAST-ONCE DELIVERY

    EXPONENTIAL RETRIES EXPONENTIAL RETRIES
17. 17.

    SYNCHRONOUS SYNCHRONOUS IS USED ONLY WHEN REQUIRED IS USED ONLY

    WHEN REQUIRED
18. 18.

    WHAT COULD POSSIBLY GO WRONG? WHAT COULD POSSIBLY GO WRONG?

    ¯\_(ツ)_/¯ ¯\_(ツ)_/¯
19. 19.

    TIMEOUT?!?! TIMEOUT?!?! RETRY RETRY

20. 20.

    CIRCUIT BREAKER CIRCUIT BREAKER KEEP CALM AND FALLBACK KEEP CALM

    AND FALLBACK
21. 21.

    IN MONOLITH TOPOLOGY IS FIXED IN MONOLITH TOPOLOGY IS FIXED

22. 22.

    IN DISTRIBUTED SYSTEMS IT'S NOT IN DISTRIBUTED SYSTEMS IT'S NOT

    CLOUD NATIVE / CONTAINERS CLOUD NATIVE / CONTAINERS DYNAMIC SCALABILITY DYNAMIC SCALABILITY
23. 23.

    ALWAYS USE SERVICE DISCOVERY ALWAYS USE SERVICE DISCOVERY

24. 24.

    LOAD BALANCERS SUCKS LOAD BALANCERS SUCKS IT'S SINGLE POINT OF

    FAILURE IT'S SINGLE POINT OF FAILURE
25. 25.

    ALWAYS USE CLIENT SIDE LOAD BALACING ALWAYS USE CLIENT SIDE

    LOAD BALACING
26. 26.

    IT'S ALL ABOUT COMMUNICATION IT'S ALL ABOUT COMMUNICATION SO NICE

    TO KNOW IT WORKS SO NICE TO KNOW IT WORKS
27. 27.

    DISTRIBUTED SYSTEMS DISTRIBUTED SYSTEMS VS VS MICROSERVICES MICROSERVICES

28. 28.

    MICROSERVICES BRING MICROSERVICES BRING AUTONOMY AUTONOMY

29. 29.

    AUTONOMY IS HARD AUTONOMY IS HARD BECAUSE WE NEED TO

    COORDINATE DEPENDENCIES BECAUSE WE NEED TO COORDINATE DEPENDENCIES AND BE BACKWARD COMPATIBLE AND BE BACKWARD COMPATIBLE
30. 30.

    LOOSE COUPLING LOOSE COUPLING OVER OVER CANONICAL MODEL CANONICAL MODEL

31. 31.

    END-TO-END TESTS ARE HARD END-TO-END TESTS ARE HARD FALSE-NEGATIVE FALSE-NEGATIVE

    FALSE-POSITIVE FALSE-POSITIVE
32. 32.

    VERIFY CONTRACTS VERIFY CONTRACTS A == C && B ==

    C A == C && B == C A == B A == B
33. 33.

    CONTRACT CONVERTS TO CONTRACT CONVERTS TO SERVER TEST SERVER TEST

    CLIENT STUB CLIENT STUB
34. 34.

    PERFORMANCE? PERFORMANCE?

35. 35.

    MONITORING IS THE NEW TESTING MONITORING IS THE NEW TESTING

36. 36.

    MONITORING MONITORING EXCEPTIONS EXCEPTIONS BUSINESS METRICS BUSINESS METRICS APPLICATION PERFORMANCE

    MANAGEMENT APPLICATION PERFORMANCE MANAGEMENT
37. 37.

    ANSCOMBE'S QUARTET ANSCOMBE'S QUARTET Mean of x => 9 Sample

    variance of x => 11 Mean of y => 7.50 Sample variance of y => 4.125 Correlation => 0.816 Linear regression => y = 3.00 + 0.500x
38. 38.

    PERCENTILES PERCENTILES P50, P90, P95, P99 P50, P90, P95, P99

39. 39.

    MONITOR MONITOR P99.9 OR P99.95 P99.9 OR P99.95

40. 40.

    THANKS THANKS

41. 41.

    DROGA.DEV DROGA.DEV

42. 42.

    QUESTIONS? QUESTIONS?

43. 43.

    "Have no fear of perfection - you'll never reach it."

    ― Salvador Dalí