Bulletproof communication in distributed systems

Transcript

1. @jkubrynski / kubrynski.com
   @jkubrynski / kubrynski.com
   BULLETPROOF
   BULLETPROOF
   COMMUNICATION
   COMMUNICATION
   JAKUB KUBRYNSKI
   JAKUB KUBRYNSKI
   jk@devskiller.com / @jkubrynski / kubrynski.blog
   

   View Slide

2. $ WHOAMI
   $ WHOAMI
   DEVSKILLER CO-FOUNDER
   DEVSKILLER CO-FOUNDER
   BOTTEGA TRAINER
   BOTTEGA TRAINER
   DEVOXX.PL PROGRAM COMMITTEE MEMBER
   DEVOXX.PL PROGRAM COMMITTEE MEMBER
   SPRING CLOUD CONTRACT CO-AUTHOR
   SPRING CLOUD CONTRACT CO-AUTHOR
   

   View Slide

3. DEVSKILLER IS LOOKING FOR
   DEVSKILLER IS LOOKING FOR
   HELLO-WORLD DEVELOPERS
   HELLO-WORLD DEVELOPERS
   

   View Slide

4. WHY DISTRIBUTED SYSTEMS?
   WHY DISTRIBUTED SYSTEMS?
   SCALABILITY
   SCALABILITY
   POLYGLOT
   POLYGLOT
   RESILIENCE
   RESILIENCE
   

   View Slide

5. DISTRIBUTED SYSTEMS FALLACIES
   DISTRIBUTED SYSTEMS FALLACIES
   THE NETWORK IS RELIABLE
   THE NETWORK IS RELIABLE
   LATENCY IS ZERO
   LATENCY IS ZERO
   THE NETWORK IS SECURE
   THE NETWORK IS SECURE
   TOPOLOGY DOESN'T CHANGE
   TOPOLOGY DOESN'T CHANGE
   

   View Slide

6. IF YOU WON'T RESPECT FALLACIES
   IF YOU WON'T RESPECT FALLACIES
   YOU'LL BUILD DISTRIBUTED MONOLITH
   YOU'LL BUILD DISTRIBUTED MONOLITH
   

   View Slide

7. BRACE YOURSELF
   BRACE YOURSELF
   FAILURE IS COMING
   FAILURE IS COMING
   

   View Slide

8. IN MONOLITH ALL COLLABORATORS ARE
   IN MONOLITH ALL COLLABORATORS ARE
   ALWAYS AVAILABLE
   ALWAYS AVAILABLE
   

   View Slide

9. IN DISTRIBUTED SYSTEMS THEY ARE NOT
   IN DISTRIBUTED SYSTEMS THEY ARE NOT
   NETWORK ISSUES
   NETWORK ISSUES
   REDEPLOYS
   REDEPLOYS
   OUTAGES
   OUTAGES
   

   View Slide

10. AVAILABILITY VS DOWNTIME
    AVAILABILITY VS DOWNTIME
    9 9 . 9 9 9 %
    35d 4d 8h 50m 5m per year
    2.5h 14m 1.5m 8s 0.8s per day
    

    View Slide

11. 99.9 (1.5 MIN/DAY)
    99.9 (1.5 MIN/DAY)
    5 * 99.9 => 99.5 (7 MIN/DAY)
    5 * 99.9 => 99.5 (7 MIN/DAY)
    

    View Slide

12. TEMPORAL COUPLING
    TEMPORAL COUPLING
    

    View Slide

13. DESIGN FOR FAILURE
    DESIGN FOR FAILURE
    

    View Slide

14. PREFER ASYNCHRONOUS
    PREFER ASYNCHRONOUS
    OVER SYNCHRONOUS
    OVER SYNCHRONOUS
    

    View Slide

15. ASYNCHRONOUS ACTORS
    ASYNCHRONOUS ACTORS
    COMMANDS (QUEUES)
    COMMANDS (QUEUES)
    EVENTS (TOPICS)
    EVENTS (TOPICS)
    

    View Slide

16. ASYNCHRONOUS PRINCIPLES
    ASYNCHRONOUS PRINCIPLES
    PERSISTENT
    PERSISTENT
    AT-LEAST-ONCE DELIVERY
    AT-LEAST-ONCE DELIVERY
    EXPONENTIAL RETRIES
    EXPONENTIAL RETRIES
    

    View Slide

17. SYNCHRONOUS
    SYNCHRONOUS
    IS USED ONLY WHEN REQUIRED
    IS USED ONLY WHEN REQUIRED
    

    View Slide

18. WHAT COULD POSSIBLY GO WRONG?
    WHAT COULD POSSIBLY GO WRONG?
    ¯\_(ツ)_/¯
    ¯\_(ツ)_/¯
    

    View Slide

19. TIMEOUT?!?!
    TIMEOUT?!?!
    RETRY
    RETRY
    

    View Slide

20. CIRCUIT BREAKER
    CIRCUIT BREAKER
    KEEP CALM AND FALLBACK
    KEEP CALM AND FALLBACK
    

    View Slide

21. IN MONOLITH TOPOLOGY IS FIXED
    IN MONOLITH TOPOLOGY IS FIXED
    

    View Slide

22. IN DISTRIBUTED SYSTEMS IT'S NOT
    IN DISTRIBUTED SYSTEMS IT'S NOT
    CLOUD NATIVE / CONTAINERS
    CLOUD NATIVE / CONTAINERS
    DYNAMIC SCALABILITY
    DYNAMIC SCALABILITY
    

    View Slide

23. ALWAYS USE SERVICE DISCOVERY
    ALWAYS USE SERVICE DISCOVERY
    

    View Slide

24. LOAD BALANCERS SUCKS
    LOAD BALANCERS SUCKS
    IT'S SINGLE POINT OF FAILURE
    IT'S SINGLE POINT OF FAILURE
    

    View Slide

25. ALWAYS USE CLIENT SIDE LOAD BALACING
    ALWAYS USE CLIENT SIDE LOAD BALACING
    

    View Slide

26. IT'S ALL ABOUT COMMUNICATION
    IT'S ALL ABOUT COMMUNICATION
    SO NICE TO KNOW IT WORKS
    SO NICE TO KNOW IT WORKS
    

    View Slide

27. DISTRIBUTED SYSTEMS
    DISTRIBUTED SYSTEMS
    VS
    VS
    MICROSERVICES
    MICROSERVICES
    

    View Slide

28. MICROSERVICES BRING
    MICROSERVICES BRING
    AUTONOMY
    AUTONOMY
    

    View Slide

29. AUTONOMY IS HARD
    AUTONOMY IS HARD
    BECAUSE WE NEED TO COORDINATE DEPENDENCIES
    BECAUSE WE NEED TO COORDINATE DEPENDENCIES
    AND BE BACKWARD COMPATIBLE
    AND BE BACKWARD COMPATIBLE
    

    View Slide

30. LOOSE COUPLING
    LOOSE COUPLING
    OVER
    OVER
    CANONICAL MODEL
    CANONICAL MODEL
    

    View Slide

31. END-TO-END TESTS ARE HARD
    END-TO-END TESTS ARE HARD
    FALSE-NEGATIVE
    FALSE-NEGATIVE
    FALSE-POSITIVE
    FALSE-POSITIVE
    

    View Slide

32. VERIFY CONTRACTS
    VERIFY CONTRACTS
    A == C && B == C
    A == C && B == C
    A == B
    A == B
    

    View Slide

33. CONTRACT CONVERTS TO
    CONTRACT CONVERTS TO
    SERVER TEST
    SERVER TEST
    CLIENT STUB
    CLIENT STUB
    

    View Slide

34. PERFORMANCE?
    PERFORMANCE?
    

    View Slide

35. MONITORING IS THE NEW TESTING
    MONITORING IS THE NEW TESTING
    

    View Slide

36. MONITORING
    MONITORING
    EXCEPTIONS
    EXCEPTIONS
    BUSINESS METRICS
    BUSINESS METRICS
    APPLICATION PERFORMANCE MANAGEMENT
    APPLICATION PERFORMANCE MANAGEMENT
    

    View Slide

37. ANSCOMBE'S QUARTET
    ANSCOMBE'S QUARTET
    Mean of x => 9
    Sample variance of x => 11
    Mean of y => 7.50
    Sample variance of y => 4.125
    Correlation => 0.816
    Linear regression => y = 3.00 + 0.500x
    

    View Slide

38. PERCENTILES
    PERCENTILES
    P50, P90, P95, P99
    P50, P90, P95, P99
    

    View Slide

39. MONITOR
    MONITOR
    P99.9 OR P99.95
    P99.9 OR P99.95
    

    View Slide

40. THANKS
    THANKS
    

    View Slide

41. DROGA.DEV
    DROGA.DEV
    

    View Slide

42. QUESTIONS?
    QUESTIONS?
    

    View Slide

43. "Have no fear of perfection - you'll never reach it." ― Salvador Dalí
    

    View Slide