$30 off During Our Annual Pro Sale. View Details »

Bulletproof communication in distributed systems

Bulletproof communication in distributed systems

Jakub Kubryński

March 30, 2019
Tweet

More Decks by Jakub Kubryński

Other Decks in Programming

Transcript

  1. @jkubrynski / kubrynski.com
    @jkubrynski / kubrynski.com
    BULLETPROOF
    BULLETPROOF
    COMMUNICATION
    COMMUNICATION
    JAKUB KUBRYNSKI
    JAKUB KUBRYNSKI
    [email protected] / @jkubrynski / kubrynski.blog

    View Slide

  2. $ WHOAMI
    $ WHOAMI
    DEVSKILLER CO-FOUNDER
    DEVSKILLER CO-FOUNDER
    BOTTEGA TRAINER
    BOTTEGA TRAINER
    DEVOXX.PL PROGRAM COMMITTEE MEMBER
    DEVOXX.PL PROGRAM COMMITTEE MEMBER
    SPRING CLOUD CONTRACT CO-AUTHOR
    SPRING CLOUD CONTRACT CO-AUTHOR

    View Slide

  3. DEVSKILLER IS LOOKING FOR
    DEVSKILLER IS LOOKING FOR
    HELLO-WORLD DEVELOPERS
    HELLO-WORLD DEVELOPERS

    View Slide

  4. WHY DISTRIBUTED SYSTEMS?
    WHY DISTRIBUTED SYSTEMS?
    SCALABILITY
    SCALABILITY
    POLYGLOT
    POLYGLOT
    RESILIENCE
    RESILIENCE

    View Slide

  5. DISTRIBUTED SYSTEMS FALLACIES
    DISTRIBUTED SYSTEMS FALLACIES
    THE NETWORK IS RELIABLE
    THE NETWORK IS RELIABLE
    LATENCY IS ZERO
    LATENCY IS ZERO
    THE NETWORK IS SECURE
    THE NETWORK IS SECURE
    TOPOLOGY DOESN'T CHANGE
    TOPOLOGY DOESN'T CHANGE

    View Slide

  6. IF YOU WON'T RESPECT FALLACIES
    IF YOU WON'T RESPECT FALLACIES
    YOU'LL BUILD DISTRIBUTED MONOLITH
    YOU'LL BUILD DISTRIBUTED MONOLITH

    View Slide

  7. BRACE YOURSELF
    BRACE YOURSELF
    FAILURE IS COMING
    FAILURE IS COMING

    View Slide

  8. IN MONOLITH ALL COLLABORATORS ARE
    IN MONOLITH ALL COLLABORATORS ARE
    ALWAYS AVAILABLE
    ALWAYS AVAILABLE

    View Slide

  9. IN DISTRIBUTED SYSTEMS THEY ARE NOT
    IN DISTRIBUTED SYSTEMS THEY ARE NOT
    NETWORK ISSUES
    NETWORK ISSUES
    REDEPLOYS
    REDEPLOYS
    OUTAGES
    OUTAGES

    View Slide

  10. AVAILABILITY VS DOWNTIME
    AVAILABILITY VS DOWNTIME
    9 9 . 9 9 9 %
    35d 4d 8h 50m 5m per year
    2.5h 14m 1.5m 8s 0.8s per day

    View Slide

  11. 99.9 (1.5 MIN/DAY)
    99.9 (1.5 MIN/DAY)
    5 * 99.9 => 99.5 (7 MIN/DAY)
    5 * 99.9 => 99.5 (7 MIN/DAY)

    View Slide

  12. TEMPORAL COUPLING
    TEMPORAL COUPLING

    View Slide

  13. DESIGN FOR FAILURE
    DESIGN FOR FAILURE

    View Slide

  14. PREFER ASYNCHRONOUS
    PREFER ASYNCHRONOUS
    OVER SYNCHRONOUS
    OVER SYNCHRONOUS

    View Slide

  15. ASYNCHRONOUS ACTORS
    ASYNCHRONOUS ACTORS
    COMMANDS (QUEUES)
    COMMANDS (QUEUES)
    EVENTS (TOPICS)
    EVENTS (TOPICS)

    View Slide

  16. ASYNCHRONOUS PRINCIPLES
    ASYNCHRONOUS PRINCIPLES
    PERSISTENT
    PERSISTENT
    AT-LEAST-ONCE DELIVERY
    AT-LEAST-ONCE DELIVERY
    EXPONENTIAL RETRIES
    EXPONENTIAL RETRIES

    View Slide

  17. SYNCHRONOUS
    SYNCHRONOUS
    IS USED ONLY WHEN REQUIRED
    IS USED ONLY WHEN REQUIRED

    View Slide

  18. WHAT COULD POSSIBLY GO WRONG?
    WHAT COULD POSSIBLY GO WRONG?
    ¯\_(ツ)_/¯
    ¯\_(ツ)_/¯

    View Slide

  19. TIMEOUT?!?!
    TIMEOUT?!?!
    RETRY
    RETRY

    View Slide

  20. CIRCUIT BREAKER
    CIRCUIT BREAKER
    KEEP CALM AND FALLBACK
    KEEP CALM AND FALLBACK

    View Slide

  21. IN MONOLITH TOPOLOGY IS FIXED
    IN MONOLITH TOPOLOGY IS FIXED

    View Slide

  22. IN DISTRIBUTED SYSTEMS IT'S NOT
    IN DISTRIBUTED SYSTEMS IT'S NOT
    CLOUD NATIVE / CONTAINERS
    CLOUD NATIVE / CONTAINERS
    DYNAMIC SCALABILITY
    DYNAMIC SCALABILITY

    View Slide

  23. ALWAYS USE SERVICE DISCOVERY
    ALWAYS USE SERVICE DISCOVERY

    View Slide

  24. LOAD BALANCERS SUCKS
    LOAD BALANCERS SUCKS
    IT'S SINGLE POINT OF FAILURE
    IT'S SINGLE POINT OF FAILURE

    View Slide

  25. ALWAYS USE CLIENT SIDE LOAD BALACING
    ALWAYS USE CLIENT SIDE LOAD BALACING

    View Slide

  26. IT'S ALL ABOUT COMMUNICATION
    IT'S ALL ABOUT COMMUNICATION
    SO NICE TO KNOW IT WORKS
    SO NICE TO KNOW IT WORKS

    View Slide

  27. DISTRIBUTED SYSTEMS
    DISTRIBUTED SYSTEMS
    VS
    VS
    MICROSERVICES
    MICROSERVICES

    View Slide

  28. MICROSERVICES BRING
    MICROSERVICES BRING
    AUTONOMY
    AUTONOMY

    View Slide

  29. AUTONOMY IS HARD
    AUTONOMY IS HARD
    BECAUSE WE NEED TO COORDINATE DEPENDENCIES
    BECAUSE WE NEED TO COORDINATE DEPENDENCIES
    AND BE BACKWARD COMPATIBLE
    AND BE BACKWARD COMPATIBLE

    View Slide

  30. LOOSE COUPLING
    LOOSE COUPLING
    OVER
    OVER
    CANONICAL MODEL
    CANONICAL MODEL

    View Slide

  31. END-TO-END TESTS ARE HARD
    END-TO-END TESTS ARE HARD
    FALSE-NEGATIVE
    FALSE-NEGATIVE
    FALSE-POSITIVE
    FALSE-POSITIVE

    View Slide

  32. VERIFY CONTRACTS
    VERIFY CONTRACTS
    A == C && B == C
    A == C && B == C
    A == B
    A == B

    View Slide

  33. CONTRACT CONVERTS TO
    CONTRACT CONVERTS TO
    SERVER TEST
    SERVER TEST
    CLIENT STUB
    CLIENT STUB

    View Slide

  34. PERFORMANCE?
    PERFORMANCE?

    View Slide

  35. MONITORING IS THE NEW TESTING
    MONITORING IS THE NEW TESTING

    View Slide

  36. MONITORING
    MONITORING
    EXCEPTIONS
    EXCEPTIONS
    BUSINESS METRICS
    BUSINESS METRICS
    APPLICATION PERFORMANCE MANAGEMENT
    APPLICATION PERFORMANCE MANAGEMENT

    View Slide

  37. ANSCOMBE'S QUARTET
    ANSCOMBE'S QUARTET
    Mean of x => 9
    Sample variance of x => 11
    Mean of y => 7.50
    Sample variance of y => 4.125
    Correlation => 0.816
    Linear regression => y = 3.00 + 0.500x

    View Slide

  38. PERCENTILES
    PERCENTILES
    P50, P90, P95, P99
    P50, P90, P95, P99

    View Slide

  39. MONITOR
    MONITOR
    P99.9 OR P99.95
    P99.9 OR P99.95

    View Slide

  40. THANKS
    THANKS

    View Slide

  41. DROGA.DEV
    DROGA.DEV

    View Slide

  42. QUESTIONS?
    QUESTIONS?

    View Slide

  43. "Have no fear of perfection - you'll never reach it." ― Salvador Dalí

    View Slide