Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bulletproof communication in distributed systems

Bulletproof communication in distributed systems

13962ee99d2c07e7b7a7776222532f1d?s=128

Jakub Kubryński

March 30, 2019
Tweet

Transcript

  1. @jkubrynski / kubrynski.com @jkubrynski / kubrynski.com BULLETPROOF BULLETPROOF COMMUNICATION COMMUNICATION

    JAKUB KUBRYNSKI JAKUB KUBRYNSKI jk@devskiller.com / @jkubrynski / kubrynski.blog
  2. $ WHOAMI $ WHOAMI DEVSKILLER CO-FOUNDER DEVSKILLER CO-FOUNDER BOTTEGA TRAINER

    BOTTEGA TRAINER DEVOXX.PL PROGRAM COMMITTEE MEMBER DEVOXX.PL PROGRAM COMMITTEE MEMBER SPRING CLOUD CONTRACT CO-AUTHOR SPRING CLOUD CONTRACT CO-AUTHOR
  3. DEVSKILLER IS LOOKING FOR DEVSKILLER IS LOOKING FOR HELLO-WORLD DEVELOPERS

    HELLO-WORLD DEVELOPERS
  4. WHY DISTRIBUTED SYSTEMS? WHY DISTRIBUTED SYSTEMS? SCALABILITY SCALABILITY POLYGLOT POLYGLOT

    RESILIENCE RESILIENCE
  5. DISTRIBUTED SYSTEMS FALLACIES DISTRIBUTED SYSTEMS FALLACIES THE NETWORK IS RELIABLE

    THE NETWORK IS RELIABLE LATENCY IS ZERO LATENCY IS ZERO THE NETWORK IS SECURE THE NETWORK IS SECURE TOPOLOGY DOESN'T CHANGE TOPOLOGY DOESN'T CHANGE
  6. IF YOU WON'T RESPECT FALLACIES IF YOU WON'T RESPECT FALLACIES

    YOU'LL BUILD DISTRIBUTED MONOLITH YOU'LL BUILD DISTRIBUTED MONOLITH
  7. BRACE YOURSELF BRACE YOURSELF FAILURE IS COMING FAILURE IS COMING

  8. IN MONOLITH ALL COLLABORATORS ARE IN MONOLITH ALL COLLABORATORS ARE

    ALWAYS AVAILABLE ALWAYS AVAILABLE
  9. IN DISTRIBUTED SYSTEMS THEY ARE NOT IN DISTRIBUTED SYSTEMS THEY

    ARE NOT NETWORK ISSUES NETWORK ISSUES REDEPLOYS REDEPLOYS OUTAGES OUTAGES
  10. AVAILABILITY VS DOWNTIME AVAILABILITY VS DOWNTIME 9 9 . 9

    9 9 % 35d 4d 8h 50m 5m per year 2.5h 14m 1.5m 8s 0.8s per day
  11. 99.9 (1.5 MIN/DAY) 99.9 (1.5 MIN/DAY) 5 * 99.9 =>

    99.5 (7 MIN/DAY) 5 * 99.9 => 99.5 (7 MIN/DAY)
  12. TEMPORAL COUPLING TEMPORAL COUPLING

  13. DESIGN FOR FAILURE DESIGN FOR FAILURE

  14. PREFER ASYNCHRONOUS PREFER ASYNCHRONOUS OVER SYNCHRONOUS OVER SYNCHRONOUS

  15. ASYNCHRONOUS ACTORS ASYNCHRONOUS ACTORS COMMANDS (QUEUES) COMMANDS (QUEUES) EVENTS (TOPICS)

    EVENTS (TOPICS)
  16. ASYNCHRONOUS PRINCIPLES ASYNCHRONOUS PRINCIPLES PERSISTENT PERSISTENT AT-LEAST-ONCE DELIVERY AT-LEAST-ONCE DELIVERY

    EXPONENTIAL RETRIES EXPONENTIAL RETRIES
  17. SYNCHRONOUS SYNCHRONOUS IS USED ONLY WHEN REQUIRED IS USED ONLY

    WHEN REQUIRED
  18. WHAT COULD POSSIBLY GO WRONG? WHAT COULD POSSIBLY GO WRONG?

    ¯\_(ツ)_/¯ ¯\_(ツ)_/¯
  19. TIMEOUT?!?! TIMEOUT?!?! RETRY RETRY

  20. CIRCUIT BREAKER CIRCUIT BREAKER KEEP CALM AND FALLBACK KEEP CALM

    AND FALLBACK
  21. IN MONOLITH TOPOLOGY IS FIXED IN MONOLITH TOPOLOGY IS FIXED

  22. IN DISTRIBUTED SYSTEMS IT'S NOT IN DISTRIBUTED SYSTEMS IT'S NOT

    CLOUD NATIVE / CONTAINERS CLOUD NATIVE / CONTAINERS DYNAMIC SCALABILITY DYNAMIC SCALABILITY
  23. ALWAYS USE SERVICE DISCOVERY ALWAYS USE SERVICE DISCOVERY

  24. LOAD BALANCERS SUCKS LOAD BALANCERS SUCKS IT'S SINGLE POINT OF

    FAILURE IT'S SINGLE POINT OF FAILURE
  25. ALWAYS USE CLIENT SIDE LOAD BALACING ALWAYS USE CLIENT SIDE

    LOAD BALACING
  26. IT'S ALL ABOUT COMMUNICATION IT'S ALL ABOUT COMMUNICATION SO NICE

    TO KNOW IT WORKS SO NICE TO KNOW IT WORKS
  27. DISTRIBUTED SYSTEMS DISTRIBUTED SYSTEMS VS VS MICROSERVICES MICROSERVICES

  28. MICROSERVICES BRING MICROSERVICES BRING AUTONOMY AUTONOMY

  29. AUTONOMY IS HARD AUTONOMY IS HARD BECAUSE WE NEED TO

    COORDINATE DEPENDENCIES BECAUSE WE NEED TO COORDINATE DEPENDENCIES AND BE BACKWARD COMPATIBLE AND BE BACKWARD COMPATIBLE
  30. LOOSE COUPLING LOOSE COUPLING OVER OVER CANONICAL MODEL CANONICAL MODEL

  31. END-TO-END TESTS ARE HARD END-TO-END TESTS ARE HARD FALSE-NEGATIVE FALSE-NEGATIVE

    FALSE-POSITIVE FALSE-POSITIVE
  32. VERIFY CONTRACTS VERIFY CONTRACTS A == C && B ==

    C A == C && B == C A == B A == B
  33. CONTRACT CONVERTS TO CONTRACT CONVERTS TO SERVER TEST SERVER TEST

    CLIENT STUB CLIENT STUB
  34. PERFORMANCE? PERFORMANCE?

  35. MONITORING IS THE NEW TESTING MONITORING IS THE NEW TESTING

  36. MONITORING MONITORING EXCEPTIONS EXCEPTIONS BUSINESS METRICS BUSINESS METRICS APPLICATION PERFORMANCE

    MANAGEMENT APPLICATION PERFORMANCE MANAGEMENT
  37. ANSCOMBE'S QUARTET ANSCOMBE'S QUARTET Mean of x => 9 Sample

    variance of x => 11 Mean of y => 7.50 Sample variance of y => 4.125 Correlation => 0.816 Linear regression => y = 3.00 + 0.500x
  38. PERCENTILES PERCENTILES P50, P90, P95, P99 P50, P90, P95, P99

  39. MONITOR MONITOR P99.9 OR P99.95 P99.9 OR P99.95

  40. THANKS THANKS

  41. DROGA.DEV DROGA.DEV

  42. QUESTIONS? QUESTIONS?

  43. "Have no fear of perfection - you'll never reach it."

    ― Salvador Dalí