Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributed Domain Destruction - Adventures in building distributed systems

Distributed Domain Destruction - Adventures in building distributed systems

For the last couple of years, I have been building, maintaining, and fighting projects that make use of distributed computing, parallel processing, message brokers, queues, and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short-sighted albeit logical architectural decisions that will cost you in the long run.

Vranac Srdjan

October 04, 2019
Tweet

More Decks by Vranac Srdjan

Other Decks in Programming

Transcript

  1. DDD
    ADVENTURES IN BUILDING
    DISTRIBUTED SYSTEMS
    1 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  2. business owner, developer, consultant, mercenary,
    writing terrible code that performs exceptionally,
    wrangling elePHPants and Pythons, obsessed with
    process automation, interested in continuous integration
    and delivery, clean code, testing, best practices and
    distributed systems
    2 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  3. WARNING
    3 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  4. DDD
    4 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  5. DISTRIBUTED
    5 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  6. DOMAIN
    6 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  7. DESTRUCTION
    7 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  8. WHAT IS
    A DISTRIBUTED SYSTEM?
    8 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  9. THE SIMPLE DEFINITON
    It is a system where you can distribute processing of
    costly tasks to other workers.
    By costly tasks I mean anything ranging to heavy
    computation and cpu utilization to long running
    processes.
    If we are talking in the context of web applications,
    basically anything that can't be done within a request is
    a candidate for distribution/background processing.
    9 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  10. A MORE REALISTIC DEFINITION
    A distributed system is one in which the failure
    of a computer you didn't even know existed can
    render your own computer unusable.
    — Leslie B. Lamport, 1987
    10 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  11. 11 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  12. 12 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  13. 13 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  14. 14 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  15. CONGRATULATIONS
    15 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  16. FINITE EXPANSE OF
    THE UNOPENING WALLET
    16 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  17. 17 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  18. 18 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  19. DAEMONS...
    DAEMONS EVERYWHERE
    19 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  20. SUPERVISOR
    UPSTART
    SYSTEMD
    20 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  21. 21 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  22. LOG AND COUNT
    EVERYTHING
    22 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  23. AGGREGATE AND SEE
    23 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  24. 24 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  25. STATSD
    25 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  26. ...BECAUSE
    LOGS AND COUNTERS
    ARE LYING TO YOU
    26 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  27. WHAT HAPPENED TO
    40 MILLION RECORDS,
    WHERE HAVE THEY GONE?
    27 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  28. WHAT IS DYNAMODB?
    28 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  29. TEMPORAL TABLES
    YOU SAY?
    29 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  30. ARE WE WRITING DATA TO
    THE DYNAMODB?
    30 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  31. THE LOGS ARE CLEAN,
    THE METRICS ARE GREEN
    31 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  32. NO EXCEPTIONS?
    32 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  33. 33 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  34. WHAT THE HELL
    DID I JUST READ?
    34 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  35. I CAN HAZ ERROR
    HANDLING NAO?
    LOL NO
    35 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  36. 36 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  37. THIS WORKER IS FAST,
    THAT WORKER IS SLOW,
    WHICH ONE WILL CRASH
    THE DATABASE,
    NOBODY KNOWS
    37 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  38. TIMING IS EVERYTHING!111
    38 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  39. 39 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  40. 40 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  41. BATCH PROCESSING
    41 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  42. 42 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  43. ABRAKAPOCUS,
    YOUR DISK SPACE IS GONE
    43 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  44. 0 BYTES FREE
    44 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  45. MAKE THIS APP GREAT AGAIN
    AND
    GET THE CLIENT TO PAY FOR IT
    45 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  46. 46 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  47. WARNING:
    INDEX OUT OF BOUNDS
    47 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  48. BUT WHY ISN'T THERE A
    SEPARATE LOG PARTITION?
    48 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  49. 10X
    49 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  50. LIVE,
    AND LEARN,
    AND DOCUMENT...
    50 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  51. WHAT DO YOU MEAN WE
    CAN'T SCALE?
    51 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  52. LOCAL...
    YOU KEEP USING THAT
    WORD...
    52 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  53. MEMORY
    REMEMBER TO USE IT
    53 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  54. THERE IS AN S IN
    SUCCESS
    54 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  55. 55 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  56. BROWSER BUG
    REPRODUCTION
    56 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  57. CURL REQUEST
    57 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  58. OUTPUT THE LAST
    REQUEST/RESPONSE
    58 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  59. WTF?????
    59 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  60. 60 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  61. 61 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  62. 18 HOURS TO LOCATE,
    20 SECONDS TO FIX
    62 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  63. LESSON IN CHAOS
    ENGINEERING
    63 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  64. HEY LOOK, IT'S GONE...
    64 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  65. LESSON IS ACTUALLY A
    MASTERCLASS
    65 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  66. AN EMAIL A DAY KEEPS
    THE CHAOS MONKEY AWAY
    66 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  67. SCORCHED EARTH POLICY
    67 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  68. CRATES, BROWSERS AND
    DATABASES
    68 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  69. 16 CPUS 122 GB OF RAM
    69 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  70. 70 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  71. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX
    71 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  72. PARKING SPACE
    72 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  73. 73 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide

  74. THANK YOU!
    SRDJAN VRANAC // [email protected] // @VRANAC
    74 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

    View full-size slide