Distributed Domain Destruction - Adventures in building distributed systems

Distributed Domain Destruction - Adventures in building distributed systems

For the last couple of years, I have been building, maintaining, and fighting projects that make use of distributed computing, parallel processing, message brokers, queues, and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short-sighted albeit logical architectural decisions that will cost you in the long run.

F2d82b268a7cbccc9809c939428df64f?s=128

Vranac Srdjan

October 04, 2019
Tweet

Transcript

  1. DDD ADVENTURES IN BUILDING DISTRIBUTED SYSTEMS 1 — Srdjan Vranac,

    Code4Hire, PyCon Balkan 2019
  2. business owner, developer, consultant, mercenary, writing terrible code that performs

    exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems 2 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  3. WARNING 3 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  4. DDD 4 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  5. DISTRIBUTED 5 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  6. DOMAIN 6 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  7. DESTRUCTION 7 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  8. WHAT IS A DISTRIBUTED SYSTEM? 8 — Srdjan Vranac, Code4Hire,

    PyCon Balkan 2019
  9. THE SIMPLE DEFINITON It is a system where you can

    distribute processing of costly tasks to other workers. By costly tasks I mean anything ranging to heavy computation and cpu utilization to long running processes. If we are talking in the context of web applications, basically anything that can't be done within a request is a candidate for distribution/background processing. 9 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  10. A MORE REALISTIC DEFINITION A distributed system is one in

    which the failure of a computer you didn't even know existed can render your own computer unusable. — Leslie B. Lamport, 1987 10 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  11. 11 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  12. 12 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  13. 13 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  14. 14 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  15. CONGRATULATIONS 15 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  16. FINITE EXPANSE OF THE UNOPENING WALLET 16 — Srdjan Vranac,

    Code4Hire, PyCon Balkan 2019
  17. 17 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  18. 18 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  19. DAEMONS... DAEMONS EVERYWHERE 19 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  20. SUPERVISOR UPSTART SYSTEMD 20 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  21. 21 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  22. LOG AND COUNT EVERYTHING 22 — Srdjan Vranac, Code4Hire, PyCon

    Balkan 2019
  23. AGGREGATE AND SEE 23 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  24. 24 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  25. STATSD 25 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  26. ...BECAUSE LOGS AND COUNTERS ARE LYING TO YOU 26 —

    Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  27. WHAT HAPPENED TO 40 MILLION RECORDS, WHERE HAVE THEY GONE?

    27 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  28. WHAT IS DYNAMODB? 28 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  29. TEMPORAL TABLES YOU SAY? 29 — Srdjan Vranac, Code4Hire, PyCon

    Balkan 2019
  30. ARE WE WRITING DATA TO THE DYNAMODB? 30 — Srdjan

    Vranac, Code4Hire, PyCon Balkan 2019
  31. THE LOGS ARE CLEAN, THE METRICS ARE GREEN 31 —

    Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  32. NO EXCEPTIONS? 32 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  33. 33 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  34. WHAT THE HELL DID I JUST READ? 34 — Srdjan

    Vranac, Code4Hire, PyCon Balkan 2019
  35. I CAN HAZ ERROR HANDLING NAO? LOL NO 35 —

    Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  36. 36 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  37. THIS WORKER IS FAST, THAT WORKER IS SLOW, WHICH ONE

    WILL CRASH THE DATABASE, NOBODY KNOWS 37 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  38. TIMING IS EVERYTHING!111 38 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  39. 39 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  40. 40 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  41. BATCH PROCESSING 41 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  42. 42 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  43. ABRAKAPOCUS, YOUR DISK SPACE IS GONE 43 — Srdjan Vranac,

    Code4Hire, PyCon Balkan 2019
  44. 0 BYTES FREE 44 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  45. MAKE THIS APP GREAT AGAIN AND GET THE CLIENT TO

    PAY FOR IT 45 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  46. 46 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  47. WARNING: INDEX OUT OF BOUNDS 47 — Srdjan Vranac, Code4Hire,

    PyCon Balkan 2019
  48. BUT WHY ISN'T THERE A SEPARATE LOG PARTITION? 48 —

    Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  49. 10X 49 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  50. LIVE, AND LEARN, AND DOCUMENT... 50 — Srdjan Vranac, Code4Hire,

    PyCon Balkan 2019
  51. WHAT DO YOU MEAN WE CAN'T SCALE? 51 — Srdjan

    Vranac, Code4Hire, PyCon Balkan 2019
  52. LOCAL... YOU KEEP USING THAT WORD... 52 — Srdjan Vranac,

    Code4Hire, PyCon Balkan 2019
  53. MEMORY REMEMBER TO USE IT 53 — Srdjan Vranac, Code4Hire,

    PyCon Balkan 2019
  54. THERE IS AN S IN SUCCESS 54 — Srdjan Vranac,

    Code4Hire, PyCon Balkan 2019
  55. 55 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  56. BROWSER BUG REPRODUCTION 56 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  57. CURL REQUEST 57 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  58. OUTPUT THE LAST REQUEST/RESPONSE 58 — Srdjan Vranac, Code4Hire, PyCon

    Balkan 2019
  59. WTF????? 59 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  60. 60 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  61. 61 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  62. 18 HOURS TO LOCATE, 20 SECONDS TO FIX 62 —

    Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  63. LESSON IN CHAOS ENGINEERING 63 — Srdjan Vranac, Code4Hire, PyCon

    Balkan 2019
  64. HEY LOOK, IT'S GONE... 64 — Srdjan Vranac, Code4Hire, PyCon

    Balkan 2019
  65. LESSON IS ACTUALLY A MASTERCLASS 65 — Srdjan Vranac, Code4Hire,

    PyCon Balkan 2019
  66. AN EMAIL A DAY KEEPS THE CHAOS MONKEY AWAY 66

    — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
  67. SCORCHED EARTH POLICY 67 — Srdjan Vranac, Code4Hire, PyCon Balkan

    2019
  68. CRATES, BROWSERS AND DATABASES 68 — Srdjan Vranac, Code4Hire, PyCon

    Balkan 2019
  69. 16 CPUS 122 GB OF RAM 69 — Srdjan Vranac,

    Code4Hire, PyCon Balkan 2019
  70. 70 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  71. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX 71 — Srdjan

    Vranac, Code4Hire, PyCon Balkan 2019
  72. PARKING SPACE 72 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  73. 73 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019

  74. THANK YOU! SRDJAN VRANAC // VRANAC@CODE4HIRE.COM // @VRANAC 74 —

    Srdjan Vranac, Code4Hire, PyCon Balkan 2019