Distributed Domain Destruction - Adventures in building distributed systems

For the last couple of years, I have been building, maintaining, and fighting projects that make use of distributed computing, parallel processing, message brokers, queues, and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short-sighted albeit logical architectural decisions that will cost you in the long run.

Transcript

1. DDD ADVENTURES IN BUILDING DISTRIBUTED SYSTEMS 1 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
2. business owner, developer, consultant, mercenary, writing terrible code that performs exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems 2 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
3. WARNING 3 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
4. DDD 4 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
5. DISTRIBUTED 5 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
6. DOMAIN 6 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
7. DESTRUCTION 7 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
8. WHAT IS A DISTRIBUTED SYSTEM? 8 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
9. THE SIMPLE DEFINITON It is a system where you can distribute processing of costly tasks to other workers. By costly tasks I mean anything ranging to heavy computation and cpu utilization to long running processes. If we are talking in the context of web applications, basically anything that can't be done within a request is a candidate for distribution/background processing. 9 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
10. A MORE REALISTIC DEFINITION A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. — Leslie B. Lamport, 1987 10 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
11. 11 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
12. 12 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
13. 13 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
14. 14 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
15. CONGRATULATIONS 15 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
16. FINITE EXPANSE OF THE UNOPENING WALLET 16 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
17. 17 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
18. 18 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
19. DAEMONS... DAEMONS EVERYWHERE 19 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
20. SUPERVISOR UPSTART SYSTEMD 20 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
21. 21 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
22. LOG AND COUNT EVERYTHING 22 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
23. AGGREGATE AND SEE 23 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
24. 24 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
25. STATSD 25 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
26. ...BECAUSE LOGS AND COUNTERS ARE LYING TO YOU 26 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
27. WHAT HAPPENED TO 40 MILLION RECORDS, WHERE HAVE THEY GONE? 27 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
28. WHAT IS DYNAMODB? 28 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
29. TEMPORAL TABLES YOU SAY? 29 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
30. ARE WE WRITING DATA TO THE DYNAMODB? 30 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
31. THE LOGS ARE CLEAN, THE METRICS ARE GREEN 31 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
32. NO EXCEPTIONS? 32 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
33. 33 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
34. WHAT THE HELL DID I JUST READ? 34 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
35. I CAN HAZ ERROR HANDLING NAO? LOL NO 35 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
36. 36 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
37. THIS WORKER IS FAST, THAT WORKER IS SLOW, WHICH ONE WILL CRASH THE DATABASE, NOBODY KNOWS 37 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
38. TIMING IS EVERYTHING!111 38 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
39. 39 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
40. 40 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
41. BATCH PROCESSING 41 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
42. 42 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
43. ABRAKAPOCUS, YOUR DISK SPACE IS GONE 43 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
44. 0 BYTES FREE 44 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
45. MAKE THIS APP GREAT AGAIN AND GET THE CLIENT TO PAY FOR IT 45 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
46. 46 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
47. WARNING: INDEX OUT OF BOUNDS 47 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
48. BUT WHY ISN'T THERE A SEPARATE LOG PARTITION? 48 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
49. 10X 49 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
50. LIVE, AND LEARN, AND DOCUMENT... 50 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
51. WHAT DO YOU MEAN WE CAN'T SCALE? 51 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
52. LOCAL... YOU KEEP USING THAT WORD... 52 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
53. MEMORY REMEMBER TO USE IT 53 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
54. THERE IS AN S IN SUCCESS 54 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
55. 55 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
56. BROWSER BUG REPRODUCTION 56 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
57. CURL REQUEST 57 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
58. OUTPUT THE LAST REQUEST/RESPONSE 58 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
59. WTF????? 59 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
60. 60 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
61. 61 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
62. 18 HOURS TO LOCATE, 20 SECONDS TO FIX 62 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
63. LESSON IN CHAOS ENGINEERING 63 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
64. HEY LOOK, IT'S GONE... 64 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
65. LESSON IS ACTUALLY A MASTERCLASS 65 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
66. AN EMAIL A DAY KEEPS THE CHAOS MONKEY AWAY 66 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
67. SCORCHED EARTH POLICY 67 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
68. CRATES, BROWSERS AND DATABASES 68 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
69. 16 CPUS 122 GB OF RAM 69 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
70. 70 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
71. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX 71 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
72. PARKING SPACE 72 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
73. 73 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019
74. THANK YOU! SRDJAN VRANAC // VRANAC@CODE4HIRE.COM // @VRANAC 74 — Srdjan Vranac, Code4Hire, PyCon Balkan 2019