Distributed Domain Destruction - Adventures in building distributed systems

For the last three years I have been building, maintaining and fighting a few projects that make use of distributed computing, parallel processing, message brokers, queues and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short sighted albeit logical architectural decisions that will cost you in the long run.

Come hear about the agony you will experience when it starts falling apart, and the thrill you will feel when everything is running juuust right.

Transcript

1. DDD ADVENTURES IN BUILDING DISTRIBUTED SYSTEMS 1 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
2. business owner, developer, consultant, mercenary, writing terrible code that performs exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems 2 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
3. WARNING 3 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
4. DDD 4 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
5. DISTRIBUTED 5 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
6. DOMAIN 6 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
7. DESTRUCTION 7 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
8. WHAT IS A DISTRIBUTED SYSTEM? 8 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
9. THE SIMPLE DEFINITON It is a system where you can distribute processing of costly tasks to other workers. By costly taks I mean anything ranging to heavy computation and cpu utilization to long running processes. If we are talking in the context of web applications, basically anything that can't be done within a request is a candidate for distribution/background processing. 9 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
10. A MORE REALISTIC DEFINITION A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. — Leslie B. Lamport, 1987 10 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
11. 11 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
12. 12 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
13. 13 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
14. 14 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
15. CONGRATULATIONS 15 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
16. FINITE EXPANSE OF THE UNOPENING WALLET 16 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
17. 17 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
18. 18 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
19. DAEMONS... DAEMONS EVERYWHERE 19 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
20. SUPERVISOR UPSTART 20 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
21. 21 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
22. LOG AND COUNT EVERYTHING 22 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
23. AGGREGATE AND SEE 23 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
24. 24 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
25. STATSD 25 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
26. ...BECAUSE LOGS AND COUNTERS ARE LYING TO YOU 26 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
27. WHAT HAPPENED TO 40 MILLION RECORDS, WHERE HAVE THEY GONE? 27 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
28. WHAT IS DYNAMODB? 28 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
29. TEMPORAL TABLES YOU SAY? 29 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
30. ARE WE WRITING DATA TO THE DYNAMODB? 30 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
31. THE LOGS ARE CLEAN, THE METRICS ARE GREEN 31 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
32. NO EXCEPTIONS? 32 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
33. 33 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
34. WHAT THE HELL DID I JUST READ? 34 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
35. I CAN HAZ ERROR HANDLING NAO? LOL NO 35 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
36. 36 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
37. THIS WORKER IS FAST, THAT WORKER IS SLOW, WHICH ONE WILL CRASH THE DATABASE, NOBODY KNOWS 37 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
38. TIMING IS EVERYTHING!111 38 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
39. 39 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
40. 40 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
41. BATCH PROCESSING 41 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
42. 42 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
43. ABRAKAPOCUS, YOUR DISK SPACE IS GONE 43 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
44. 0 BYTES FREE 44 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
45. MAKE THIS APP GREAT AGAIN AND GET THE CLIENT TO PAY FOR IT 45 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
46. 46 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
47. WARNING: INDEX OUT OF BOUNDS 47 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
48. BUT WHY ISN'T THERE A SEPARATE LOG PARTITION? 48 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
49. 10X 49 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
50. LIVE, AND LEARN, AND DOCUMENT... 50 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
51. WHAT DO YOU MEAN WE CAN'T SCALE? 51 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
52. LOCAL... YOU KEEP USING THAT WORD... 52 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
53. MEMORY REMEMBER TO USE IT 53 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
54. THERE IS AN S IN SUCCESS 54 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
55. 55 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
56. BROWSER BUG REPRODUCTION 56 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
57. CURL REQUEST 57 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
58. OUTPUT THE LAST REQUEST/RESPONSE 58 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
59. WTF????? 59 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
60. 60 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
61. 61 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
62. 18 HOURS TO LOCATE, 20 SECONDS TO FIX 62 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
63. CRATES, BROWSERS AND DATABASES 63 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
64. 16 CPUS 122 GB OF RAM 64 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
65. 65 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
66. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX 66 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
67. 67 — Srdjan Vranac, Code4Hire, CODEstantine 2.0
68. THANK YOU! SRDJAN VRANAC // VRANAC@CODE4HIRE.COM // @VRANAC 68 — Srdjan Vranac, Code4Hire, CODEstantine 2.0