Distributed Domain Destruction - Adventures in building distributed systems

For the last three years I have been building, maintaining and fighting a few projects that make use of distributed computing, parallel processing, message brokers, queues and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short sighted albeit logical architectural decisions that will cost you in the long run.

Come hear about the agony you will experience when it starts falling apart, and the thrill you will feel when everything is running juuust right.

Transcript

1. DDD ADVENTURES IN BUILDING DISTRIBUTED SYSTEMS 1 — Srdjan Vranac, Code4Hire, WebcampZg 2016
2. business owner, developer, consultant, mercenary, writing terrible code that performs exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems 2 — Srdjan Vranac, Code4Hire, WebcampZg 2016
3. WARNING 3 — Srdjan Vranac, Code4Hire, WebcampZg 2016
4. DDD 4 — Srdjan Vranac, Code4Hire, WebcampZg 2016
5. DISTRIBUTED 5 — Srdjan Vranac, Code4Hire, WebcampZg 2016
6. DOMAIN 6 — Srdjan Vranac, Code4Hire, WebcampZg 2016
7. DESTRUCTION 7 — Srdjan Vranac, Code4Hire, WebcampZg 2016
8. WHAT IS A DISTRIBUTED SYSTEM? 8 — Srdjan Vranac, Code4Hire, WebcampZg 2016
9. THE SIMPLE DEFINITON It is a system where you can distribute processing of tasks to other workers. By costly taks I mean anything ranging to heavy computation and cpu utilization to long running processes. If we are talking in the context of web applications, basically anything that can't be done within a request is a candidate for distribution/background processing. 9 — Srdjan Vranac, Code4Hire, WebcampZg 2016
10. A MORE REALISTIC DEFINITION A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. — Leslie B. Lamport, 1987 10 — Srdjan Vranac, Code4Hire, WebcampZg 2016
11. 11 — Srdjan Vranac, Code4Hire, WebcampZg 2016
12. 12 — Srdjan Vranac, Code4Hire, WebcampZg 2016
13. 13 — Srdjan Vranac, Code4Hire, WebcampZg 2016
14. 14 — Srdjan Vranac, Code4Hire, WebcampZg 2016
15. CONGRATULATIONS 15 — Srdjan Vranac, Code4Hire, WebcampZg 2016
16. FINITE EXPANSE OF THE UNOPENING WALLET 16 — Srdjan Vranac, Code4Hire, WebcampZg 2016
17. 17 — Srdjan Vranac, Code4Hire, WebcampZg 2016
18. 18 — Srdjan Vranac, Code4Hire, WebcampZg 2016
19. DAEMONS... DAEMONS EVERYWHERE 19 — Srdjan Vranac, Code4Hire, WebcampZg 2016
20. SUPERVISOR UPSTART 20 — Srdjan Vranac, Code4Hire, WebcampZg 2016
21. 21 — Srdjan Vranac, Code4Hire, WebcampZg 2016
22. LOG AND COUNT EVERYTHING 22 — Srdjan Vranac, Code4Hire, WebcampZg 2016
23. AGGREGATE AND SEE 23 — Srdjan Vranac, Code4Hire, WebcampZg 2016
24. 24 — Srdjan Vranac, Code4Hire, WebcampZg 2016
25. STATSD 25 — Srdjan Vranac, Code4Hire, WebcampZg 2016
26. ...BECAUSE LOGS AND COUNTERS ARE LYING TO YOU 26 — Srdjan Vranac, Code4Hire, WebcampZg 2016
27. WHAT HAPPENED TO 40 MILLION RECORDS, WHERE HAVE THEY GONE? 27 — Srdjan Vranac, Code4Hire, WebcampZg 2016
28. WHAT IS DYNAMODB? 28 — Srdjan Vranac, Code4Hire, WebcampZg 2016
29. TEMPORAL TABLES YOU SAY? 29 — Srdjan Vranac, Code4Hire, WebcampZg 2016
30. ARE WE WRITING DATA TO THE DYNAMODB? 30 — Srdjan Vranac, Code4Hire, WebcampZg 2016
31. THE LOGS ARE CLEAN, THE METRICS ARE GREEN 31 — Srdjan Vranac, Code4Hire, WebcampZg 2016
32. NO EXCEPTIONS? 32 — Srdjan Vranac, Code4Hire, WebcampZg 2016
33. 33 — Srdjan Vranac, Code4Hire, WebcampZg 2016
34. WHAT THE HELL DID I JUST READ? 34 — Srdjan Vranac, Code4Hire, WebcampZg 2016
35. I CAN HAZ ERROR HANDLING NAO? LOL NO 35 — Srdjan Vranac, Code4Hire, WebcampZg 2016
36. 36 — Srdjan Vranac, Code4Hire, WebcampZg 2016
37. THIS WORKER IS FAST, THAT WORKER IS SLOW, WHICH ONE WILL CRASH THE DATABASE, NOBODY KNOWS 37 — Srdjan Vranac, Code4Hire, WebcampZg 2016
38. TIMING IS EVERYTHING!111 38 — Srdjan Vranac, Code4Hire, WebcampZg 2016
39. 39 — Srdjan Vranac, Code4Hire, WebcampZg 2016
40. 40 — Srdjan Vranac, Code4Hire, WebcampZg 2016
41. BATCH PROCESSING 41 — Srdjan Vranac, Code4Hire, WebcampZg 2016
42. 42 — Srdjan Vranac, Code4Hire, WebcampZg 2016
43. ABRAKAPOCUS, YOUR DISK SPACE IS GONE 43 — Srdjan Vranac, Code4Hire, WebcampZg 2016
44. 0 BYTES FREE 44 — Srdjan Vranac, Code4Hire, WebcampZg 2016
45. MAKE THIS APP GREAT AGAIN AND GET THE CLIENT TO PAY FOR IT 45 — Srdjan Vranac, Code4Hire, WebcampZg 2016
46. 46 — Srdjan Vranac, Code4Hire, WebcampZg 2016
47. WARNING: INDEX OUT OF BOUNDS 47 — Srdjan Vranac, Code4Hire, WebcampZg 2016
48. BUT WHY ISN'T THERE A SEPARATE LOG PARTITION? 48 — Srdjan Vranac, Code4Hire, WebcampZg 2016
49. 10X 49 — Srdjan Vranac, Code4Hire, WebcampZg 2016
50. LIVE, AND LEARN, AND DOCUMENT... 50 — Srdjan Vranac, Code4Hire, WebcampZg 2016
51. THERE IS AN S IN SUCCESS 51 — Srdjan Vranac, Code4Hire, WebcampZg 2016
52. 52 — Srdjan Vranac, Code4Hire, WebcampZg 2016
53. BROWSER BUG REPRODUCTION 53 — Srdjan Vranac, Code4Hire, WebcampZg 2016
54. CURL REQUEST 54 — Srdjan Vranac, Code4Hire, WebcampZg 2016
55. OUTPUT THE LAST REQUEST/RESPONSE 55 — Srdjan Vranac, Code4Hire, WebcampZg 2016
56. WTF????? 56 — Srdjan Vranac, Code4Hire, WebcampZg 2016
57. 57 — Srdjan Vranac, Code4Hire, WebcampZg 2016
58. 58 — Srdjan Vranac, Code4Hire, WebcampZg 2016
59. 18 HOURS TO LOCATE, 20 SECONDS TO FIX 59 — Srdjan Vranac, Code4Hire, WebcampZg 2016
60. CRATES, BROWSERS AND DATABASES 60 — Srdjan Vranac, Code4Hire, WebcampZg 2016
61. 16 CPUS 122 GB OF RAM 61 — Srdjan Vranac, Code4Hire, WebcampZg 2016
62. 62 — Srdjan Vranac, Code4Hire, WebcampZg 2016
63. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX 63 — Srdjan Vranac, Code4Hire, WebcampZg 2016
64. 64 — Srdjan Vranac, Code4Hire, WebcampZg 2016
65. THANK YOU! SRDJAN VRANAC // VRANAC@CODE4HIRE.COM // @VRANAC HTTPS://JOIND.IN/TALK/17558 65 — Srdjan Vranac, Code4Hire, WebcampZg 2016