Pro Yearly is on sale from $80 to $50! »

Distributed Domain Destruction - Adventures in building distributed systems

F2d82b268a7cbccc9809c939428df64f?s=47 Vranac Srdjan
September 21, 2018

Distributed Domain Destruction - Adventures in building distributed systems

For the last few years I have been building, maintaining and fighting a few projects that make use of distributed computing, parallel processing, message brokers, queues and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short sighted albeit logical architectural decisions that will cost you in the long run.

Come hear about the agony you will experience when it starts falling apart, and the thrill you will feel when everything is running juuust right.

F2d82b268a7cbccc9809c939428df64f?s=128

Vranac Srdjan

September 21, 2018
Tweet

Transcript

  1. DDD ADVENTURES IN BUILDING DISTRIBUTED SYSTEMS 1 — Srdjan Vranac,

    Code4Hire, PHP-Usergroup Frankfurt Am Main
  2. business owner, developer, consultant, mercenary, writing terrible code that performs

    exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems 2 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  3. WARNING 3 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  4. DDD 4 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  5. DISTRIBUTED 5 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  6. DOMAIN 6 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  7. DESTRUCTION 7 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  8. WHAT IS A DISTRIBUTED SYSTEM? 8 — Srdjan Vranac, Code4Hire,

    PHP-Usergroup Frankfurt Am Main
  9. THE SIMPLE DEFINITON It is a system where you can

    distribute processing of costly tasks to other workers. By costly taks I mean anything ranging to heavy computation and cpu utilization to long running processes. If we are talking in the context of web applications, basically anything that can't be done within a request is a candidate for distribution/background processing. 9 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  10. A MORE REALISTIC DEFINITION A distributed system is one in

    which the failure of a computer you didn't even know existed can render your own computer unusable. — Leslie B. Lamport, 1987 10 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  11. 11 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  12. 12 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  13. 13 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  14. 14 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  15. CONGRATULATIONS 15 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  16. FINITE EXPANSE OF THE UNOPENING WALLET 16 — Srdjan Vranac,

    Code4Hire, PHP-Usergroup Frankfurt Am Main
  17. 17 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  18. 18 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  19. DAEMONS... DAEMONS EVERYWHERE 19 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt

    Am Main
  20. SUPERVISOR UPSTART 20 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am

    Main
  21. 21 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  22. LOG AND COUNT EVERYTHING 22 — Srdjan Vranac, Code4Hire, PHP-Usergroup

    Frankfurt Am Main
  23. AGGREGATE AND SEE 23 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt

    Am Main
  24. 24 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  25. STATSD 25 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  26. ...BECAUSE LOGS AND COUNTERS ARE LYING TO YOU 26 —

    Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  27. WHAT HAPPENED TO 40 MILLION RECORDS, WHERE HAVE THEY GONE?

    27 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  28. WHAT IS DYNAMODB? 28 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt

    Am Main
  29. TEMPORAL TABLES YOU SAY? 29 — Srdjan Vranac, Code4Hire, PHP-Usergroup

    Frankfurt Am Main
  30. ARE WE WRITING DATA TO THE DYNAMODB? 30 — Srdjan

    Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  31. THE LOGS ARE CLEAN, THE METRICS ARE GREEN 31 —

    Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  32. NO EXCEPTIONS? 32 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am

    Main
  33. 33 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  34. WHAT THE HELL DID I JUST READ? 34 — Srdjan

    Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  35. I CAN HAZ ERROR HANDLING NAO? LOL NO 35 —

    Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  36. 36 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  37. THIS WORKER IS FAST, THAT WORKER IS SLOW, WHICH ONE

    WILL CRASH THE DATABASE, NOBODY KNOWS 37 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  38. TIMING IS EVERYTHING!111 38 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt

    Am Main
  39. 39 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  40. 40 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  41. BATCH PROCESSING 41 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am

    Main
  42. 42 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  43. ABRAKAPOCUS, YOUR DISK SPACE IS GONE 43 — Srdjan Vranac,

    Code4Hire, PHP-Usergroup Frankfurt Am Main
  44. 0 BYTES FREE 44 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt

    Am Main
  45. MAKE THIS APP GREAT AGAIN AND GET THE CLIENT TO

    PAY FOR IT 45 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  46. 46 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  47. WARNING: INDEX OUT OF BOUNDS 47 — Srdjan Vranac, Code4Hire,

    PHP-Usergroup Frankfurt Am Main
  48. BUT WHY ISN'T THERE A SEPARATE LOG PARTITION? 48 —

    Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  49. 10X 49 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  50. LIVE, AND LEARN, AND DOCUMENT... 50 — Srdjan Vranac, Code4Hire,

    PHP-Usergroup Frankfurt Am Main
  51. WHAT DO YOU MEAN WE CAN'T SCALE? 51 — Srdjan

    Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  52. LOCAL... YOU KEEP USING THAT WORD... 52 — Srdjan Vranac,

    Code4Hire, PHP-Usergroup Frankfurt Am Main
  53. MEMORY REMEMBER TO USE IT 53 — Srdjan Vranac, Code4Hire,

    PHP-Usergroup Frankfurt Am Main
  54. THERE IS AN S IN SUCCESS 54 — Srdjan Vranac,

    Code4Hire, PHP-Usergroup Frankfurt Am Main
  55. 55 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  56. BROWSER BUG REPRODUCTION 56 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt

    Am Main
  57. CURL REQUEST 57 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am

    Main
  58. OUTPUT THE LAST REQUEST/RESPONSE 58 — Srdjan Vranac, Code4Hire, PHP-Usergroup

    Frankfurt Am Main
  59. WTF????? 59 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  60. 60 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  61. 61 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  62. 18 HOURS TO LOCATE, 20 SECONDS TO FIX 62 —

    Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  63. CRATES, BROWSERS AND DATABASES 63 — Srdjan Vranac, Code4Hire, PHP-Usergroup

    Frankfurt Am Main
  64. 16 CPUS 122 GB OF RAM 64 — Srdjan Vranac,

    Code4Hire, PHP-Usergroup Frankfurt Am Main
  65. 65 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  66. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX 66 — Srdjan

    Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main
  67. 67 — Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main

  68. THANK YOU! SRDJAN VRANAC // VRANAC@CODE4HIRE.COM // @VRANAC 68 —

    Srdjan Vranac, Code4Hire, PHP-Usergroup Frankfurt Am Main