Pro Yearly is on sale from $80 to $50! »

Distributed Domain Destruction - Adventures in building distributed systems

Distributed Domain Destruction - Adventures in building distributed systems

For the last three years I have been building, maintaining and fighting a few projects that make use of distributed computing, parallel processing, message brokers, queues and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short sighted albeit logical architectural decisions that will cost you in the long run.

Come hear about the agony you will experience when it starts falling apart, and the thrill you will feel when everything is running juuust right.

F2d82b268a7cbccc9809c939428df64f?s=128

Vranac Srdjan

October 28, 2016
Tweet

Transcript

  1. DDD ADVENTURES IN BUILDING DISTRIBUTED SYSTEMS 1 — Srdjan Vranac,

    Code4Hire, WebcampZg 2016
  2. business owner, developer, consultant, mercenary, writing terrible code that performs

    exceptionally, wrangling elePHPants and Pythons, obsessed with process automation, interested in continuous integration and delivery, clean code, testing, best practices and distributed systems 2 — Srdjan Vranac, Code4Hire, WebcampZg 2016
  3. WARNING 3 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  4. DDD 4 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  5. DISTRIBUTED 5 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  6. DOMAIN 6 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  7. DESTRUCTION 7 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  8. WHAT IS A DISTRIBUTED SYSTEM? 8 — Srdjan Vranac, Code4Hire,

    WebcampZg 2016
  9. THE SIMPLE DEFINITON It is a system where you can

    distribute processing of tasks to other workers. By costly taks I mean anything ranging to heavy computation and cpu utilization to long running processes. If we are talking in the context of web applications, basically anything that can't be done within a request is a candidate for distribution/background processing. 9 — Srdjan Vranac, Code4Hire, WebcampZg 2016
  10. A MORE REALISTIC DEFINITION A distributed system is one in

    which the failure of a computer you didn't even know existed can render your own computer unusable. — Leslie B. Lamport, 1987 10 — Srdjan Vranac, Code4Hire, WebcampZg 2016
  11. 11 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  12. 12 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  13. 13 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  14. 14 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  15. CONGRATULATIONS 15 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  16. FINITE EXPANSE OF THE UNOPENING WALLET 16 — Srdjan Vranac,

    Code4Hire, WebcampZg 2016
  17. 17 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  18. 18 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  19. DAEMONS... DAEMONS EVERYWHERE 19 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  20. SUPERVISOR UPSTART 20 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  21. 21 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  22. LOG AND COUNT EVERYTHING 22 — Srdjan Vranac, Code4Hire, WebcampZg

    2016
  23. AGGREGATE AND SEE 23 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  24. 24 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  25. STATSD 25 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  26. ...BECAUSE LOGS AND COUNTERS ARE LYING TO YOU 26 —

    Srdjan Vranac, Code4Hire, WebcampZg 2016
  27. WHAT HAPPENED TO 40 MILLION RECORDS, WHERE HAVE THEY GONE?

    27 — Srdjan Vranac, Code4Hire, WebcampZg 2016
  28. WHAT IS DYNAMODB? 28 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  29. TEMPORAL TABLES YOU SAY? 29 — Srdjan Vranac, Code4Hire, WebcampZg

    2016
  30. ARE WE WRITING DATA TO THE DYNAMODB? 30 — Srdjan

    Vranac, Code4Hire, WebcampZg 2016
  31. THE LOGS ARE CLEAN, THE METRICS ARE GREEN 31 —

    Srdjan Vranac, Code4Hire, WebcampZg 2016
  32. NO EXCEPTIONS? 32 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  33. 33 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  34. WHAT THE HELL DID I JUST READ? 34 — Srdjan

    Vranac, Code4Hire, WebcampZg 2016
  35. I CAN HAZ ERROR HANDLING NAO? LOL NO 35 —

    Srdjan Vranac, Code4Hire, WebcampZg 2016
  36. 36 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  37. THIS WORKER IS FAST, THAT WORKER IS SLOW, WHICH ONE

    WILL CRASH THE DATABASE, NOBODY KNOWS 37 — Srdjan Vranac, Code4Hire, WebcampZg 2016
  38. TIMING IS EVERYTHING!111 38 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  39. 39 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  40. 40 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  41. BATCH PROCESSING 41 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  42. 42 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  43. ABRAKAPOCUS, YOUR DISK SPACE IS GONE 43 — Srdjan Vranac,

    Code4Hire, WebcampZg 2016
  44. 0 BYTES FREE 44 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  45. MAKE THIS APP GREAT AGAIN AND GET THE CLIENT TO

    PAY FOR IT 45 — Srdjan Vranac, Code4Hire, WebcampZg 2016
  46. 46 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  47. WARNING: INDEX OUT OF BOUNDS 47 — Srdjan Vranac, Code4Hire,

    WebcampZg 2016
  48. BUT WHY ISN'T THERE A SEPARATE LOG PARTITION? 48 —

    Srdjan Vranac, Code4Hire, WebcampZg 2016
  49. 10X 49 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  50. LIVE, AND LEARN, AND DOCUMENT... 50 — Srdjan Vranac, Code4Hire,

    WebcampZg 2016
  51. THERE IS AN S IN SUCCESS 51 — Srdjan Vranac,

    Code4Hire, WebcampZg 2016
  52. 52 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  53. BROWSER BUG REPRODUCTION 53 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  54. CURL REQUEST 54 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  55. OUTPUT THE LAST REQUEST/RESPONSE 55 — Srdjan Vranac, Code4Hire, WebcampZg

    2016
  56. WTF????? 56 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  57. 57 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  58. 58 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  59. 18 HOURS TO LOCATE, 20 SECONDS TO FIX 59 —

    Srdjan Vranac, Code4Hire, WebcampZg 2016
  60. CRATES, BROWSERS AND DATABASES 60 — Srdjan Vranac, Code4Hire, WebcampZg

    2016
  61. 16 CPUS 122 GB OF RAM 61 — Srdjan Vranac,

    Code4Hire, WebcampZg 2016
  62. 62 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  63. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX 63 — Srdjan

    Vranac, Code4Hire, WebcampZg 2016
  64. 64 — Srdjan Vranac, Code4Hire, WebcampZg 2016

  65. THANK YOU! SRDJAN VRANAC // VRANAC@CODE4HIRE.COM // @VRANAC HTTPS://JOIND.IN/TALK/17558 65

    — Srdjan Vranac, Code4Hire, WebcampZg 2016