Distributed Domain Destruction - Adventures in building distributed systems

For the last three years I have been building, maintaining and fighting a few projects that make use of distributed computing, parallel processing, message brokers, queues and workers. This is one of those "from the trenches" talks, where I will regale you with tales about the series of unfortunate events that may happen as your application grows in complexity. Tales like disk space fluctuations, importance of logging, NoSql problems, restructuring your order of execution in code for performance gains, short sighted albeit logical architectural decisions that will cost you in the long run.

Come hear about the agony you will experience when it starts falling apart, and the thrill you will feel when everything is running juuust right.

Transcript

1. DDD
   ADVENTURES IN BUILDING
   DISTRIBUTED SYSTEMS
   1 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

2. business owner, developer, consultant, mercenary,
   writing terrible code that performs exceptionally,
   wrangling elePHPants and Pythons, obsessed with
   process automation, interested in continuous integration
   and delivery, clean code, testing, best practices and
   distributed systems
   2 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

3. WARNING
   3 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

4. DDD
   4 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

5. DISTRIBUTED
   5 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

6. DOMAIN
   6 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

7. DESTRUCTION
   7 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

8. WHAT IS
   A DISTRIBUTED SYSTEM?
   8 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

9. THE SIMPLE DEFINITON
   It is a system where you can distribute processing of
   tasks to other workers.
   By costly taks I mean anything ranging to heavy
   computation and cpu utilization to long running
   processes.
   If we are talking in the context of web applications,
   basically anything that can't be done within a request is
   a candidate for distribution/background processing.
   9 — Srdjan Vranac, Code4Hire, WebcampZg 2016
   

   View Slide

10. A MORE REALISTIC DEFINITION
    A distributed system is one in which the failure
    of a computer you didn't even know existed can
    render your own computer unusable.
    — Leslie B. Lamport, 1987
    10 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

11. 11 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

12. 12 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

13. 13 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

14. 14 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

15. CONGRATULATIONS
    15 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

16. FINITE EXPANSE OF THE
    UNOPENING WALLET
    16 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

17. 17 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

18. 18 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

19. DAEMONS...
    DAEMONS EVERYWHERE
    19 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

20. SUPERVISOR
    UPSTART
    20 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

21. 21 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

22. LOG AND COUNT
    EVERYTHING
    22 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

23. AGGREGATE AND SEE
    23 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

24. 24 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

25. STATSD
    25 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

26. ...BECAUSE
    LOGS AND COUNTERS
    ARE LYING TO YOU
    26 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

27. WHAT HAPPENED TO
    40 MILLION RECORDS,
    WHERE HAVE THEY GONE?
    27 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

28. WHAT IS DYNAMODB?
    28 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

29. TEMPORAL TABLES
    YOU SAY?
    29 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

30. ARE WE WRITING DATA TO
    THE DYNAMODB?
    30 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

31. THE LOGS ARE CLEAN,
    THE METRICS ARE GREEN
    31 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

32. NO EXCEPTIONS?
    32 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

33. 33 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

34. WHAT THE HELL
    DID I JUST READ?
    34 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

35. I CAN HAZ ERROR
    HANDLING NAO?
    LOL NO
    35 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

36. 36 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

37. THIS WORKER IS FAST,
    THAT WORKER IS SLOW,
    WHICH ONE WILL CRASH
    THE DATABASE,
    NOBODY KNOWS
    37 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

38. TIMING IS EVERYTHING!111
    38 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

39. 39 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

40. 40 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

41. BATCH PROCESSING
    41 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

42. 42 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

43. ABRAKAPOCUS, YOUR
    DISK SPACE IS GONE
    43 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

44. 0 BYTES FREE
    44 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

45. MAKE THIS APP GREAT AGAIN
    AND GET THE CLIENT TO PAY FOR IT
    45 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

46. 46 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

47. WARNING: INDEX OUT OF
    BOUNDS
    47 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

48. BUT WHY ISN'T THERE A
    SEPARATE LOG PARTITION?
    48 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

49. 10X
    49 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

50. LIVE,
    AND LEARN,
    AND DOCUMENT...
    50 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

51. THERE IS AN S IN
    SUCCESS
    51 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

52. 52 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

53. BROWSER BUG
    REPRODUCTION
    53 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

54. CURL REQUEST
    54 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

55. OUTPUT THE LAST
    REQUEST/RESPONSE
    55 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

56. WTF?????
    56 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

57. 57 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

58. 58 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

59. 18 HOURS TO LOCATE,
    20 SECONDS TO FIX
    59 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

60. CRATES, BROWSERS AND
    DATABASES
    60 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

61. 16 CPUS 122 GB OF RAM
    61 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

62. 62 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

63. BARCODE SCANNERS AND KEYBOARDS DO NOT MIX
    63 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

64. 64 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide

65. THANK YOU!
    SRDJAN VRANAC // VRANAC@CODE4HIRE.COM // @VRANAC
    HTTPS://JOIND.IN/TALK/17558
    65 — Srdjan Vranac, Code4Hire, WebcampZg 2016
    

    View Slide