Learning to Build Distributed Systems the Hard Way

Transcript

1. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara

2. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA

   @iconara

3. speakerdeck.com/u/iconara (real time!)

4. Theo / @iconara

5. chief architect at BURT

6. let’s make online advertising a great experience

7. None

8. MAKING THIS

9. INTO THIS

10. HOW HARD CAN IT BE?

11. None

12. 30K REQUESTS PER SECOND more than a billion requests per

    day, over 1 TB raw data

13. ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of

    millions of individual counters per day, plus counting uniques and visitor histories

14. IN REAL TIME or near real time, if you want

    to be pedantic ×

15. HOW HARD CAN IT BE?

16. START WITH TWO OF EVERYTHING going from one to two

    is the hardest, solve the scaling problem up front

17. START WITH TWO OF EVERYTHING you’ll solve the scaling problem,

    and need less overcapacity THREE

18. GIVE A LOT OF THOUGHT TO KEYS AND IDS and

    think about your queries first

19. MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random

20. JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random

    a timestamp

21. CONSISTENCY IS OVERRATED don’t fear R + W < N

22. PRECOMPUTE ALL THE THINGS your users most likely don’t know

    what they want, so why let them do ad hoc queries?

23. SEPARATE PROCESSING FROM STORAGE that way you can scale each

    independently

24. PLAN HOW TO GET RID OF YOUR DATA deleting stuff

    is harder than you might think × × × × × × ×

25. NoDB keep things streaming ×

26. DIVIDE THE LOAD big data systems are all about routing

    and partitioning

27. RANDOM when you have no interdependencies between things it’s easy

    to scale out

28. CONSISTENT when there are interdependencies you need to route using

    some property of the objects, but make sure you get a uniform distribution

29. NUMEROLOGY

30. 12

31. 2 | 12 3 | 12 4 | 12 6

    | 12

32. 8 | 24 5 | 60

33. A DIVERSION ABOUT COUNTING TO 60 the reason why there’s

    60 seconds to a minute, and 360 degrees to a circle × ×

34. 3 SEGMENTS ON EACH FINGER = 12

35. 3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON

    OTHER HAND = 60

36. 12, 60, 120, 360 superior highly composite numbers

37. 12, 60, 120, 360 superior highly composite numbers

38. 12, 60, 120, 360 superior highly composite numbers

39. 12, 60, 120, 360 superior highly composite numbers

40. 12, 60, 120, 360 superior highly composite numbers

41. 12, 60, 120, 360 superior highly composite numbers

42. 12, 60, 120, 360 superior highly composite numbers

43. 12, 60, 120, 360 superior highly composite numbers

44. 12, 60, 120, 360 superior highly composite numbers

45. 12, 60, 120, 360 superior highly composite numbers

46. 12, 60, 120, 360 superior highly composite numbers

47. 12, 60, 120, 360 superior highly composite numbers

48. use multiples of 12 to scale without always having to

    double

49. BLAH BLAH BLAH use multiples of 12 to scale without

    always having to double

50. log2(366) ≈ 31

51. $-$ (ASCII code 36)-----

52. log2(366) ≈ 31

53. log2(366) ≈ 31 six characters 0-9, A-Z can represent 31

    bits, which is kind of almost very close to four bytes

54. MEIHO0

55. MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase

56. None

57. YOU CAN’T SCALE TO REAL TIME and don’t trust code

    that doesn’t run continuously ×

58. DO YOU REALLY NEED A BACKUP? if you got 3x

    replication over multiple availability zones, is that backup really worth it?

59. PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of

    things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =

60. GÖTEBORG, DISTRIBUTED @gbgdistr

61. KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com