Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Learning to Build Distributed Systems the Hard Way

Learning to Build Distributed Systems the Hard Way

Presentation held at JDays 4 December, 2012

8c21306523b16ba5dd35c3549bf90994?s=128

Theo Hultberg

December 04, 2012
Tweet

Transcript

  1. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara

  2. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA

    @iconara
  3. speakerdeck.com/u/iconara (real time!)

  4. Theo / @iconara

  5. chief architect at BURT

  6. let’s make online advertising a great experience

  7. None
  8. MAKING THIS

  9. INTO THIS

  10. HOW HARD CAN IT BE?

  11. None
  12. 30K REQUESTS PER SECOND more than a billion requests per

    day, over 1 TB raw data
  13. ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of

    millions of individual counters per day, plus counting uniques and visitor histories
  14. IN REAL TIME or near real time, if you want

    to be pedantic ×
  15. HOW HARD CAN IT BE?

  16. START WITH TWO OF EVERYTHING going from one to two

    is the hardest, solve the scaling problem up front
  17. START WITH TWO OF EVERYTHING you’ll solve the scaling problem,

    and need less overcapacity THREE
  18. GIVE A LOT OF THOUGHT TO KEYS AND IDS and

    think about your queries first
  19. MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random

  20. JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random

    a timestamp
  21. CONSISTENCY IS OVERRATED don’t fear R + W < N

  22. PRECOMPUTE ALL THE THINGS your users most likely don’t know

    what they want, so why let them do ad hoc queries?
  23. SEPARATE PROCESSING FROM STORAGE that way you can scale each

    independently
  24. PLAN HOW TO GET RID OF YOUR DATA deleting stuff

    is harder than you might think × × × × × × ×
  25. NoDB keep things streaming ×

  26. DIVIDE THE LOAD big data systems are all about routing

    and partitioning
  27. RANDOM when you have no interdependencies between things it’s easy

    to scale out
  28. CONSISTENT when there are interdependencies you need to route using

    some property of the objects, but make sure you get a uniform distribution
  29. NUMEROLOGY

  30. 12

  31. 2 | 12 3 | 12 4 | 12 6

    | 12
  32. 8 | 24 5 | 60

  33. A DIVERSION ABOUT COUNTING TO 60 the reason why there’s

    60 seconds to a minute, and 360 degrees to a circle × ×
  34. 3 SEGMENTS ON EACH FINGER = 12

  35. 3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON

    OTHER HAND = 60
  36. 12, 60, 120, 360 superior highly composite numbers

  37. 12, 60, 120, 360 superior highly composite numbers

  38. 12, 60, 120, 360 superior highly composite numbers

  39. 12, 60, 120, 360 superior highly composite numbers

  40. 12, 60, 120, 360 superior highly composite numbers

  41. 12, 60, 120, 360 superior highly composite numbers

  42. 12, 60, 120, 360 superior highly composite numbers

  43. 12, 60, 120, 360 superior highly composite numbers

  44. 12, 60, 120, 360 superior highly composite numbers

  45. 12, 60, 120, 360 superior highly composite numbers

  46. 12, 60, 120, 360 superior highly composite numbers

  47. 12, 60, 120, 360 superior highly composite numbers

  48. use multiples of 12 to scale without always having to

    double
  49. BLAH BLAH BLAH use multiples of 12 to scale without

    always having to double
  50. log2(366) ≈ 31

  51. $-$ (ASCII code 36)-----

  52. log2(366) ≈ 31

  53. log2(366) ≈ 31 six characters 0-9, A-Z can represent 31

    bits, which is kind of almost very close to four bytes
  54. MEIHO0

  55. MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase

  56. None
  57. YOU CAN’T SCALE TO REAL TIME and don’t trust code

    that doesn’t run continuously ×
  58. DO YOU REALLY NEED A BACKUP? if you got 3x

    replication over multiple availability zones, is that backup really worth it?
  59. PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of

    things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =
  60. GÖTEBORG, DISTRIBUTED @gbgdistr

  61. KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com