Learning to Build Distributed Systems the Hard Way

Presentation held at JDays 4 December, 2012

Transcript

1. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY @iconara
2. LEARNING TO BUILD DISTRIBUTED SYSTEMS THE HARD WAY BIG DATA @iconara
3. speakerdeck.com/u/iconara (real time!)
4. Theo / @iconara
5. chief architect at BURT
6. let’s make online advertising a great experience
8. MAKING THIS
9. INTO THIS
10. HOW HARD CAN IT BE?
12. 30K REQUESTS PER SECOND more than a billion requests per day, over 1 TB raw data
13. ONE VISIT CAN CHANGE UP TO 100K COUNTERS hundreds of millions of individual counters per day, plus counting uniques and visitor histories
14. IN REAL TIME or near real time, if you want to be pedantic ×
15. HOW HARD CAN IT BE?
16. START WITH TWO OF EVERYTHING going from one to two is the hardest, solve the scaling problem up front
17. START WITH TWO OF EVERYTHING you’ll solve the scaling problem, and need less overcapacity THREE
18. GIVE A LOT OF THOUGHT TO KEYS AND IDS and think about your queries first
19. MEIHO0 JME57Z monotonically increasing, sorts nicely a timestamp something random
20. JME57Z MEIHO0 uniformly distributed, works nicely with sharding something random a timestamp
21. CONSISTENCY IS OVERRATED don’t fear R + W < N
22. PRECOMPUTE ALL THE THINGS your users most likely don’t know what they want, so why let them do ad hoc queries?
23. SEPARATE PROCESSING FROM STORAGE that way you can scale each independently
24. PLAN HOW TO GET RID OF YOUR DATA deleting stuff is harder than you might think × × × × × × ×
25. NoDB keep things streaming ×
26. DIVIDE THE LOAD big data systems are all about routing and partitioning
27. RANDOM when you have no interdependencies between things it’s easy to scale out
28. CONSISTENT when there are interdependencies you need to route using some property of the objects, but make sure you get a uniform distribution
29. NUMEROLOGY
31. 2 | 12 3 | 12 4 | 12 6 | 12
32. 8 | 24 5 | 60
33. A DIVERSION ABOUT COUNTING TO 60 the reason why there’s 60 seconds to a minute, and 360 degrees to a circle × ×
34. 3 SEGMENTS ON EACH FINGER = 12
35. 3 SEGMENTS ON EACH FINGER = 12 FIVE FINGERS ON OTHER HAND = 60
36. 12, 60, 120, 360 superior highly composite numbers
37. 12, 60, 120, 360 superior highly composite numbers
38. 12, 60, 120, 360 superior highly composite numbers
39. 12, 60, 120, 360 superior highly composite numbers
40. 12, 60, 120, 360 superior highly composite numbers
41. 12, 60, 120, 360 superior highly composite numbers
42. 12, 60, 120, 360 superior highly composite numbers
43. 12, 60, 120, 360 superior highly composite numbers
44. 12, 60, 120, 360 superior highly composite numbers
45. 12, 60, 120, 360 superior highly composite numbers
46. 12, 60, 120, 360 superior highly composite numbers
47. 12, 60, 120, 360 superior highly composite numbers
48. use multiples of 12 to scale without always having to double
49. BLAH BLAH BLAH use multiples of 12 to scale without always having to double
50. log2(366) ≈ 31
51. $-$ (ASCII code 36)-----
52. log2(366) ≈ 31
53. log2(366) ≈ 31 six characters 0-9, A-Z can represent 31 bits, which is kind of almost very close to four bytes
54. MEIHO0
55. MEIHO0 a timestamp Time.now.to_i.to_s(36).upcase
57. YOU CAN’T SCALE TO REAL TIME and don’t trust code that doesn’t run continuously ×
58. DO YOU REALLY NEED A BACKUP? if you got 3x replication over multiple availability zones, is that backup really worth it?
59. PRODUCTION IS THE ONLY REAL TEST ENVIRONMENT when thousands of things happen every second, new, weird and unforeseen things happen all the time, your tests can only cover the foreseeable =
60. GÖTEBORG, DISTRIBUTED @gbgdistr
61. KTHXBAI @iconara github.com/iconara architecturalatrocities.com burtcorp.com