LXJS 2013: backpack — scalable photo storage

http://bobrik.name/talks/lxjs2013.pdf — slides with notes.
http://youtu.be/T4DgxvS9Xho — video.

Transcript

1. HI THERE, LXJS
2. % whoami Ian Babrou, Topface.com
3. 60+ million users 100+ million photos
4. 16 photos on main page up to 200 in feed many small “previews”
5. This talk is about PHOTOS
6. Let’s look at some more numbers
7. 12 storage nodes 70TB total space 44TB used
8. 250 TB per month 1.6 Gbps peak 850 Mbps average
9. powered by node.js & nginx! open-source FTW
10. ARCHITECTURE aka part 1
11. frontend cache resizer storage
12. of course there are frontends, probably more than one frontend frontend frontend frontend
13. round-robin dns + ipvs, probably frontend frontend frontend frontend
14. NGINX ngx_http_upstream_hash_module is your friend
15. NGINX ngx_http_upstream_hash_module because you need more than one cache, right?
16. #protip don’t cache anything twice
17. don’t do: frontend cache cache cache file#3 file#1 file#2 file#1 file#2 file#2 file#3 file#3 file#1
18. do: frontend cache cache cache file#3 file#1 file#2
19. NGINX + SSD is just great for caching, forget about tmpfs
20. #protip overallocate caches
21. RESIZING resizing on the fly saves disk, but eats cpu
22. NGINX ngx_http_image_filter is your friend
23. BACKPACK aka part 2
24. first try nginx
25. okay for 1k files
26. okay for 10k files
27. okay for 50k files
28. okay until you fit in memory or have ssd
29. RANDOM ACCESS
30. DISKS ARE SPINNING
31. node.js to the rescue!
32. ... and redis
33. ... and zookeeper
34. simple idea: no extra fseek(3)
35. inspired by haystack from facebook
36. concatenate small files into bigger
37. always keep index in memory
38. REALIZATION ON DISK
39. 3.5 gb files as many as you need
40. index for each name:offset:length name:offset:length name:offset:length
41. but.. no worries, this is only needed if redis goes crazy
42. REALIZATION IN MEMORY
43. keys for files name -> file:offset:length name -> file:offset:length name -> file:offset:length
44. redis 3.5gb data + index 3.5gb data + index 3.5gb data + index memory disk all together:
45. POWERED BY node.js looks like webdav
46. PUT: 1. write data 2. write index 3. write redis key
47. GET: 1. read redis key 2. read data data files are always open!
48. let’s read 100K files! 0 37,5 75 112,5 150 backpack nginx
49. LESS SEEKS LEAD TO BIGGER THROUGHPUT
50. BONUS! linearized access for processing
51. BUT WHAT ABOUT FUTURE?
52. NO MORE MEMORY vs DISK* Probably, someday.
53. MANAGEMENT aka part 3
54. 1. adding servers 2. replication 3. failover
55. COORDINATOR
56. COORDINATOR combines servers into shards
57. COORDINATOR that’s where we need zookeeper
58. I KNOW let’s use DHT! like dynamo!
59. Rebalancing on capacity change
60. NO!
61. NO. THANK YOU!
62. LET’S MAKE IT SIMPLE
63. SHARDS (aka buckets) backpack #1 backpack #2 backpack #3 backpack #4 backpack #5 backpack #6 shard #1 (50%) shard #2 (50%) 1:lol.jpg 2:wtf.jpg
64. ADDING SHARD backpack #1 backpack #3 backpack #3 backpack #4 backpack #5 backpack #6 1:lol.jpg 1:wtf.jpg shard #1 (50%) shard #2 (50%) backpack #7 backpack #8 backpack #9 50% chance shard #3 (0%)
65. COORDINATOR knows how to handle next file
66. NO REBALANCING SIMPLE
67. REPLICATOR
68. WHAT IF METEORITE WILL HIT YOUR NODE?
69. IT HAPPENS. YOU NEED TO ACCEPT THAT.
70. REPLICATOR to the rescue!
71. make multi-node SHARDS
72. DISTRIBUTE SHARDS ACROSS SERVERS
73. backpack #1 shard #1 lol.jpg backpack #1 lol.jpg backpack #1 lol.jpg server #1 server #1 server #1 coordinator replicator
74. REPLICATOR EVENTUALLY MAKES COPIES
75. THE WHOLE THING IS BULLET-PROOF IF YOU NEED IT
76. backpack #1 backpack #4 backpack #2 backpack #5 backpack #3 backpack #6 backpack #7 backpack #8 backpack #9 server #1 server #2 server #3 zookeeper #1 zookeeper #2 zookeeper #3 redis-queue #1 redis-queue #2 redis-queue #3 coordinator #1 coordinator #2 coordinator #3 replicator #1 replicator #2 replicator #3
77. GET THE CODE /Topface/backpack npm install backpack{,-coordinator,-replicator}
78. That’s it! bobrik ibobrik