Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
LXJS 2013: backpack — scalable photo storage
Search
Ivan Babrou
October 02, 2013
Programming
2
150
LXJS 2013: backpack — scalable photo storage
http://bobrik.name/talks/lxjs2013.pdf
— slides with notes.
http://youtu.be/T4DgxvS9Xho
— video.
Ivan Babrou
October 02, 2013
Tweet
Share
More Decks by Ivan Babrou
See All by Ivan Babrou
node.js for millions of images
bobrik
7
1.4k
Other Decks in Programming
See All in Programming
Flutterで分数(Fraction)を表示する方法
koukimiura
0
130
iOSエンジニア向けの英語学習アプリを作る!
yukawashouhei
0
200
株式会社 Sun terras カンパニーデック
sunterras
0
340
CSC509 Lecture 03
javiergs
PRO
0
340
What's new in Spring Modulith?
olivergierke
1
160
CSC305 Lecture 08
javiergs
PRO
0
220
One Enishi After Another
snoozer05
PRO
0
110
タスクの特性や不確実性に応じた最適な作業スタイルの選択(ペアプロ・モブプロ・ソロプロ)と実践 / Optimal Work Style Selection: Pair, Mob, or Solo Programming.
honyanya
3
180
『毎日の移動』を支えるGoバックエンド内製開発
yutautsugi
2
250
Building, Deploying, and Monitoring Ruby Web Applications with Falcon (Kaigi on Rails 2025)
ioquatix
4
2.2k
Web フロントエンドエンジニアに開かれる AI Agent プロダクト開発 - Vercel AI SDK を観察して AI Agent と仲良くなろう! #FEC余熱NIGHT
izumin5210
3
550
bootcamp2025_バックエンド研修_WebAPIサーバ作成.pdf
geniee_inc
0
110
Featured
See All Featured
Agile that works and the tools we love
rasmusluckow
331
21k
Building a Scalable Design System with Sketch
lauravandoore
463
33k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.9k
GraphQLとの向き合い方2022年版
quramy
49
14k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Making Projects Easy
brettharned
120
6.4k
How STYLIGHT went responsive
nonsquared
100
5.8k
Documentation Writing (for coders)
carmenintech
75
5.1k
Building an army of robots
kneath
306
46k
Docker and Python
trallard
46
3.6k
A Modern Web Designer's Workflow
chriscoyier
697
190k
Transcript
HI THERE, LXJS
% whoami Ian Babrou, Topface.com
60+ million users 100+ million photos
16 photos on main page up to 200 in feed
many small “previews”
This talk is about PHOTOS
Let’s look at some more numbers
12 storage nodes 70TB total space 44TB used
250 TB per month 1.6 Gbps peak 850 Mbps average
powered by node.js & nginx! open-source FTW
ARCHITECTURE aka part 1
frontend cache resizer storage
of course there are frontends, probably more than one frontend
frontend frontend frontend
round-robin dns + ipvs, probably frontend frontend frontend frontend
NGINX ngx_http_upstream_hash_module is your friend
NGINX ngx_http_upstream_hash_module because you need more than one cache, right?
#protip don’t cache anything twice
don’t do: frontend cache cache cache file#3 file#1 file#2 file#1
file#2 file#2 file#3 file#3 file#1
do: frontend cache cache cache file#3 file#1 file#2
NGINX + SSD is just great for caching, forget about
tmpfs
#protip overallocate caches
RESIZING resizing on the fly saves disk, but eats cpu
NGINX ngx_http_image_filter is your friend
BACKPACK aka part 2
first try nginx
okay for 1k files
okay for 10k files
okay for 50k files
okay until you fit in memory or have ssd
RANDOM ACCESS
DISKS ARE SPINNING
node.js to the rescue!
... and redis
... and zookeeper
simple idea: no extra fseek(3)
inspired by haystack from facebook
concatenate small files into bigger
always keep index in memory
REALIZATION ON DISK
3.5 gb files as many as you need
index for each name:offset:length name:offset:length name:offset:length
but.. no worries, this is only needed if redis goes
crazy
REALIZATION IN MEMORY
keys for files name -> file:offset:length name -> file:offset:length name
-> file:offset:length
redis 3.5gb data + index 3.5gb data + index 3.5gb
data + index memory disk all together:
POWERED BY node.js looks like webdav
PUT: 1. write data 2. write index 3. write redis
key
GET: 1. read redis key 2. read data data files
are always open!
let’s read 100K files! 0 37,5 75 112,5 150 backpack
nginx
LESS SEEKS LEAD TO BIGGER THROUGHPUT
BONUS! linearized access for processing
BUT WHAT ABOUT FUTURE?
NO MORE MEMORY vs DISK* Probably, someday.
MANAGEMENT aka part 3
1. adding servers 2. replication 3. failover
COORDINATOR
COORDINATOR combines servers into shards
COORDINATOR that’s where we need zookeeper
I KNOW let’s use DHT! like dynamo!
Rebalancing on capacity change
NO!
NO. THANK YOU!
LET’S MAKE IT SIMPLE
SHARDS (aka buckets) backpack #1 backpack #2 backpack #3 backpack
#4 backpack #5 backpack #6 shard #1 (50%) shard #2 (50%) 1:lol.jpg 2:wtf.jpg
ADDING SHARD backpack #1 backpack #3 backpack #3 backpack #4
backpack #5 backpack #6 1:lol.jpg 1:wtf.jpg shard #1 (50%) shard #2 (50%) backpack #7 backpack #8 backpack #9 50% chance shard #3 (0%)
COORDINATOR knows how to handle next file
NO REBALANCING SIMPLE
REPLICATOR
WHAT IF METEORITE WILL HIT YOUR NODE?
IT HAPPENS. YOU NEED TO ACCEPT THAT.
REPLICATOR to the rescue!
make multi-node SHARDS
DISTRIBUTE SHARDS ACROSS SERVERS
backpack #1 shard #1 lol.jpg backpack #1 lol.jpg backpack #1
lol.jpg server #1 server #1 server #1 coordinator replicator
REPLICATOR EVENTUALLY MAKES COPIES
THE WHOLE THING IS BULLET-PROOF IF YOU NEED IT
backpack #1 backpack #4 backpack #2 backpack #5 backpack #3
backpack #6 backpack #7 backpack #8 backpack #9 server #1 server #2 server #3 zookeeper #1 zookeeper #2 zookeeper #3 redis-queue #1 redis-queue #2 redis-queue #3 coordinator #1 coordinator #2 coordinator #3 replicator #1 replicator #2 replicator #3
GET THE CODE /Topface/backpack npm install backpack{,-coordinator,-replicator}
That’s it! bobrik ibobrik