◦ Price, performance, security ◦ Flexibility to use big JBODS ◦ Power efficiency ★ We use Go programming language ★ 100k LoC (lines of code) We built our own storage software
small files 100KB - 500KB HDD can’t handle IO operation per every file limits throughput Packing data to blocks increases latency uncontrollably Add SSD Often the solution is to add either SSD write cache or convert whole storage to SSD Lower protection If service is more expensive then less data is saved. For example, backup frequency or retention period is lowered. TB Price increases SSD is more expensive, so the service becomes more expensive or less profitable and sustainable.
to do the work of SSD ★ Small blocks on low load moments ★ Big blocks on high load moments ★ Result: steady streams of data that are easy to write fast to HDD and have latency guarantees to customers SSD performance on HDD
for different tasks ★ Every server is connected to the internet ★ Every server is connected to a JBOD ★ All software services run on all servers Physical setup is hyperconverged
V4 S3 V4 Data in rest encrypted by AES-XTS AES-XTS Erasure Coding (EC) to protect data integrity EC Key ID and Secret Key Keys Data integrity re-checked every 60 days (bit-rot and other silent failures) Data integrity