LeoFS - Open the New Door

This slides is moving from [http://www.slideshare.net/rakutentech2013/tech-conf2013d3-1]; because we can not access the slideshare site in China.
I want to embed this slides in my blog, so I move it here.

All rights reserved by Rakuten
----
- LeoFS is an object storage system shaped for the Web.
- LeoFS is built to operate in highly distributed environments, such as the cloud.
- LeoFS has NO single point of failure (SPOF).
- A LeoFS cluster is a set of loosely connected nodes working as a single massive cloud storage entity.
- LeoFS is modular, allowing you to control the size and operation costs of your cluster.

Transcript

1. Open the New Door Yosuke Hara Oct 26, 2013 (rev 2.2) The Lion of Storage Systems 1
2. Started OSS-project on July 4, 2012 www.leofs.org LeoFS is "Unstructured Big Data Storage for the Web" and a highly available, distributed, eventually consistent storage system. Organizations can use LeoFS to store lots of data efficently, safely and inexpensively. 2
3. Motivation 3
4. 1. High Costs (Initial Costs, Running Costs) 2. Possibility of "SPOF" 3. NOT Easily Scale Storage Expansion is difficult during periods of increasing data Expensive Storage Problems: Motivation ? Get Away From Using "Expensive H/W Based Storages" As of 2010 4
5. 
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
   
    REST-API / AWS S3-API The Lion of Storage Systems HIGH Availability HIGH Cost Performance Ratio HIGH Scalability LeoFS Non Stop Velocity: Low Latency Minimum Resources Volume: Petabyte / Exabyte Variety: Photo, Movie, Unstructured-data 3 Vs in 3 HIGHs 5
6. Overview 6
7. metadata Object Store Storage Engine/Router metadata Object Store Storage Engine/Router LeoFS-Manager 3&45
   PWFS
   )551
    31$
    Request from Web Applications/ Browsers w/REST-API / S3-API metadata Object Store Storage Engine/Router Load Balancer Monitor GUI Console    
    31$
    No Master No SPOF LeoFS Overview LeoFS-Gateway LeoFS-Storage 7
8. Gateway 8
9. LeoFS Overview - Gateway Stateless Proxy + Object Cache REST-API / S3-API Use Consistent Hashing for decision of a primary node [ Memory Cache, Disc Cache ] Storage Cluster Gateway(s) Clients Handle HTTP Request and Response Built in "Object Cache Mechanism" Storage Cluster Choosing Replica Target Node(s) RING 2 ^ 128 (MD5) # of replicas = 3 KEY = “bucket/leofs.key” Hash = md5(Filename) Secondary-1 Secondary-2 Primary Node 9
10. Storage 10
11. Storage (Storage Cluster) Gateway Automatically Replicate an Object and a Metadata to Remote Node(s) LeoFS Overview - Storage Use "Consistent Hashing" for Replication in the Storage Cluster Choosing Replica Target Node(s) RING 2 ^ 128 (MD5) # of replicas = 3 KEY = “bucket/leofs.key” Hash = md5(Filename) Secondary-1 Secondary-2 Primary Node "P2P" 11
12. Request From Gateway LeoFS Overview - Storage  LeoFS Storage Metadata : Keeps an in-memory index of all data Object Container : Manages "Log Structured File" Replicator Repairer w/Queue ... Storage Engine Workers Storage Engine, Metadata + Object Container Gateway Storage Engine consits of "Object Storage" and "Metadata Storage" Built in "Replicator", "Recoverer" w/Queue for the Eventual Consistency 12
13. LeoFS Storage Engine - Retrieve an object from the storage < META DATA > ID Filename Offset Size Checksum Header File Footer < META DATA > Id Filename Offset, Size Checksum (MD5) Version# Storage Engine Worker Object Container Metadata Storage Storage Engine Worker 13
14. LeoFS Storage Engine - Retrieve an object from the storage < META DATA > ID Filename Offset Size Checksum Header File Footer < META DATA > Id Filename Offset, Size Checksum (MD5) Version# Object Container Metadata Storage Storage Engine Worker Insert a metadata Append an object into the object container Storage Engine Worker 14
15. LeoFS Storage Engine - Remove unnecessary objects from the storage Compact Old Object Container/Metadata Storage Engine Worker New Object Container/Metadata Storage Engine Worker 15
16. Offset Version Time- stamp {VNodeId, Key} <Metadata> Checksum for Sync KeySize Custom Meta Size File Size for Retrieve an File (Object) Footer (8B) Checksum KeySize DataSize Offset Version Time- stamp {VNodeId, Key} User-Meta Footer Header (Metadata - Fixed length) Body (Variable Length) User-Meta Size Actual File <Needle> Super-block Needle-1 Needle-2 Needle-3 <Object Container> Needle-4 Needle-5 LeoFS Overview - Storage - Data Structure/Relationship an object 16
17. To Equalize Disk Usage of Every Storage Node To Realize High I/O efficiency and High Availability LeoFS Overview - Storage - Large Object Support chunk-0 chunk-1 chunk-2 chunk-3 An Original Object’s Metadata Original Object Name Original Object Size # of Chunks Storage Cluster Gateway Client(s) [ WRITE Operation ] Chunked Objects Every chunked object and metadata are replicated in the cluster 17
18. Manager 18
19. Storage Cluster LeoFS Overview - Manager Monitor Operate RING, Node State status, suspend, resume, detach, whereis, ... Gateway(s) Storage Cluster Gateway(s) Manager(s) Operate LeoFS - Gateway and Storage Cluster "RING Monitor" and "NodeState Monitor" 19
20. New Features 20
21. "Insight" 21
22. Give Insight into the State of LeoFS 1. To control requests from Clients to LeoFS 2. To check and see "Traffic info" and "State of Every Node" for Keeping Availability New Features - LeoInsight (v1.0) 22
23. Storage Cluster Manager Gateway The Lion of Storage Systems TimeSeriesDB (Savannah) Persistent calculated statistics-data REST-API (JSON) Operate LeoFS Notifier Distributed Queue (ElkDB) Traffic-Info from G atew ay Consume MSG Retrieve Proves of a Node from G ateway/Storage/M anager Notify New Features - LeoInsight (v1.0) 23
24. More Scalability & More Availability 24
25. Tokyo Europe US New Features - Multi Data Center Data Replication (v1.0) HIGH-Scalability HIGH-Availability Easy Operation for Admins + NO SPOF NO Performance Degration Singapore 25
26. DC-3 DC-2 v1.0 - Multi Data Center Data Replication Storage cluster Manager cluster Client DC-1 Monitor and Replicate each “RING” and “System Configuration” "Leo Storage Platform" [replicas:1] [replicas:1] Request to the Target Region Application(s) [ 3 Regions & 5 Replicas ] Method of MDC-Replication: Async: Bulked Transfer Sync+Tran: Consensus Algorithm DC-1 Configuration: - Method of Replication: - Consistency Level: - local-quorum:[N=3, W=2, R=1, D=2] - # of target DC(s):2 - # of replicas a DC:1 >> Total of Replicas: 5 [replicas:3] 26
27. 1) 3 replicas are written in "Local Region" DC-3 DC-2 v1.0 - Multi Data Center Data Replication Storage cluster Manager cluster Client DC-1 Monitor and Replicate each “RING” and “System Configuration” "Leo Storage Platform" [replicas:1] [replicas:1] Request to the Target Region Application(s) [ 3 Regions & 5 Replicas ] Method of MDC-Replication: Async: Bulked Transfer Sync+Tran: Consensus Algorithm DC-1 Configuration: - Method of Replication: - Consistency Level: - local-quorum:[N=3, W=2, R=1, D=2] - # of target DC(s):2 - # of replicas a DC:1 >> Total of Replicas: 5 [replicas:3] 27
28. 2) Sync (or Async) Rplicaion to Other Region(s) DC-3 DC-2 v1.0 - Multi Data Center Data Replication Storage cluster Manager cluster Client DC-1 Monitor and Replicate each “RING” and “System Configuration” "Leo Storage Platform" Request to the Target Region Application(s) [ 3 Regions & 5 Replicas ] [replicas:3] Leader Follower DC1.node_0 - Primary DC1.node_1 DC1.node_2 DC2.node_3 DC3.node_4 Local-follower Remote-follower [replicas:1] [replicas:1] 28
29. v1.0 - Multi Data Center Data Replication Storage cluster Manager cluster Client "Leo Storage Platform" DC-3 US DC-2 Singapore DC-1 Tokyo Monitor and Replicate each “RING” and “System Configuration” [replicas:3] [replicas:1] [replicas:1] DC-4 Europe Request to the Target Region Application(s) [ 3 Regions & 5 Replicas ] 3) Replication for Geographical Optimization Local Region Remote-1 Remote-2 Tokyo Singapore US Singapore Tokyo Europe Europe US Singapore US Europe Tokyo 29
30. "Center" 30
31. Web-based administrative console for inspecting and manipulating LeoFS Storage Clusters and LeoFS Gateway Operate LeoFS New Features - LeoCenter Admin Tools Access Log Analysis 31
32. Access Log Analysis (β) 32
33. 
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
    
     REST-API / AWS S3-API The Lion of Storage Systems HIGH Availability HIGH Cost Performance Ratio HIGH Scalability LeoFS Non Stop Velocity: Low Latency Minimum Resources Volume: Petabyte / Exabyte Variety: Photo, Movie, Unstructured-data 3 Vs in 3 HIGHs 33
34. Set Sail for “Cloud Storage” Website: www.leofs.org Twitter: @LeoFastStorage Facebook: www.facebook.com/org.leofs 34