PWLSF#11 => Alex Rasmussen on Flat Datacenter Storage

Alex Rasmussen presents the "Flat Datacenter Storage" paper by Edmund B. Nightingale, Jeremy Elson,Jinliang Fan, Owen Hofmann, Jon Howell, and Yutaka Suzue. http://css.csail.mit.edu/6.824/2014/papers/fds.pdf

Alex tells us: " Flat Datacenter Storage (FDS) is, as the intro describes, "a high-performance, fault-tolerant, large-scale, locality-oblivious blob store". It's also a great example of how carefully thought-out co-design of software and hardware for a target workload can yield really impressive performance results, even in the presence of heterogeneity and operating at scale. In my (admittedly biased) opinion, this style of system design doesn't get enough attention outside of academia, and has a lot to teach us about how data-intensive systems should be designed."

If you have any questions, thoughts, or related information, please visit our *github-thread* on the matter: https://github.com/papers-we-love/papers-we-love/issues/198

Alex's Bio:

Alex Rasmussen (@alexras) got his Ph.D. from UC San Diego in 2013. While at UCSD, he worked on really efficient large-scale data processing and set a few world sorting records, which makes him a hit at parties. He's currently working at Trifacta, helping build the industry's leading big data washing machine.

The video of this talk is up!! https://www.youtube.com/watch?v=F7heZ9ZBWqI

Transcript

1. Flat Datacenter Storage Presented by Alex Rasmussen Papers We Love SF #11 2015-01-22 Edmund B. Nightingale, Jeremy Elson, Jinliang Fan,
 Owen Hofmann, Jon Howell, and Yutaka Suzue
2. @alexras
3. Sort Really Fast THEMIS MapReduce Really Fast
4. Image Credit: http://bit.ly/17Vf8Hb
5. A Perfect World
6. “Magic RAID”
9. The Real World
11. Move the Computation to the Data!
12. Location Awareness Adds Complexity
13. Why
    “Move the Computation
    to the Data”?
14. Remote Data Access is Slow.
    Why?
15. The Network is Oversubscribed
16. Core Aggregation Edge ure 1: Common data center interconnect topology. Host to switch links are GigE and links between switches are 10 G 25 30 35 40 1:1 3:1 7:1 Fat-tree Hierarchical design Fat-tree Year 10 GigE Hosts Cost/ GigE Hosts C GigE G 2002 28-port 4,480 $25.3K 28-port 5,488 $ 2004 32-port 7,680 $4.4K 48-port 27,648 $ 2006 64-port 10,240 $2.1K 48-port 27,648 $ 2008 128-port 20,480 $1.8K 48-port 27,648 $ Aggregate Bandwidth Above Less Than Aggregate Demand Below Sometimes by 100x or more A B
17. What if I told you the network isn’t oversubscribed?
18. Consequences • No local vs. remote disk distinction • Simpler work schedulers • Simpler programming models
19. FDS
    Object Storage Assuming
    No Oversubscription
20. Motivation Architecture and API Metadata Management Replication and Recovery Data Transport Why FDS Matters
22. Blob 0xbadf00d Tract 0 Tract 1 Tract 2 Tract n ... 8 MB CreateBlob OpenBlob CloseBlob DeleteBlob GetBlobSize ExtendBlob ReadTract WriteTract
23. API Guarantees • Tractserver writes are atomic • Calls are asynchronous - Allows deep pipelining • Weak consistency to clients
24. Motivation Architecture and API Metadata Management Replication and Recovery Data Transport Why FDS Matters
25. Tract Locator Version TS 1 0 A 2 0 B 3 2 D 4 0 A 5 3 C 6 0 F ... ... ... Tract Locator Table
26. Tract_Locator = 
 TLT[(Hash(GUID) + Tract) % len(TLT)]
27. Randomize blob’s tractserver, even if GUIDs aren’t random (uses SHA-1) Tract_Locator = 
 TLT[(Hash(GUID) + Tract) % len(TLT)]
28. Large blobs use all TLT entries uniformly Tract_Locator = 
 TLT[(Hash(GUID) + Tract) % len(TLT)]
29. Blob Metadata is Distributed Tract_Locator = 
 TLT[(Hash(GUID) - 1) % len(TLT)]
30. TLT Construction • m Permutations of Tractserver List • Weighted by disk speed • Served by metadata server to clients • Only update when cluster changes
31. Tract Locator Version TS 1 0 A 2 0 B 3 2 D 4 0 A 5 3 C 6 0 F ... ... ... Cluster Growth
32. Tract Locator Version TS 1 1 NEW / A 2 0 B 3 2 D 4 1 NEW / A 5 4 NEW / C 6 0 F ... ... ... Cluster Growth
33. Tract Locator Version TS 1 2 NEW 2 0 A 3 2 A 4 2 NEW 5 5 NEW 6 0 A ... ... ... Cluster Growth
34. Motivation Architecture and API Metadata Management Replication and Recovery Data Transport Why FDS Matters
35. Tract Locator Version Replica 1 Replica 2 Replica 3 1 0 A B C 2 0 A C Z 3 0 A D H 4 0 A E M 5 0 A F G 6 0 A G P ... ... ... ... ... Replication
36. Tract Locator Version Replica 1 Replica 2 Replica 3 1 0 A B C 2 0 A C Z 3 0 A D H 4 0 A E M 5 0 A F G 6 0 A G P ... ... ... ... ... Replication
37. Replication • Create, Delete, Extend: - client writes to primary - primary 2PC to replicas • Write to all replicas • Read from random replica
38. Tract Locator Version Replica 1 Replica 2 Replica 3 1 0 A B C 2 0 A C Z 3 0 A D H 4 0 A E M 5 0 A F G 6 0 A G P ... ... ... ... ... Recovery
39. Tract Locator Version Replica 1 Replica 2 Replica 3 1 0 A B C 2 0 A C Z 3 0 A D H 4 0 A E M 5 0 A F G 6 0 A G P ... ... ... ... ... Recovery Recover 1TB from 3000 disks in < 20 seconds H E A L M E
40. MIND BLOWN
41. Motivation Architecture and API Metadata Management Replication and Recovery Data Transport Why FDS Matters
42. Networking Pod 0 10.0.2.1 10.0.1.1 Pod 1 Pod 3 Pod 2 10.2.0.2 10.2.0.3 10.2.0.1 10.4.1.1 10.4.1.2 10.4.2.1 10.4.2.2 Core 10.2.2.1 10.0.1.2 Edge Aggregation Figure 3: Simple fat-tree topology. Using the two-level routing tables described in Section 3.3, packets from source 10.0.1.2 to destination 10.2.0.3 would take the dashed path. Prefix 10.2.0.0/24 10.2.1.0/24 0.0.0.0/0 Output port 0 1 Suffix Output port Next hop 10.2.0.1 10.2.1.1 10.4.1.1 Address 00 01 10 Output port 0 1 2 RAM Encoder 10.2.0.X 10.2.1.X X.X.X.2 TCAM CLOS topology: small switches + ECMP 
 = full bisection bandwidth
43. Networking • Network bandwidth = disk bandwidth • Full bisection bandwidth is stochastic • Short flows good for ECMP • TCP hates short flows • RTS/CTS to mitigate incast; see paper
44. Motivation Architecture and API Metadata Management Replication and Recovery Data Transport Why FDS Matters
45. Co-Design
46. Hardware/Software Combination Designed for a Specific Workload
47. FDS Works Great for Blob Storage on CLOS Networks
48. www.sortbenchmark.org
49. Indy Daytona
50. MinuteSort - Daytona System! (Nodes) Year Data Sorted Speed 
 per Disk Hadoop (1408) 2009 500GB 3 MB/s FDS (256) 2012 1470GB 46 MB/s
51. MinuteSort - Indy System! (Nodes) Year Data Sorted Speed 
 per Disk TritonSort (66) 2012 1353GB 43.3 MB/s FDS (256) 2012 1470GB 47.9 MB/s
52. FDS isn’t built for oversubscribed networks.
 It’s also not a DBMS.
53. MapReduce and GFS:
    Thousands of
    Cheap PCs,
    Bulk Synchronous Processing
54. 10x MapReduce and GFS Aren’t Designed for or Iterative, or OLAP
55. FDS’ Lessons • Great example of ground-up rethink - Ambitious but implementable • Big wins possible with co-design • Constantly re-examine assumptions
57. TritonSort & Themis • Balanced hardware architecture • Full bisection-bandwidth network • Job-level fault tolerance • Huge wins possible - Beat 3000+ node cluster by 35% with 52 nodes • NSDI 2012, SoCC 2013
58. Thanks @alexras • alexras@acm.org • alexras.info