Tuning Lustre WAN for 100 Gbit Networks

In this talk I describe the SC11 SCinet Research Sandbox entry by Indiana University. Results and lessons learned (specifically from LNET) are presented.

Transcript

1. Experiences from the SCinet Research Sandbox How to Tune Your Wide Area File System for a 100 Gbps Network Scott Michael LUG2012 April 24,2012
2. April 24, 2012 100 Gbps Wide Area Lustre Talk Roadmap •  Background: IU’s Lustre-WAN efforts to date •  Lustre-WAN at 100 Gbps: SC11 SCinet Research Sandbox entry •  LNET measurements: Important tunables 2 of 14
3. Wide Area Lustre in Production at IU April 24, 2012 100 Gbps Wide Area Lustre 3 of 14 Researcher's Computer Compute Resource #1 Compute Resource #2 Data Source Visualization Resource Tape Archive Data Capacitor
4. Lustre-WAN at IU •  We have had and currently have several remote client production mounts with a range of bandwidths and latencies •  Clients connected at 1 Gbit and 10 Gbit •  Clients connected across various regional, national, and international networks •  Latencies ranging from a few milliseconds to 120 milliseconds April 24, 2012 100 Gbps Wide Area Lustre 4 of 14
5. 100 Gbits Over Low Latency April 24, 2012 100 Gbps Wide Area Lustre 5 of 14 Dresden to Freiberg − 60 km − 0.72 ms Throughput 10.8 GB/s − 86% efficiency
6. 100 Gbits Over a Bit More Latency •  Indiana University submitted an entry to the SC11 SCinet Research Sandbox program to demonstrate cross-country 100 Gbit/s Lustre performance •  The demonstration included network benchmarks, LNET testing, file system benchmarks, and a suite of real-world scientific workflows April 24, 2012 100 Gbps Wide Area Lustre 6 of 14
7. SCinet Research Sandbox Setup Seattle to Indianapolis − 3,500 km − 50.5 ms April 24, 2012 100 Gbps Wide Area Lustre 7 of 14
8. SCinet Research Sandbox Outcome Measurement Efficiency Latency 50.5 ms − TCP iperf 96 Gbit/s 96% IOR 6.5 GB/s 52% Applications 6.2 GB/s 50% April 24, 2012 100 Gbps Wide Area Lustre 8 of 14 •  Relatively small cluster •  20 hours of test, troubleshoot, and demo time
9. Workflow Suite •  Enzo – astronomical adaptive mesh code •  Vampir – parallel tracing code and debugger •  Heat3d – heat diffusion code •  ODI – astronomical image reduction pipeline •  NCGAS – genomics codes •  OLAM – climate code •  CMES – Computational Model for Electroencephalography responses in Schizophrenia - computational neuroscience •  Gromacs – molecular dynamics code April 24, 2012 100 Gbps Wide Area Lustre 9 of 14
10. More RPCs Are Needed! •  For high latency links max_rpcs_in_flight has to be increased from the default of 8 •  One can show the max throughput for a given connection is: or to maximize a given link… April 24, 2012 100 Gbps Wide Area Lustre 10 of 14 throughput = RPCs×block _ size 2× RTT
11. More RPCs Are Needed! •  For high latency links max_rpcs_in_flight has to be increased from the default of 8 •  One can show the max throughput for a given connection is: or to maximize a given link… April 24, 2012 100 Gbps Wide Area Lustre 10 of 14 RPCs > 2× BDP block _ size
12. What We Learned About credits •  Initial LNET testing for a single client/ server showed we were unable to achieve theoretical throughput •  Throughput leveled off past RPCs of 8 •  This was due to the default settings of credits and peer_credits April 24, 2012 100 Gbps Wide Area Lustre 11 of 14
13. What We Learned About credits •  Initial LNET testing for a single client/ server showed we were unable to achieve theoretical throughput •  Throughput leveled off past RPCs of 8 •  This was due to the default settings of credits and peer_credits April 24, 2012 100 Gbps Wide Area Lustre 11 of 14
14. What We Learned About credits •  Single client/server LNET performance was 1092 MB/s − 89% efficiency •  We saw somewhat improved performance with the entire system and increased credits, but less than expected April 24, 2012 100 Gbps Wide Area Lustre 12 of 14
15. What We Learned About credits •  Single client/server LNET performance was 1092 MB/s − 89% efficiency •  We saw somewhat improved performance with the entire system and increased credits, but less than expected April 24, 2012 100 Gbps Wide Area Lustre 12 of 14
16. Summary and Implications •  Cross-country 100 Gbit networks are here or coming soon •  Lustre-WAN is a useful tool for empowering geographically distributed scientific workflows •  Centers that deploy Lustre-WAN systems should consider the impact of RPCs and credits! •  Multiple wide area/local client endpoints require some planning when setting tunables April 24, 2012 100 Gbps Wide Area Lustre 13 of 14
17. Thank You for Your Attention Questions? Scott Michael Indiana University scamicha@iu.edu April 24, 2012 100 Gbps Wide Area Lustre 14 of 14 Look for the LNET paper at DIDC2012 in conjunction with HPDC A Study of Lustre Networking Over a 100 Gigabit Wide Area Network with 50 milliseconds of Latency, DIDC ‘12