Share
Embed
#50 “Scalable Hierarchical Aggregation Protocol (SHArP): A Hardware Architecture for Efficient Data Reduction”
COM-HPC '16: Proceedings of the First Workshop on Optimization of Communication in HPC in SC
- https://dl.acm.org/doi/10.5555/3018058.3018059
- https://network.nvidia.com/sites/default/files/related-docs/solutions/hpc/paperieee_copyright.pdf
- https://dl.acm.org/doi/proceedings/10.5555/3018058
cafenero_777
January 25, 2024
Transcript
23 slides
-
Research Paper Introduction #50 “Scalable Hierarchical Aggregation Protocol (SHArP): A
Hardware Architecture for E ff i cient Data Reduction” ௨ࢉ#123 @cafenero_777 2024/01/25 1 -
ରจ •Scalable Hierarchical Aggregation Protocol (SHArP): A Hardware Architecture for
E ff i cient Data Reduction • Richard L. Graham, et al • Mellanox Technologies, Inc. • COM-HPC '16: Proceedings of the First Workshop on Optimization of Communication in HPC in SC • https://dl.acm.org/doi/10.5555/3018058.3018059 • https://network.nvidia.com/sites/default/ fi les/related-docs/solutions/hpc/paperieee_copyright.pdf • https://dl.acm.org/doi/proceedings/10.5555/3018058 2 -
Agenda •ରจ •֓ཁͱಡ͏ͱͨ͠ཧ༝ 1. INTRODUCTION 2. PREVIOUS WORK 3. AGGREGATION
PROTOCOL 4. MPI IMPLEMENTATION 5. EXPERIMENTS 6. MICRO-BENCHMARK RESULTS 7. OPENFOAM PREFORMANCE 8. DISCUSSION 9. CONCLUSIONS 3 -
֓ཁͱಡ͏ͱͨ͠ཧ༝ •֓ཁ • ΑΓେنͳγϛϡϨʔγϣϯ༻ͷΞʔΩςΫνϟΛࡧ • DCͷେنฒྻ௨৴༻ʢCollective Communication༻ʣͷco- processerΛεΠονʹ࣮ • 2~3.2ഒੑೳ্
•ಡ͏ͱͨ͠ཧ༝ • SmartNIC/DPUͰݴٴ͞Ε͍ͯͨͷͰɻ • ੲMPIΛͬͨ͜ͱ͕͋ΔͷͰɻ 4 େنฒྻܭࢉʢΠϝʔδਤʣ -
Collective Communication? •ฒྻܭࢉ༻ͷܭࢉॲཧ •Map/Shu ffl e/Reduce, AllReduce, AllGather, ͳͲͳͲɻ •
https://www.janog.gr.jp/meeting/janog53/dcqos/ P.35-39 • https://www.janog.gr.jp/meeting/janog53/ainw/ P.15-17 •NCCL • https://developer.nvidia.com/nccl 5 ߹ͷͨΊɺreductionΛred.ͱུه͠·͢ɻ CCO: Operation CCL: Library -
1. INTRODUCTION •HPCੑೳ্ͷྺ࢙: ϕΫτϧܕܭࢉػ -> PCΫϥελʢࢢൢCPUʣΫϥελ -> ϚϧνίΞCPU •Compute Engineͷ૿Ճ͕伴:
CPU (X86, Power, ARM), GPGPU, FPGA • C.E.ؒΛNWॏཁ͕ͩɺdataC.E.ͷlocality͕ڧ͗͢ɻదٓࢄ͍ͤͨ͞ -> Co-Designඞཁ •Mellanox: CPUΦϑϩʔυʹϑΥʔΧε͠ɺpipeline࠷దͰϨΠςϯγԽΛࢦͨ͠ • MPIAPIఏڙ͢Δ͜ͱͰརศੑΛ࣮ݱ •Scalable Hierarchical Aggregation ProtocolʢSHArPʣ • ReductionΦϖϨʔγϣϯ༻ͷϓϩτίϧఆٛͱSwitchIB-2σόΠε࣮ • Reduction treeΛ༻͍Δ͜ͱͰɺCPUܭࢉΦϑϩʔυͭͭ͠ɺreductionࣗମݮ 6 -
2. PREVIOUS WORK •ઌߦݚڀ: blocking/non-blockingͷόϦΞɾϦμΫγϣϯΞϧΰϦζϜվળ • ΞϧΰϦζϜ: Sequential, chain, binary,
binomial tree • τϙϩδ: ϝογϡɺϋΠύʔΩϡʔϒ •HW࣮Ͱੑೳ্: broadcast, barrier, reduction, torus interconnect w/all- to-all CCͰ2ഒੑೳ্ @ IBM Blue Gene 7 -
3. AGGREGATION PROTOCOL •NW co-processor architectureͷత • ʢߴසͳʣ௨৴ྃ࣌ؒͷ࠷దԽͰCPU༻ͷ࠷খԽ • ࠷ॳͷλʔήοτglobal
reduction, barrierಉظ •SHArP: reductionॲཧΛநԽ • AN: Aggregation Node • (ॲཧ͞ΕΔtreeͰ)ANͰred.͞Εͳ͕ΒπϦʔΛ্Δɻ • RootʹͳΔͱglobal aggregateྃɻूύλʔϯʹґଘ͠ͳ͍ɻ • NW্Ͱॲཧ͞Εɺσʔλͷಈ͖࠷খԽɻHigh-radix (shallow red. tree)ͷԸܙɻ 8 -
3. AGGREGATION PROTOCOL A: Aggregation Nodes. •AN: ूπϦʔͷϊʔυ • ࢠϊʔυ͔ΒσʔλΛड͚औΓɺred.͠ɺ
(or ֤ϊʔυ)ʹసૹ • σʔλίώʔϨϯεҡ࣋ʢbarrierಉظvector reductionʣ • ࣮ํ๏: αʔόεΠον্ͷϓϩηεɺεΠονͷHW 9 -
3. AGGREGATION PROTOCOL B: Aggregation Tree. •End-nodeANʹଓ͞ΕΔ • ඞͣ͠ཧτϙϩδʢ࠷ଓʣʹै͏ඞཁແ͠ •ANANʹଓ͞ΕΔ
•IB/RoCElosslessͳτϥϯεϙʔτલఏ •πϦʔީิ: Fat-tree, Dragon fl y+, hypercube 10 -
3. AGGREGATION PROTOCOL C: Aggregation Group. •AG: AN͕௨৴͢Δ૬खͷఆٛʢGroupʣΛ࡞ΓɺϦιʔε༻Λ্ • ෳjob,
ෳgroupΛαϙʔτɻ • ʢࢠʣSub-treeΛ࡞Γɺrootʹసૹ͢ΔσʔλΛݮ͢Δ • ࢄάϧʔϓੜͷΞϧΰϦζϜຊߘൣғ֎ • Dead lockͷՄೳੑΛճආ͢ΔͨΊʹANͰόοϑΝΛอ࣋ •ิ: “௨৴͢Δ૬ख” • communicator: MPIจ຺Ͱͷ௨৴૬ख: ϓϩηεϊʔυͳͲɻMPI_Commܕɻ 11 -
3. AGGREGATION PROTOCOL D: Aggregation Operations. 12 •AO: ֤AGͷϝϯόʔͷϦΫΤετͰ։࢝ •
SHArPϦΫΤετϝοηʔδͱred.ϖΠϩʔυ • Data type, size, # of element, operation (ex. Min) • ྃͨ͠ΒAN݁ՌΛసૹ •MPI Reduce/AllReduceͷ߹Agg TreeΛ༻͍ͯIBϚϧνΩϟεɻ -
3. AGGREGATION PROTOCOL E: Fault Handling. 13 •Errorछผ: τϥϯεϙʔτɺΤϯυϊʔυɺSHArPϓϩτίϧ •
ANϦιʔε։์͠ɺϗετϓϩηεʹ௨ɻ • ΞϓϦଆͰΤϥʔճ෮orதஅͤ͞Δ •Aggregation Manager (AM)͕Τϥʔཧ͠Ϧιʔε։์ • SHArP INTͱIB NWࢹΛར༻ɻMPI/SHMEM༷Ͱtimeoutར༻Ͱ͖ͣɻ -
3. AGGREGATION PROTOCOL F: SwitchIB-2-Based Aggregation Support. •ANϩδοΫεΠονASICʹ౷߹͞ΕͨIB TCAͱͯ͠AGΛ࣮ •SHArpϓϩτίϧʹج͍ͮͯಈ࡞
•ANػೳ • ALU: 32/64bit, un/signed int/ fl oat • Sum, Min/Max, Min/MaxLoc, per bit OR/AND/XOR •request౸ணॱͰܭࢉ”ग़དྷͳ͍” • FloatͷੵՄ͕ͩassociative(݁߹ଇ)Λຬͨ͞ͳ͍ɻ fl oatingܭࢉͰ0 or notɻ • ݻఆॱংͰܭࢉ͢Δ: SwitchIB-2༧ଌՄೳͳԋࢉॱংΛ࣮͠ɺܭࢉʹ࠶ݱੑΛ࣋ͨͤͨɻ 14 1020 − (1020 + ϵ) = 0 (1020 − 1020) + ϵ = ϵ -
4. MPI IMPLEMENTATION •SHArPάϧʔϓΛMPIίϛϡχέʔλʹϚοϐϯά • άϧʔϓੜɺάϧʔϓτϦϜૢ࡞ɺϦιʔεׂ • Non-blocking I/Fܦ༝ͰLeaf SWʹRC
QPཁٻ • MPI Barrier() : SHArP barrier() • MPI Reduce(), MPI Allreduce() : allreduce SHArP work request • ະྃCCOͱϦιʔεΛτϥοΩϯά͠ɺ৽نॲཧΛԆͤ͞Δɻ 15 -
5. EXPERIMENTS •128 nodes, dual socket 2.6GHz Intel E5-2697 v3
Haswell •SwitchIB-2, ConnectX-4 HCA •RHEL 7.2, OFED v3.2-1.0.1.1 IB verbs •Latencyͷ҆ఆԽ • C-Stateݻఆɺdisable THP • BIOSͰSMIׂΓࠐΈແޮԽɺHT/TBແޮԽ 16 $cat /etc/cmdline intel pstate=disable, isolcpus=1, nohz full=1, rcu nocbs=1 $ sysctl -a … vm_stat interval=1200 kernel.numa balancing=0 kernel.nmi watchdog=0 and disable THP … -
5. EXPERIMENTS •Library • HPC-XϓϨϦϦʔε൛ w/ SHArP: MellanoxHPCιϑτΣΞελοΫ • MPI
collective FCA library v3.4 w/ SHArP • MVAPICH-2 v2.2: OSSͳMPI࣮ •Performance Tests • Low-level verbs test, OSU collective latency test v5.2 for MPI Barrier() & MPI Allreduce() • Single process per host • SHArPόϦΞϝοηʔδΛHCAʹϙετ -> SHAaPΩϡʔΛpollingͯ͠red.࣮ߦ • 1 MPIϝοηʔδ͕ෳSHArPϝοηʔδʹׂ͞Εɺಉ࣌ʹin- fl ight͢Δ 17 -
6. MICRO-BENCHMARK RESULTS A. Native SHArP Measurements 18 ϕʔεϥΠϯͷଌఆ -ϗετ͕ଟ͍΄ͲԆඍ૿
-ϝοηʔδ͕ଟ͍΄ͲԆඍ૿ OS/BIOSͷ࠷దԽ༗Γແ͠ൺֱ -64ϗετͩͱࠩ5%ఔ -128ϗετͩͱࠩ10-20%ʂ -
6. MICRO-BENCHMARK RESULTS B. MPI-Level SHArP Measurements 19 MPIܦ༝20-40%ఔͷΦʔόʔϔου ʢ200-400nsͷΦʔόʔϔουʣ
Red.ͷαΠζʹґଘ͠ͳ͍ɻ SHArP w/ HWΦϑϩʔυɺ HPC-X (IB w/ multicast)ΑΓҰ؏͍ͯ͠ɻ ಉ͘͡ϗετϕʔεͷIB MVAPICH2ඇৗʹ͍ɻɻ ࠷େαΠζΛ͑ͨ߹ɺ ͘ͳΔ͕ɺred.ΛฒྻԽɾ pipelineԽͰԆ૿ՃΛ੍ -
7. OPENFOAM PERFORMANCE •ΞϓϦͰͷੑೳධՁ • OpenFOAMʢྲྀମղੳΞϓϦʣ • 100ສηϧͰicoFoamιϧόʔ༷ • όϦΞΦϖϨʔγϣϯ͕খ͍͞ͷʢͪ
͕ଟ͍ͱ”ͪ”Λଌͬͯ͠·͏ʣ • ΞϓϦෛՙͷภΓcollective operation durationΑΓখ͍͞ํ͕ྑ͍ʢͦ͏͠ͳ͍ ͱ”ภΓ”Λଌͬͯ͠·͏ʣ • 8byte MPI Allreduce()͕Ұ൪༻͞ΕΔ 20 Host baseͱൺͯɺ -0.5%~15%ఔੑೳ্ɻ - ΑΓεέʔϧ͍͢͠ɻ - ੑೳ্5%ఔ্͕ݶɻ - OpenMPIͱൺֱ͢Δͱ4-50%্ʂ -
8. DISCUSSION •খɾதنred.ॲཧՊֶٕज़ܭࢉͰසൟʹར༻ • ಉظ͕໌ࣔతɾඇ໌ࣔతʹߦΘΕΔɻ • SHArPͰͷੑೳ্͕ߩݙͰ͖Δ •SwitchIB-2ͷ36port (High-radix)ͰANؒΛऩ༰͠ϨΠςϯγίετͱऩ༰Λ࠷దԽɻ2 level
FAT-TreeͰଌఆ Λ࣮ݱɻ •SHArPͷNWΞϓϩʔνΛڧௐ͢ΔͨΊɺطଘͷϗετϕʔεͱൺֱ͍ͯ͠Δ •BG/PͷϨΠςϯγʹඖఢ • εΠονόοϑΝ֬อͷͨΊʹԾϨʔϯΛઐ༗͢Δඞཁͳ͠ɻ • ΑΓ͍πϦʔɺNodeΛαϙʔτՄೳɻ 21 -
9. CONCLUSION •SwitchIB-2 ASIC SHArP architecture/implementation • Small-size data reductionͰੑೳ্
• AggTreeΛܗ͠ฒྻԽ͠ɺεέʔϧΞτΛظ • reduction operationͷpipelineԽͰreduction sizeΛ͛ΒΕΔ • HCA (ΤϯυϙΠϯτ)ΞϓϩʔνΑΓɺhigh-radixͳIBεΠονͰߴू ʢσʔλύεॖʣɾߴฒྻԽ͕༰қ 22