GPUネットワーク設計・運用 基礎勉強会 Lossless Ethernet – PFC/ECN編

1. GPUネットワーク設計・運⽤ 基礎勉強会 Lossless Ethernet ‒ PFC/ECN編  .BTBZVLJ
   ,PCBZBTIJ
2. ˙ର৅ 3P$&Wʹશ͘৮Εͨ͜ͱͷແ͍ωοτϫʔΫΤϯδχΞ -εΠονͷӡ༻ܦݧ͕͋Δ͜ͱ ˙ΰʔϧ -PTTMFTT
   &UIFSOFUͷ֓ཁΛཧղ͢Δ 3P$&WωοτϫʔΫΛߏங͢ΔͷʹͲͷΑ͏ͳٕज़ͱߟྀࣄ߲͕͋Δͷ͔ཧղ͢Δ ˙஫ҙࣄ߲ ຊࢿྉͷ಺༰͸ωοτϫʔΫΤϯδχΞ͕शಘ͠΍͍͢جૅతͳ෦෼ͷΈΛهࡌ͍ͯ͠·͢ (16Ϋϥελಛ༗ͷωοτϫʔΫߏ੒΍αʔόͷ಺෦ߏ੒ɺ$$-ʹ͍ͭͯ͸͜͜Ͱ͸৮Ε·ͤΜ ର৅ͱΰʔϧ  はじめに
3. Lossless Ethernet (16ωοτϫʔΫʹϩεϨε͕ඞཁͳཧ༝  IUUQTTQFBLFSEFDLDPNNBSLVOFULVSBVUPUFUBTFOUBOFUVUPXBLVOPJNBUPLPSFLBSB
4. Backend Network Ϋϥελؒ௨৴ *OUFSDPOOFDU ઐ༻ͷωοτϫʔΫ  5SBJOJOH $MVTUFS *OGFSFODF $MVTUFS 4UPSBHF $MVTUFS -FBG -FBG 4QJOF 4QJOF -FBG -FBG -FBG -FBG -FBG -FBG 4QJOF 4QJOF -PTTMFTT
   'BCSJD
   #BDLFOE • "*
   $IJQؒͷ௨৴ʹ࠷దԽ • 3BJMPQUJNJ[FE
   5PQPMPHZ • ௒௿஗Ԇɾ௒޿ଳҬɾඇڝ߹ • -PTTMFTT
   3%." • (16$FOUSJD -PTTZ
   'BCSJD
   'SPOUFOE • ΠϯλʔωοταʔϏεʹ࠷దԽ • $MPT
   5PQPMPHZ • ϕετΤϑΥʔτ • -PTTZ
   5$1 • $16$FOUSJD
5. Backend Network ͳͥඞཁͳͷ͔ʁ ಉ͡ωοτϫʔΫʹࠞࡏͤͯ͞͸͍͚ͳ͍ͷ͔ʁ  "*.- 4FSWFST -FBG -FBG 4QJOF 4QJOF -FBG -FBG 'SPOUFOE 4FSWFST • 5$16%1
   
   ύέοτϩεલఏ • 3%."
   ύέοτϩεϨεલఏ • ࠶ૹ͢Δઃܭ WT
   ࠶ૹͤ͞ͳ͍ઃܭ • 3P$&W͸ઌʹϒϨʔΩΛ͔͚Δٕज़ • 3%."༻ʹઃܭ͞ΕͨωοτϫʔΫͰ͸ 5$1͸ύϑΥʔϚϯεΛൃشͰ͖ͳ͍ 3P$&W͕༗ޮͳωοτϫʔΫͰ͸5$1ͷ᫔᫓੍ޚΞϧΰϦζϜ͕཯଎͞ΕΔͨΊɺ෼཭͢Δ
6. ᫔᫓ʹΑΔόοϑΝҲΕΛ๷͙όοϑΝ؅ཧٕज़ͷू߹ ˠ ͜Ε͕ϩεϨε *ODBTU
   5SBGGJD΍ෆۉߧͳ෼ࢄ΁ͷରॲͳͲΛର৅ ϦϯΫμ΢ϯ΍఻ૹΤϥʔ͸ର৅ʹ͍ͯ͠ͳ͍ #&3ͱ'&$ͷϨΠϠͰΧόʔ ϩεϨεͱ͸Կ͔  Lossless Ethernet (16
   4FSWFS
   /7*%*"
   ) ( 1 6
    ( 1 6
    ( 1 6
    ( 1 6
    ( 1 6
   ̑ ( 1 6
   ̒ ( 1 6
   ̓ ( 1 6
   ̔ -FBG
   4XJUDI 3BJM
    -FBG
   4XJUDI 3BJM
    -FBG
   4XJUDI 3BJM
    -FBG
   4XJUDI 3BJM
    -FBG
   4XJUDI 3BJM
    4QJOF
   4XJUDI 4QJOF
   4XJUDI 4QJOF
   4XJUDI 4QJOF
   4XJUDI (16
   4FSWFS
   /7*%*"
   ) ( 1 6
    ( 1 6
    ( 1 6
    ( 1 6
    ( 1 6
   ̑ ( 1 6
   ̒ ( 1 6
   ̓ ( 1 6
   ̔ ϑϩʔͷภΓ *ODBTU ɾɾɾ *ODBTU ϑϩʔͷภΓ
7. 4XJUDI • ωοτϫʔΫػث /*$΋ؚΉ ͷ֤ϙʔτʹ͸ɺૹ৴ͱड৴ͦΕͧΕʹݸͷԾ૝Ωϡʔ͕͋Δ ͜ΕΛ 5$
   5SBGGJD
   $MBTT
   2VFVFͱݺͿɻ෺ཧతͳϝϞϦྖҬΛڞ༗͍ͯ͠Δɻ ωοτϫʔΫػثͷ2VFVFʹ͍ͭͯ  パケットのプライオリティと分類 &HSFTT
   5$ ༻్  ະ࢖༻  ͦͷଞ  ະ࢖༻  ༏ઌ౓ߴ  ະ࢖༻  ະ࢖༻  ઈର༏ઌ  ະ࢖༻ *OHSFTT
   5$         &HSFTT
   5$ ༻్  ະ࢖༻  ͦͷଞ  ະ࢖༻  ༏ઌ౓ߴ  ະ࢖༻  ະ࢖༻  ઈର༏ઌ  ະ࢖༻ 4FSWFS 4FSWFS *OHSFTT
   5$        
8. • 3P$&WͰ͸ૹ৴ଆͷݸͷΩϡʔʹ3P$&Wઐ༻ͷΩϡʔΛݻఆͰઃఆ͢Δඞཁ͕͋Δ 3P$&Wͷύέοτ͸͸5$
   ͦͷଞͷύέοτ͸5$Λ࢖͏ͳͲ ωοτϫʔΫػثͷ2VFVFʹ͍ͭͯ  パケットのプライオリティと分類 4XJUDI &HSFTT
   5$ ༻్  ະ࢖༻  ͦͷଞ  ະ࢖༻  ༏ઌ౓ߴ  ະ࢖༻  ະ࢖༻  ઈର༏ઌ  ະ࢖༻ *OHSFTT
   5$         &HSFTT
   5$ ༻్  ະ࢖༻  ͦͷଞ  ະ࢖༻  ༏ઌ౓ߴ  ະ࢖༻  ະ࢖༻  ઈର༏ઌ  ະ࢖༻ 4FSWFS 4FSWFS *OHSFTT
   5$        
9. • 3P$&WͰ͸ૹ৴ଆͷ֤Ωϡʔʹద੾ͳϓϥΠΦϦςΟΛઃఆ͢Δ͜ͱͰ੍ޚ͞ΕΔ ड৴ଆͰͲͷΩϡʔΛར༻͢Δ͔͸࣮૷ґଘ ωοτϫʔΫػثͷ2VFVFʹ͍ͭͯ  パケットのプライオリティと分類 &HSFTT
   5$
   2VFVF ༏ઌ੍ޚઃఆ ׂΓ౰ͯର৅  ະ࢖༻  %833
    ͦͷଞ  ະ࢖༻  %833
    3P$&W௨৴  ະ࢖༻  ະ࢖༻  4USJDU $/1  %833ɿ %FGJDJU
   8FJHIUFE
   3PVOE
   3PCJO
10. • ػثͰͦͷύέοτΛͲͷΩϡʔͰॲཧ͢Δͷ͔ΛࣝผͰ͖ͳ͚Ε͹ͳΒͳ͍ ύέοτ͸7-"/
    UBH΋͘͠͸%4$1ͷCJUͰࣝผ͢Δ ͲͷϓϥΠΦϦςΟΛ࢖༻͢Δ͔͸εΠονͷઃఆͰબ୒Մೳ ͜ΕΛ 5SVTU
    .PEF
    ͱݺͿ ύέοτͷ෼ྨ  パケットのプライオリティと分類 %."$ 4."$ 7-"/
    5BH
    2 &UIFS5ZQF *1
    )FBEFS 1BZMPBE %4
    'JFME 51*% 1$1 %&* 7*% CJU CJU CJU CJU &$/ %4$1 CJU CJU 1SJPSJUZ
    $PEF
    1PJOU
    %JGG4FSW $PEF
    1PJOU
    CJU
11. • ػثͰͦͷύέοτΛͲͷΩϡʔͰॲཧ͢Δͷ͔ΛࣝผͰ͖ͳ͚Ε͹ͳΒͳ͍ ύέοτ͸7-"/
    UBH΋͘͠͸%4$1ͷCJUͰࣝผ͢Δ ͲͷϓϥΠΦϦςΟΛ࢖༻͢Δ͔͸εΠονͷઃఆͰબ୒Մೳ ͜ΕΛ 5SVTU
    .PEF
    ͱݺͿ ύέοτͷ෼ྨ  パケットのプライオリティと分類 cumulus@leaf_rail-1:mgmt:~$ nv show interface swp1 qos roce status operational ------------------ ------------- pfc pfc-priority 3 rx-enabled yes tx-enabled yes trust trust-mode pcp,dscp congestion-control congestion-mode ecn, absolute enabled-tc 0,3 min-threshold 153.00 KB max-threshold 1.43 MB mode lossless leaf_rail-4#show qos interfaces ethernet 1/1 Ethernet1/1: Trust Mode: DSCP Default COS: 0 Default DSCP: 0 Port shaping rate: disabled
12. • ड৴ଆͷػثͰͦͷύέοτΛͲͷΩϡʔͰॲཧ͢Δͷ͔ΛࣝผͰ͖ͳ͚Ε͹ͳΒͳ͍ ଟ͘ͷ৔߹ɺ%4$1Ͱࣝผ͢Δ ύέοτͷ෼ྨ  パケットのプライオリティと分類 4XJUDI &HSFTT
    5$
    2VFVF         4FSWFS 3P$&W #(1 *1W
    3" 3P$&W 3P$&W #(1 *1W
    3" 3P$&W %4$1
     %4$1
    /"
13. ઃఆ஋͸೚ҙ͕ͩɺσϑΝΫτελϯμʔυ͕͋Δɻ 3P$&W
    
    %4$1
    
    
    5$
     $/1
    
    %4$1
    
    
    5$
     ͕ҰൠతͳϚοϐϯάߏ੒ɻ/7*%*" .FMMBOPY ͷਪ঑஋ɻ Ϛοϐϯά͸ͦͷػثϩʔΧϧͷಈ࡞ʹӨڹ͢Δ͕ɺ ӡ༻্ͷ؍఺Ͱ͢΂ͯͷػثͰઃఆ஋Λ߹ΘͤΔ͜ͱɻ 3P$&Wͷ%4$1൪߸ʹ͍ͭͯ  パケットのプライオリティと分類 IUUQTFOUFSQSJTF TVQQPSUOWJEJBDPNTBSUJDMFMPTTMFTTSPDF DPOGJHVSBUJPOGPSNMOYPTTXJUDIFTJOETDQCBTFE RPTNPEFBEWBODFENPEFY IUUQTXXXBSJTUBDPNBTTFUTEBUBQEG#SPBEDPN 3P$&%FQMPZNFOU(VJEFQEG
14. パケットのプライオリティと分類 /*$ଆͷઃఆ ࠷ॏཁ  # use L3 PFC, default=pcp (L2 PFC) sudo mlnx_qos -i $IF_NAME --trust dscp # enable PFC on PFC Priority 3 sudo mlnx_qos -i $IF_NAME --pfc 0,0,0,1,0,0,0,0 # clear Traffic Class (TC) settings echo "tclass=-1" | sudo tee /sys/class/infiniband/$DEV_NAME/tc/1/traffic_class # set default ToS (= DSCP value * 4) for RoCE traffic echo 106 | sudo tee /sys/class/infiniband/$DEV_NAME/tc/1/traffic_class # set default ToS for RoCE traffic sudo cma_roce_tos -d $DEV_NAME -t 106 &$/ %4$1 CJUT CJUT  
     5P4 CJUT 
     1'$
    %4$1ઃఆ
15. パケットのプライオリティと分類 /*$ଆͷઃఆ ࠷ॏཁ  [markunet@hgx200]$ sudo mlnx_qos -i enp14s0np0 --trust dscp DCBX mode: OS controlled Priority trust state: dscp dscp2prio mapping: prio:0 dscp:00, default priority: Receive buffer size (bytes): 19872,220896,0,0,0,0,0,0,max_buffer_size=4151520 Cable len: 7 PFC configuration: priority 0 1 2 3 4 5 6 7 enabled 0 0 0 1 0 0 0 0 buffer 0 0 0 1 0 0 0 0 tc: 0 ratelimit: unlimited, tsa: vendor priority: 1 tc: 1 ratelimit: unlimited, tsa: vendor priority: 0 tc: 2 ratelimit: unlimited, tsa: vendor priority: 2 tc: 3 ratelimit: unlimited, tsa: vendor priority: 3 tc: 4 ratelimit: unlimited, tsa: vendor priority: 4 tc: 5 ratelimit: unlimited, tsa: vendor priority: 5 tc: 6 ratelimit: unlimited, tsa: vendor priority: 6 tc: 7 ratelimit: unlimited, tsa: vendor priority: 7 1'$ઃఆ֬ೝ
16. パケットのプライオリティと分類 /*$ଆͷઃఆ ࠷ॏཁ  IUUQTCMPHNZMBCDD&OBCMF-1'$%$2$/GPS3P$&PO.FMMBOPY$POOFDU9/*$T
17. パケットのプライオリティと分類 %4$1
    UP
    5$
    .BQQJOH
    $POGJHVSBUJPO
    εΠονଆ  RoCE PCP/DSCP->SP mapping configurations =========================================== pcp dscp switch-prio - --- ----------------------- ----------- 0 0 0,1,2,3,4,5,6,7 0 1 1 8,9,10,11,12,13,14,15 1 2 2 16,17,18,19,20,21,22,23 2 3 3 24,25,26,27,28,29,30,31 3 4 4 32,33,34,35,36,37,38,39 4 5 5 40,41,42,43,44,45,46,47 5 6 6 48,49,50,51,52,53,54,55 6 7 7 56,57,58,59,60,61,62,63 7 qos map DSCP 0 1 2 3 4 5 6 7 to traffic-class 0 qos map DSCP 8 9 10 11 12 13 14 15 to traffic-class 1 qos map DSCP 16 17 18 19 20 21 22 23 to traffic-class 2 qos map DSCP 24 25 26 27 28 29 30 31 to traffic-class 3 qos map DSCP 32 33 34 35 36 37 38 39 to traffic-class 4 qos map DSCP 40 41 42 43 44 45 46 47 to traffic-class 5 qos map DSCP 48 49 50 51 52 53 54 55 to traffic-class 6 qos map DSCP 56 57 58 59 60 61 62 63 to traffic-class 7 Dscp-tc map: d1 : d2 0 1 2 3 4 5 6 7 8 9 -------------------------------------- 0 : 0 0 0 0 0 0 0 0 1 1 1 : 1 1 1 1 1 1 2 2 2 2 2 : 2 2 2 2 3 3 3 3 3 3 3 : 3 3 4 4 4 4 4 4 4 4 4 : 5 5 5 5 5 5 5 5 6 6 5 : 6 6 6 6 6 6 7 7 7 7 6 : 7 7 7 7 /7*%*"
    $VNVMVT
    σϑΥϧτ஋ ઃఆෆཁ "SJTUBΛ/7*%*"ͱಉ༷ͷϚοϐϯάͳΔΑ͏ʹઃఆͨ͠ྫ
18. パケットのプライオリティと分類 1SJPSJUZ
    $POGJHVSBUJPO  &HSFTT
    5SBGGJD
    $MBTT
    5$ %4$1 ༏ઌ੍ޚઃఆ ׂΓ౰ͯର৅ &$/1'$   %833
     ͦͷଞͷ௨৴    %833
     3P$&Wͷ௨৴ ༗ޮ     4USJDU $/1  • %833ͱͦͷઃఆ஋͸͋͘·Ͱྫɺػثͷ࢓༷ʹ߹Θͤͯνϡʔχϯά͢Δඞཁ͕͋Δ • ػثʹΑͬͯ͸833ͳͲҟͳΔΞϧΰϦζϜΛαϙʔτ͢Δ৔߹΋͋Δ • ۭཝ͸ະ࢖༻ʢະઃఆʣͷΩϡʔ
19. パケットのプライオリティと分類 ઃఆྫ "SJTUB
    &04  leaf_rail-4#show qos interfaces ethernet 1/1 Ethernet1/1: Trust Mode: DSCP Default COS: 0 Default DSCP: 0 Port shaping rate: disabled Tx Bandwidth Bandwidth Shape Rate Priority ECN/WRED Queue Guaranteed (units) (units) ------------------------------------------------------------------------------------------ 7 - / - - / - ( - ) - / - ( - ) SP / SP D 6 - / - - / - ( - ) - / - ( - ) SP / SP D 5 - / - - / - ( - ) - / - ( - ) SP / SP D 4 - / - - / - ( - ) - / - ( - ) SP / SP D 3 95% / 95% - / - ( - ) - / - ( - ) RR / RR L 2 - / - - / - ( - ) - / - ( - ) RR / SP D 1 5% / 5% - / - ( - ) - / - ( - ) RR / RR D 0 - / - - / - ( - ) - / - ( - ) RR / SP D Note: Values are displayed as Operational/Configured Legend: RR -> Round Robin SP -> Strict Priority - -> Not Applicable / Not Configured % -> Percentage of reference ECN/WRED: L -> Queue Length ECN Enabled W -> WRED Enabled D -> Disabled
20. 4XJUDI • ͋ͱ͸ׂΓ౰ͯͨΩϡʔͷόοϑΝ͕ҲΕͳ͍Α͏ʹ੍ޚ͢Δ ˠ -PTTMFTT
    &UIFSOFU
    ʹඞཁͳٕज़ ύέοτͷ෼ྨ  パケットのプライオリティと分類 &HSFTT
    5$ ༻్  ະ࢖༻  ͦͷଞ  ະ࢖༻  ༏ઌ౓ߴ  ະ࢖༻  ະ࢖༻  ઈର༏ઌ  ະ࢖༻ *OHSFTT
    5$         &HSFTT
    5$ ༻్  ະ࢖༻  ͦͷଞ  ະ࢖༻  ༏ઌ౓ߴ  ະ࢖༻  ະ࢖༻  ઈର༏ઌ  ະ࢖༻ 4FSWFS 4FSWFS *OHSFTT
    5$        
21. -PTTMFTT
    &UIFSOFU
    Λߏ੒͢Δओͳٕज़ ᶃ 1'$
    1SJPSJUZ
    'MPX
    $POUSPM 
    *&&&2CC ᶄ &$/
    &YQMJDJU
    $POHFTUJPO
    /PUJGJDBUJPO 
    *&&&2BV ᶅ $/1
    $POHFTUJPO
    /PUJGJDBUJPO
    1BDLFU 
    *#5"ن֨ 3P$&Wͷύέοτʹ͜ΕΒͷٕज़Λద༻Ͱ͖ΔΑ͏ʹɺύέοτΛࣝผ͢Δඞཁ͕͋Δ ϩεϨεͷ࣮ݱํ๏  Lossless Ethernet
22. Lossless Ethernet ϩεϨεͷ࣮ݱํ๏  &UIFSOFU *16%1 *OGJOJCBOE #5)
    - /*$ -
    4XJUDI &$/1'$
    FOBCMFE -
    4XJUDI &$/1'$
    FOBCMFE &UIFSOFU *16%1 *OGJOJCBOE #5)
    - /*$ $/1 1'$ 1"64& &$/ 1'$ 1"64& %$2$/ %$2$/ 1'$ 1"64& 1'$ɿϦϯΫϨϕϧϑϩʔ੍ޚ )PQCZ)PQ &$/ɿ&OE
    UP
    &OE
    ϑϩʔ੍ޚ
23. • ᫔᫓͕ൃੜͨ͠5SBGGJD
    $MBTT d ͷΩϡʔ͔Β1"64&ϑϨʔϜΛྡ઀ϊʔυʹૹ৴ • 1"64&ϑϨʔϜΛड৴ͨ͠ϊʔυ͸ͦͷ5$ʹରԠ͢ΔΩϡʔ͔Βͷύέοτͷૹ৴ΛҰ࣌ఀࢭ • ૹ৴ఀࢭͱ࠶։ґཔ 9PGG PGGPO ΛΩϡʔ͝ͱʹ੍ޚ 1SJPSJUZCBTFE
    'MPX
    $POUSPM  PFC IUUQTXXXKBOPHHSKQNFFUJOHKBOPHXQDPOUFOUVQMPBET+"/0(@%$204@QEG
24. Headroom Buffer 1'$ͷ࠷ॏཁ࣮૷  1'$ൃಈ࣌ʹɺ఻ૹ࿏্ͷύέοτΛड৴͢ΔͨΊͷόοϑΝͷ҆શྖҬ IUUQTXXXJFFFPSHGJMFTQVCMJDEPDTOFXMWBEBQUJWFQGDIFBESPPNWQEG
25. Headroom Buffer έʔϒϧ௕͔ΒͷඞཁͳόοϑΝྖҬΛࢉग़  IUUQTEPDTOWJEJBDPNOFUXPSLJOHFUIFSOFUTPGUXBSFDVNVMVTMJOVY-BZFS BOE4XJUDI1PSUT2VBMJUZPG4FSWJDFMJOLQBVTF IUUQTXXXKVOJQFSOFUEPDVNFOUBUJPOVTFOTPGUXBSFKVOPTDMJ SFGFSFODFUPQJDTSFGTUBUFNFOUDBCMFMFOHUIFEJUDMBTTPGTFSWJDFDPOHFTUJPO OPUJGJDBUJPORGYTFSJFTIUNM ϋʔυ΢ΣΞࣄʹ࢓༷͕શ͘ҟͳΔɻ جຊతʹσϑΥϧτ஋ͷ··Ͱ໰୊ͳ͍͕ɺߴ౓ͳνϡʔχϯάΛ͢Δ৔߹͸ ӡ༻ऀ͕ܭࢉͯ͠ઃఆ͢Δɻ
26. Headroom Buffer ࣮ࡍͷΫϥελͰͷݸผઃఆྫ  έʔϒϧ௕͕ҟͳΔઃܭʹ͢Δͱܭࢉ͕໘౗ͳͷͰɺ&P3ͷϥοΫσβΠϯ͕औΕͳ͘ͳΔ ͜ͷܭࢉͷͨΊʹࠨӈରশͷϥοΫσβΠϯʹͳ͍ͬͯΔ ඇެ։
27. 1'$͸Ωϡʔ͝ͱͷ੍ޚͱ͸͍͑ɺύέοτͷૹ৴ΛࢭΊͯ͠·͏ ࢭΊΔͷͰ͸ͳ͘ɺૹ৴ϨʔτΛ཈੍ͯ͠΋Β͏᫔᫓੍ޚ͕๬·͍͠ όοϑΝ͕ຒ·Δલʹ᫔᫓Λݕ஌ɺૹ৴ݩʹ᫔᫓Λ௨஌͠᫔᫓੍ޚ 1'$ͷܧଓ͸ੑೳ௿ԼΛট͘  ECN / CNP 4XJUDI 4FSWFS 4FOEFS 4FSWFS 3FDFJWFS $POHFTUJPO $POHFTUJPO
    .BSLJOH $POHFTUFE
    5SBGGJD $POHFTUJPO
    /PUJGJDBUJPO )JHI
    1SJPSJUZ
    .BSLJOH
28. छྨͷ᫔᫓௨஌ύέοτΛར༻͢Δ • &$/
    &YQMJDJU
    $POHFTUJPO
    /PUJGJDBUJPO • $/1
    $POHFTUJPO
    /PUJGJDBUJPO
    1BDLFU ಈ࡞ϝΧχζϜ  ECN / CNP • C
    /PU&$5
    /PU
    &$/$BQBCMF
    5SBOTQPSU
    • C
    &$5
    &$/$BQBCMF
    5SBOTQPSU
    • C
    &$5
    &$/$BQBCMF
    5SBOTQPSU
    • C
    $&
    $POHFTUJPO
    &YQFSJFODFE
    4XJUDI 4FSWFS 4FOEFS 4FSWFS 3FDFJWFS $POHFTUJPO $POHFTUJPO
    .BSLJOH $POHFTUFE
    5SBGGJD $POHFTUJPO
    /PUJGJDBUJPO )JHI
    1SJPSJUZ
    .BSLJOH *# 6%1 *1
    )FBEFS &$/
    C &UIFSOFU *# 6%1 *1
    )FBEFS &$/
    C &UIFSOFU $/1 &$/
    C ᶃ &$5 CJUΛηοτ ᶄ $& CJUΛมߋ ᶅ $&Λड৴ͨ͠Β$/1Λૹ৴ ᶆ $/1ͷड৴Ͱૹ৴Ϩʔτௐ੔ 0Q  *# 6%1 *1
    )FBEFS &$/
    C &UIFSOFU
29. ECN όοϑΝͷ࢖༻ঢ়گʹԠͯ͡ɺύέοτʹϥϯμϜͰϚʔΩϯάʢ&$/ $& CJUʣΛ࣮ࢪ    .JOJNVN
    5ISFTIPME .BYJNVN
    5ISFTIPME 83&%ʹجͮ͘ϚʔΩϯά ͜ͷ۠ؒ͸ϚʔΩϯά͞Εͨύέοτͱ ͞Ε͍ͯͳ͍ύέοτ͕ࠞࡏ͢Δ ͜ΕҎ߱͸ ϚʔΩϯά 83&% 8FJHIUFE
    3BOEPN
    &BSMZ
    %FUFDUJPO &$/͸ઐ༻ύέοτͰ͸ͳ͘ɺ 3P$&WͷσʔλύέοτʹϚʔΩϯά͢Δ #VGGFS
    6TBHF &$/
    .BSLJOH
    1SPCBCJMJUZ
30. ECN ֬཰తͳϚʔΩϯάͷ༨༟Λ࣋ͨͳ͍έʔε    .JOJNVN
    5ISFTIPME .BYJNVN
    5ISFTIPME ࠷খᮢ஋ͱ࠷େᮢ஋ΛಉҰʹ͢Δ͜ͱͰ શύέοτʹϚʔΩϯά͢Δઃఆ ֬཰తͳϚʔΩϯάΛ࣮ࢪ͠ͳ͍ύλʔϯ /7*%*"ͷσϑΥϧτ஋͸ͪ͜Β )1$ͳͲύϑΥʔϚϯεʹහײͳγεςϜ޲͚ #VGGFS
    6TBHF &$/
    .BSLJOH
    1SPCBCJMJUZ ͜ΕҎ߱͸ ϚʔΩϯά
31. CNP $/1
    1BDLFU
    'PSNBU  #5)
    #BTF
    5SBOTQPSU
    )FBEFS
    *OGJOJCBOE
    0QDPEF
    Y
    C`
    
    $/1͸௨஌ઐ༻ͷ"DLύέοτͷͨΊɺ 3P$&WͷϖΠϩʔυ͸࣋ͨͳ͍ 0Q 
32. &$/͸*1ϔομͷϑΟʔϧυ͕ͩɺ$/1͸*#
    #5ϔομͷϑΟʔϧυͰ͋Δ͜ͱʹ஫ҙ͢Δ 3P$&W
    1BDLFU
    'PSNBU  ECN / CNP &$/͸*1ϔομͷٕज़ - &$/ %4$1 CJUT CJUT $/1
    0QDPEF CJUT $/1͸*#ϔομͷٕज़ -
33. DCQCN ᫔᫓੍ޚΞϧΰϦζϜ  IUUQTFOUFSQSJTFTVQQPSUOWJEJBDPNTBSUJDMF%$2$/$$BMHPSJUIN *&&&
    2BV
    2VBOUJ[FE
    $POHFTUJPO
    /PUJGJDBUJPO
    2$/
    ͷεΩʔϜʹج͖ͮૹ৴ϨʔτΛ੍ޚ͢Δɻ 5$1ͷΑ͏ͳ΢Οϯυ΢αΠζ੍ޚͱ͸ҟͳΔɻ # Check DCQCN is enabled on Prio 3 cat /sys/class/net/$IF_NAME/ecn/roce_np/enable/3 cat /sys/class/net/$IF_NAME/ecn/roce_rp/enable/3 # Check counters related to DCQCN cat /sys/class/infiniband/$DEV_NAME/ports/1/hw_counters/np_cnp_sent cat /sys/class/infiniband/$DEV_NAME/ports/1/hw_counters/np_ecn_marked_roce_packets cat /sys/class/infiniband/$DEV_NAME/ports/1/hw_counters/rp_cnp_handled
34. DCQCN ҰׅઃఆεΫϦϓτ  IUUQTHJUIVCDPN/7*%*"EPSPDFMJOVY
35. ᫔᫓͕ղফ͞ΕΔ·Ͱɺ&$/ͷϚʔΩϯάͱ$/1ͷϨεϙϯε͸ܧଓ͢Δ ૹ৴ଆͷ᫔᫓੍ޚΞϧΰϦζϜ %$2$/ ͕Ϩʔτௐ੔Λߦ͏ εΠονͰϦϯΫμ΢ϯ͍ͯ͠Δؒɺૹ৴ϊʔυ͕ϨʔτΛௐ੔͍ͯ͠Δɺ͜ͷؒύέϩεͳ͠ ࣮ࡍͷಈ࡞ྫ  ECN / CNP ඇެ։
36. PFC / ECN / CNP ϋʔυ΢ΣΞΧ΢ϯλ  markunet@leaf_rail-1:mgmt:~$ ethtool -S swp3 | egrep "Q3|Q6|Ecn|Pfc3" HwIfInPfc3Pkt: 31848 HwIfOutPfc3Pkt: 0 HwIfOutQ3WredDrops: 0 HwIfOutQ6WredDrops: 0 HwIfOutQ3BuffDiscards: 0 HwIfOutQ6BuffDiscards: 0 HwIfOutQ3Pkts: 318357735060 HwIfOutQ3Octets: 326513898313396 HwIfOutEcnMarkedPkts: 11375061 HwIfOutQ6Pkts: 8245964 HwIfOutQ6Octets: 643185192 HwIfInPfc3Duration: 635412 HwIfOutPfc3Duration: 0 HwIfInQ3Pkts: 0 HwIfInQ6Pkts: 0 HwIfInQ3BuffDiscards: 0 HwIfInQ6BuffDiscards: 0 HwIfInQ3SharedBuffDiscards: 0 HwIfInQ6SharedBuffDiscards: 0 3P$&WͷύέοτΧ΢ϯλ &$/ͷϚʔΩϯάΛͨ͠ύέοτΧ΢ϯλ $/1ͷύέοτΧ΢ϯλ 1'$ͷύέοτΧ΢ϯλ
37. PFC / ECN / CNP ϋʔυ΢ΣΞΧ΢ϯλ  leaf_rail-4#show interfaces ethernet 1/1 counters queue detail Port TxQ Counter/pkts Counter/bytes Drop/pkts Drop/bytes ------- ---- ------------ ------------ ------------ ------------ Et1/1 UC0 5 638 0 0 Et1/1 UC1 0 0 0 0 Et1/1 UC2 0 0 0 0 Et1/1 UC3 7242261179 477989237814 0 0 Et1/1 UC4 0 0 0 0 Et1/1 UC5 0 0 0 0 Et1/1 UC6 128963 12930230 0 0 Et1/1 UC7 0 0 0 0 Et1/1 UC8 1976 539448 0 0 leaf_rail-4#show qos interfaces ethernet 1/1 ecn counters queue Ethernet1/1: Tx-Queue Marked Packets ---------- ----------------------- 0 - 1 - 2 - 3 0 4 - 5 - 6 - 7 - ద੾ͳΩϡʔΛར༻͍ͯ͠Δͷ͔ͷ֬ೝ͸ඞਢ 
    ͳͲ͸γεςϜͷಛघ༻్ͷ৔߹͋Γ
38. RoCEv2 設定例 Arista EOS ϓϥΠΦϦςΟͷϚοϐϯάͱ..6ϓϩϑΝΠϧͷઃఆ  qos map DSCP 0 1 2 3 4 5 6 7 to traffic-class 0 qos map DSCP 8 9 10 11 12 13 14 15 to traffic-class 1 qos map DSCP 16 17 18 19 20 21 22 23 to traffic-class 2 qos map DSCP 24 25 26 27 28 29 30 31 to traffic-class 3 qos map DSCP 32 33 34 35 36 37 38 39 to traffic-class 4 qos map DSCP 40 41 42 43 44 45 46 47 to traffic-class 5 qos map DSCP 48 49 50 51 52 53 54 55 to traffic-class 6 qos map DSCP 56 57 58 59 60 61 62 63 to traffic-class 7 platform trident mmu queue profile RoCE_MMU_Profile ingress threshold 1/16 egress unicast queue 3 threshold 8 ! ..6
    
    .FNPSZ
    .BOBHFNFOU
    6OJU όοϑΝ༧໿Λߏ੒͢ΔϓϩϑΝΠϧͷ͜ͱ Ωϡʔͷᮢ஋Λཁ݅ʹ߹Θͤͯมߋ͢Δ ..6Λมߋ͠ͳ͍ͱ1'$͕ܧଓಈ࡞͠ύϑΥʔϚϯε͕௿Լ͠·͢ 4USBUB9(4 $IJQͰͷઃఆྫɻ%/9Ͱ͸ҟͳΔͷͰ஫ҙɻ
39. RoCEv2 設定例 Arista EOS 3P$&WϓϩϑΝΠϧͷ࡞੒ͱ*OUFSGBDF΁ͷΞλον  qos profile RoCEv2 priority-flow-control on priority-flow-control priority 3 no-drop ! tx-queue 1 no priority bandwidth percent 5 ! tx-queue 3 no priority bandwidth percent 95 random-detect ecn minimum-threshold 512 kbytes maximum-threshold 768 kbytes max-mark-probability 100 ! interface Ethernet1/1 description DOWNLINK mtu 9216 speed forced 100gfull no switchport ipv6 enable service-profile RoCEv2 ! tx-queue 3 random-detect ecn count platform trident mmu queue interface-profile RoCE_MMU_Profile ! &$/ͷᮢ஋͸σϑΥϧτ஋Ͱݕূͯ͠ɺඞཁͳΒมߋ͢Δ 1'$Λ5$
    Ͱ༗ޮԽ͢Δઃఆ OPESPQ
    
    -PTTMFTT &$/ͷઃఆ
40. Load Balancing &$.1Ͱͷ෼ࢄ໰୊  IUUQTXXXKBOPHHSKQNFFUJOHKBOPHXQ DPOUFOUVQMPBETKBOPHMULPCBZBTIJQEG ৄ͘͠͸ͪ͜ΒͷࢿྉͰ֬ೝ ! ip hardware fib load-balance distribution dynamic ip hardware fib load-balance distribution dynamic member-selection optimal always !
41. RoCEv2 設定例 Scheduled Fabric 7P2ϕʔεͷઃఆ  ඇެ։
42. RoCEv2 設定例 SONiC 1VSF
    40/J$ ͰҰ෦ػೳʹະରԠ  ඇެ։
43. RoCEv2 設定例 Juniper Junos ࢀߟ৘ใ  ඇެ։
44. RoCEv2 設定例 Cisco Nexus ࢀߟ৘ใ  ඇެ։
45. Lossless Ethernet ্ҐϨΠϠͷ᫔᫓੍ޚ͔Βॱʹಈ࡞͢Δ͜ͱ͕ద੾ɺ1'$͸ॠؒతͳ࠷ऴखஈ  /FUXPSL
    #FTU
    1SBDUJDFT
    GPS
    "SUJGJDJBM
    *OUFMMJHFODF
    %BUB
    $FOUSF
    /FNBOKB
    ,BNFOJDB
    5FDIOJDBM
    .BSLFUJOH
    &OHJOFFS
    $JTDP
    -JWF
    #3,%$/ ΑΓը૾Ҿ༻
46. • ͦͷωοτϫʔΫͰಈ࡞͢ΔϫʔΫϩʔυ͕Կ͔ཧղ͢Δ ֶश
    ਪ࿦
    ετϨʔδ
    --.େن໛෼ࢄֶश FUD • ϋʔυ΢ΣΞ࢓༷͕େ͖͘ҟͳΔ੡඼Λࠞࡏͤ͞ͳ͍ 4XJUDI
    /*$ όοϑΝͷ༰ྔ΍ઃఆՄೳͳ஋͕ҟͳΔ • ࣮૷͕େ͖͘ҟͳΔιϑτ΢ΣΞ΍ઃఆΛࠞࡏͤ͞ͳ͍ /04 ύέοτͷ෼ࢄํࣜͳͲ͕ҟͳΔͱύϑΥʔϚϯεʹӨڹ͢Δ • ωοτϫʔΫΛ&OEUP&OEͰӡ༻͢Δ εΠον͚ͩͰͳ͘(16αʔό΋ωοτϫʔΫΤϯδχΞͷ୲౰ྖҬ αʔό಺෦τϙϩδΛ·ͣཧղ͢Δ جຊతͳߟྀࣄ߲  RoCEv2の運⽤
47. EoF