Research Paper Introduction #43 “Extending ECMP toward A Practical Hardware Load Balancer” ௨ࢉ#112 @cafenero_777 2022/01/19 1

Agenda •ର৅࿦จ •֓ཁͱಡ΋͏ͱͨ͠ཧ༝ 1. ͸͡Ίʹ 2. ؔ࿈ݚڀ 3. ECMP with Explicit Retransmission 4. ධՁ 5. ݁࿦ͱࠓޙͷ՝୊ 2

ର৅࿦จ •Extending ECMP toward A Practical Hardware Load Balancer • ECMP ͷ֦ுʹΑΔϋʔ ド ΢ΣΞϩʔ ド バ ϥϯαͷఏҊ • Ryo Nakamura1 Kentaro Ebisawa2 Tomoko Okuzawa2 Chunghan Lee2 Yuji Sekiya1 • 1: Information Technology Center, The University of Tokyo • 2: Toyota Motor Corporation • IOTS 2022 • http://id.nii.ac.jp/1001/00222634/ • https://github.com/ToyotaInfoTech/ecmper 3

֓ཁͱಡ΋͏ͱͨ͠ཧ༝ •֓ཁ • ECMPͰLB͢ΔͱΨϥΨϥϙϯͷՄೳੑ • TCP reset͕hookͯ͠ਖ਼͍͠ѼઌʹૹΓ௚͢LBํࣜ (ECMP-ER) • Data planeͷΈͰ࢓૊Έ͕׬݁Մೳɻ৭ʑධՁͯ͠Έͨɻ •ಡ΋͏ͱͨ͠ཧ༝ • ஶऀຊਓ͔Βڭ͑ͯ΋ΒͬͨͷͰɻ • P4࢖ͬͯΔLBͳͷͰɻ 4

1. ͸͡Ίʹ (1/2) •LBॏཁ •ECMPΛͦͷ··࢖͑Δʁ • μϝ: Per-Connection Consistency͕ҡ࣋Ͱ͖ͳ͍ • ʢͳͷͰҰൠతʹ͸LBػثΛ࢖͏ʣ •ECMP-ER: ECMP with Explicit Retransmission • ϧʔλͱαʔό͚ͩͰLBͰ͖Δ 5

ؔ࿈ݚڀ (2/2) •Stateful • Stateอ͕࣋ʢಛʹHWͩͱʣେมɻಉҰC-hashར༻ͷ޻෉ • C-planeඞཁ •Stateless • stateΛpacket/socketʹຒΊࠐΜͰసૹ (Daisy Chaining)͢Δ • ͜͜Ͱ͸”ηογϣϯಉظΛ͠ͳ͍”ͱ͍͏ҙຯͰ͸ͳ͍ •LBࣗମͷ੍ޚ΋ඞཁ 6 Ref: https://github.com/ToyotaInfoTech/ecmper/blob/main/ecmper-english-IOTS2022.pdf

3. ECMP with Explicit Retransmission •LBύέοτϑϩʔ • جຊ͸5tuple ECMPͰLB͢Δ • αʔόμ΢ϯݕ஌ͰNHςʔϒϧߋ৽ • ߋ৽ޙͷNHςʔϒϧͩͱҧ͏αʔόʹసૹ͞ΕɺTCP reset ͕ฦ͞Εͯ͠·͏ • ͜ΕΛτϦΨʔͱͯ͠ɺαʔόଆͰToSΛཱͯͯϧʔλʹసૹʢ࠶ૹʣ͢Δ • ࠶ૹύέοτͷΈɺߋ৽લͷNHςʔϒϧΛࢀর͢Δ 7

3. ECMP with Explicit Retransmission ࣮૷ฤ •ϧʔλଆ: P4 (WEDGE100BF-32X) • ܦ࿏ຖʹաڈͷNHΛอ࣋ • ࠶ૹ࣌͸աڈͷNHΛࢀরɾసૹ͢Δ •αʔόଆ: XDP/eBPF • resetͷ৔߹͸ToSϏοτΛཱͯͯ౤͛ฦ͢ 8 Ref: https://github.com/ToyotaInfoTech/ecmper/blob/main/ecmper-english-IOTS2022.pdf

4. ධՁ 9 αʔό૿ݮ࣌ͷτϥϑΟοΫ -> ෼ࢄΛ֬ೝ Churn test: ϥϯμϜʹαʔόΛ֎͢ -> consistent-hashͷޮՌΛ֬ೝ -> ECMP-ER@1sͷ৔߹͸ೋੈ୅Ҏ্ඞཁ •SR-IOV (VF) + netnsͰαʔό෼཭ •qdescͰ1Gbpsʹߜͬͨ

5. ݁࿦ͱࠓޙͷ՝୊ •PCC໰୊Λղܾͨ͠ECMP: ECMP-ER • C-plane͕ෆඞཁͳLBϝΧχζϜ • εΠονଆɺαʔόଆͰD-planeͷΈࣄલ४උ •ࠓޙͷ՝୊ • TCPҎ֎ͷద༻Մೳੑɻେن໛DC΁ͷ໛ࡧɻৄࡉͳධՁ࣮ݧɻ 10

׬૸ͨ͠ײ૝ •FastlyͷfaildͬΆ͍ͳ •UDP͸࢖͑ͳ͍ • FWΈ͍ͨͳԾ૝ηογϣϯ࢖͏ʁͦΕ͸ͪΐͬͱͳ͋ɺɺ •LBʹ௚Ͱresetฦ͞ͳ͍ͱμϝ • Direct Server Return͕ݫ͍͠ʢLB௨Βͳ͍ͷͰʣ • Reset͚ͩLBʹసૹ͢ΔΑ͏ʹαʔόଆͷhookΛ͢ΔΑ͏ʹมߋ͢Δʁ • ToRͳΒτϥϑΟοΫύε͕ࣗવͳܗͰͰ͖ͦ͏͕ͩɺٯʹͦΕ੍͕໿ʹͳΓͦ͏ • τϯωϧ௨৴ (SRv6ͱ͔IPIPτϯωϧͱ͔)ͩͱ΋͏ͻͱखؒʢͦΕҎ্ʁʣ͔͔Γͦ͏ •upload͕໨తͷLBͳͷͰDSR͸ඞཁͳ͍ɻͦ΋ͦ΋ݱঢ়Ͱ͸DSRͰ͖ͳ͍ɻ •αʔό͕ϨεϙϯεΛฦ͞ͳ͍৔߹͸Ͳ͏͢Δʁ Health Check agent͕ඞཁʁ΋͘͠͸hello packet time࢖͏ʁ • ͦ͏͢ΔͱC-plane͕ཁΒͳ͍ɺͱ͍͏໨త͔Βԕ͔͟ΔΑ͏ͳɻɻ •Ұ୆͔ͣͭ͠αʔϏεΞ΢τͰ͖ͳ͍ʢաڈ̍ੈ୅෼͔͠ͳ͍ͨΊʣ • ଟॏো֐͸ڐ༰ͯ͠΋ɺϩʔϦϯάΞοϓσʔτͰsercie in/out͠·͘Δcaseͩͱݫ͍͠ 11

EoP 12