Research Paper Introduction #22 VTrace: Automatic Diagnostic System for Persistent Packet Loss in Cloud-Scale Overlay Network

https://dl.acm.org/doi/10.1145/3387514.3405851

Transcript

1. Research Paper Introduction #22 “VTrace: Automatic Diagnostic System for Persistent Packet Loss in Cloud-Scale Overlay Network” ௨ࢉ#72 @cafenero_777 2021/05/27 1
2. Agenda • ର৅࿦จ • ֓ཁͱಡ΋͏ͱͨ͠ཧ༝ 1. INTRODUCTION 2. BACKGROUND 3. DESIGN AND IMPLEMENTATION 4. EVALUATION 5. DEPLOYMENT EXPERIENCES 6. DISCUSSIONS 7. RELATED WORK 8. CONCLUSION 2
3. ର৅࿦จ • VTrace: Automatic Diagnostic System for Persistent Packet Loss in Cloud- Scale Overlay Network • Chongrong Fang– Haoyu Liu–Mao Miao— Jie Ye— Lei Wang— 
 Wansheng Zhang— Daxiang Kang— Biao Lyv— Peng Cheng– Jiming Chen– • –State Key Laboratory of Industrial Control Technology, Zhejiang University • —Alibaba Group • SIGCOMM 2020 3
4. ֓ཁͱಡ΋͏ͱͨ͠ཧ༝ • ֓ཁ • Ϋϥ΢υεέʔϧͷΦʔόʔϨΠNWͷύέϩε؂ࢹ • VFD (vSwitchͳͲ)ʹ࢓ࠐΜͰɺసૹঢ়گΛ೺Ѳ • 20ϲ݄ͷӡ༻ܦݧΛڞ༗ • ಡ΋͏ͱͨ͠ཧ༝ͱײ૝ • ڊେClos؂ࢹʹ໾ʹཱͪͦ͏: pingmesh, Net-bouncerͷ࣍ʁ • Ծ૝NWʹϑΥʔΧε -> ྑ͍ҙຯͰཪ੾ΒΕͨ • VPCߏ੒ͷ؂ࢹͷ࢓ํ -> ͪΌΜͱͨ͠ઃܭɾγεςϜԽ͕ඞਢ 4
5. 1. INTRODUCTION • ҰൠతͳΫϥ΢υNWͱͦͷ՝୊ • ෺ཧNWΛந৅Խͯ͠ςφϯτNWͱͯ͠ʢΦʔόʔϨΠͱͯ͠ʣఏڙ • ෺ཧ/Ծ૝SW (VFD: vSwitch/vRouter͸ߋ৽සൟʂ)ͷ͔ᷮͳΤϥʔ͕඼࣭Өڹɺtcpdump͡Όʢ෺ཧNWҎ্ʹʣແཧʂ • VFD਺͕ലେɾ஍ཧ෼ࢄɺઃఆɾτϙϩδͷมߋɾSLBʹΑΔϑϩʔ೉ಡԽ • ݪҼͷಛఆͷϞσϧԽʢϢʔβͷޡૢ࡞ؚΉʣ • VTrace • ϑϩʔͷ fi rst hop VFDͰύέοτΛϚʔΫɻ֤hopͰmatch/logging͢Δɻ͜ΕΛूΊͯϑϩʔΛ࠶ߏங • ੑೳධՁ͠ɺ2018೥݄̑ҠߦʹAlibaba Cloudʹల։͠ɺௐࠪ࣌ؒͷେ෯࡟ݮ • ྙཧ໰୊͸Ұ୴আ֎ ʢѱ༻͢ΔͱτϥϑΟοΫ؂ࢹͰ͖ΔͨΊʁʣ 5
6. 2. BACKGROUND • Alibaba Cloud • VXLANͰΦʔόʔϨΠ • ECS (VM), SLB, VFD (vSW/vRT, ͍͕ͭ͜VXLAN͢Δ) • VPC಺௨৴ (vSW <-> vSW) • Internetͱͷ௨৴ (vSW <-> vRT) • Ϣʔβ໢ͱͷ௨৴ (vSW <-> vRT <-> vBR <-> Ϣʔβ໢) • VPCؒ௨৴ (vSW <-> vRT … vRT <-> vSW) • ύέϩε • ҰաੑɾܧଓੑɺVFDͷઃఆϛε΍ιϑτ΢ΣΞόά • Ϣʔβૢ࡞ʹΑΔτϙϩδมߋӨڹ • ଳҬෆ଍ɾACLϛε͸Πϯϑϥো֐Ͱ͸ͳ͍͕ސ٬Өڹ͸͋Δɻਖ਼ৗӡ༻͕ͩɻɻ • ૣظʹ໰୊ݕ஌͍ͨ͠ɿ͖͍͠஋ͰΞϥʔτൃಈʢଳҬͷٸܹͳ௿Լ΍99%ile latencyʣ • VM ID, NIC_ID, 5tupleΛ࢖ͬͯϑϩʔߜΓࠐΈɺVTraceͰ਍அ 6
7. 3. DESIGN AND IMPLEMENTATION (Overview) • ׬શͳpacket traceͱݪҼVFDՕॴಛఆ͕ඞਢཁ݅ • ΦʔόʔϨΠNW্Ͱͷϑϩʔநग़ • ςφϯτ௨৴Өڹͳ͠ɺ͔ͭɺςφϯτଆ࣮૷ͳ͠ • Ϋϥ΢υεέʔϧͳઃܭʢଟ਺ͷϑϩʔɾλεΫɺԠ౴ੑʣ • VTrace֓ཁɿ2ஈ֊Ͱ΍Δ • VTrace Data Generation (VDG) • ֤VFD΁ઃఆɾλʔήοτͷύέοτॲཧ • ϩάग़ྗɾΤʔδΣϯτసૹ • Data Collection and Analysis (DCA) • JStorm (Streamingॲཧ)ͰԾ૝ϑϩʔύεͷߏஙɾ෼ੳ • task؅ཧɺ݁Ռ؅ཧɺAPI࿈ܞ 7
8. 3. DESIGN AND IMPLEMENTATION VDG: VTrace Data Generation • ଞͷํ๏ɾྨࣅྫ • SDNతʹίϯτϩʔϥ΁ϝοηʔδૹΔɺΦʔόʔϔουɺin/out໰୊ • RSPANతΞϓϩʔνɺϛϥʔ(dump)͚ͩݟͯ΋ݪҼ͕෼͔Βͳ͍ • ݪҼϊʔυಛఆ͕ओ؟ɺݪҼಛఆ·ͰࣗಈʢʁʣͰ΍ͬͯ͘Εͳ͍ • ઃܭ • fast path-slow path • ·ͣ͸slow pathॲཧɺͦͷޙ͸match/action rule update (fast path)ɺcoloring/matching/logging • coloring: ͲͷVM௨৴ΛԿύέοτऔಘ͢Δ͔ΛvFWʹઃఆɻϨʔτ্ݶ෇͖ɻ400 probes. • ࣮૷ 1. coloring: DSCP஋ΛϚʔΫɺ࣍ͷVFWͰ΋࢖͏ɻϢχʔΫͳPacket_ID෇༩ɻର৅ͷશvFDʹdeploy 2. match/action͢Δɻର৅FDͷΈઃఆ͞ΕΔɻ਺~਺ेϧʔϧఔ౓ɻ 3. ֤MetricsऔಘɺNATରԠɺdscp_mark, ϩάసૹ 8
9. 3. DESIGN AND IMPLEMENTATION DCA: Data Collection and Analysis • ઃܭɾ࣮૷ • JStorm (Apache Storm based) ετϦʔϜϓϩηεΤϯδϯ • task/log spouts -> ֤bolts (ฒྻॲཧ) -> ݁ՌΛDB΁ • Logूܭ஗Ԇ͕͋Δ఺Λߟྀ (trace_time, trace_time, ྫ֎ॲཧ) • Filter/Transfer/Group • 5tuple+NAT_key fi lter, Packet_IDͷ౷ҰԽʢendianness໰୊ʣ, TSͰ͸ͳ͘ packet_ID, node_ipͰgrouping • Path࠶ߏங • dscp_mark=1, loc:in͕࠷ॳɺ out_dip,out_sip͕มΘΒͳ͚Ε͹ಉ͡node, ࠷ޙ͸dscp_mark=2 • Log Spouts෼ࢄͰෛՙ෼ࢄ 9 σʔλੜ੒ σʔλॲཧ ʢલॲཧʣ σʔλॲཧ
10. 4. EVALUATION • ςετ؀ڥʢ35୆+2vRouterʣͱ࣮ࡍͷΫϥ΢υ؀ڥʢ90%ͷvFD͸VTraceಋೖࡁΈʣͰධՁ • Өڹ͸ڐ༰ൣғ • vRT: rateͰ14%ఔ౓ɺCPUͰ7%ఔ౓ʣ//ӈਤ • vSW: 4%~18%(શύέοτର৅ͷ৔߹) • σʔλΦʔόʔϔου • Ұ࣌อ࣋ྖҬɿ7.2GB (= 600B*2 * 4hop * 300packet * 5000task)ఔ౓ • ଳҬతʹ͸ແࢹͰ͖Δ • ࣮ߦ଎౓ • ςετ؀ڥɿ800task (4hop, 100packet) • 45ඵҎ಺ʹtask׬ྃɺ࠶ߏஙॲཧ͸0.54msఔ౓ • ຊ൪Ϋϥ΢υ؀ڥ • 1packetॲཧʹ5msఔ౓ɻλεΫ׬ྃʹ਺෼ʙ਺े෼͔͔Δʢͭ·Γσʔλੜ੒͕࣌ؒࢧ഑తʣ • खಈͰͷ੾Γ෼͚࡞ۀʢ਺࣌ؒʣʹൺ΂Δͱര଎ 10
11. 5. DEPLOYMENT EXPERIENCES • 2018/05ʹAlibaba Cloud্ͷ90%ͷVFDʹVTraceΛల։ޙͷܦݧ • 2018/05: ϢʔβىҼͷΦʔόʔϨΠNWো֐ • SLBΛܦ༝ͨ͠Πϯλʔωοτ௨৴͕Ͱ͖ͳ͍ʂ • VTraceͰvSwitch಺loc=in,loc=errorΛൃݟɻݪҼ͸ϢʔβACLઃఆʢͭ·ΓΠϯϑϥࣗମ͸ਖ਼ৗʣ • 2018/09: ෺ཧNWো֐ͷݕ஌ • ϥετϗοϓ͕ϥετϗοϓ͡Όͳ͍ʂ(dscp_mark!=2, loc=out) • ݪҼ͸NICϦιʔεෆ଍ʢετϨʔδ௨৴ʹ৯ΘΕ͍ͯͨʣ • ஌ݟ • VFDؒͷʢΦʔόʔϨΠͷʣlatencyҟৗ͸ݕ஌͠΍͍͢ • ਖ਼ৗੑͷ֬౓͕ߴ͍ͨΊΤεύʔ͠΍͍͢ɿྫɿ ଳҬઃఆʹ໰୊͕͋Δ͔΋ʁ • औಘύέοτ਺ɺ࣌ؒ͸Θ͔ͣͰे෼ • ϏδωεϢʔεέʔεͱ࿈ಈʢύέϩεͱௐࠪґཔͷ૿Ճʣ • ࠜຊݪҼͷຆͲ͸ϢʔβʹΑΔςφϯτૢ࡞ɻɻɻ 11
12. 6. DISCUSSIONS • ෺ཧNWো֐͸݁ߏ෼͔Δʢલทʣ • VFDؒ௨৴͕͓͔͍͠ -> PFDͷpathΛ࠶ߏ੒Ͱ͖Δ͔΋ʁ • Ұ࣌తͳύέϩεݪҼΛ஌Γ͍ͨ • ௕࣌ؒΩϟϓνϟ͸ݫ͍͠ɻͦͷ୅ΘΓʹ౷ܭ஋ʢVTrace-statsʣΛ࢖͑Δ͔΋ʁ • stats͸ͦΕͳΓʹෛՙ͕͔͔ΔΒ͍͠ 12 Physical: ෺ཧ
13. 7. RELATED WORK • Active NW monitoring • pingmesh, ATPG, Cisco IPSLA: ຊ൪ͱҧ͏ܦ࿏Λ௨ΔՄೳੑ͕͋Γ • Passive NW monitoring • VeriFlow౳ɿΤϯυϗετʹಋೖ͕ඞཁ • Network debugging method • ৭ʑ͋Δ͕ɺݪҼϊʔυಛఆʹࢸΒͳ͍ʢͱ͍͏ओுʣ • INT (In-band Network Telemetry)͸ലେͳτϥϑΟοΫ/many hop؀ڥͩͱ࣮༻͸Ṗ 13
14. CONCLUSION • VTrace • Ϋϥ΢υεέʔϧͳύέϩεࣗಈݕ஌γεςϜ • VFDͰcoloring/matching/logging͠ɺର৅ϑϩʔΛௐࠪ • 20ϲ݄Ҏ্Քಇ͠ɺ੒Ռ΋ग़͍ͯΔ • ਺෼Ͱ਍அͰ͖ΔɻԾ૝ɾ෺ཧ໰ΘͣτϥϒϧγϡʔςΟϯάͰ͖ͨ 14