Research Paper Introduction #42 “Robotron: Top-down Network Management at Facebook Scale” ௨ࢉ#110 @cafenero_777 2022/12/08 1

Agenda •ର৅࿦จ •֓ཁͱಡ΋͏ͱͨ͠ཧ༝ 1. INTRODUCTION 2. THE NETWORK AND USE CASES 3. ROBOTRON OVERVIEW 4. FBNET: MODELING THE NETWORK 5. MANAGEMENT LIFE SYCLE 6. USAGE STATISTICS 7. ROBOTRON EVOVUTION 8. EXPERIENCE AND FUTURE WORK 9. RELATED WORK 10.CONCLUTION 2

ର৅࿦จ •Robotron: Top-down Network Management at Facebook Scale • Yu-Wei Eric Sung, Xiaozheng Tie, Starsky H.Y. Wong, and Hongyi Zeng • Facebook, Inc. • SIGCOMM ‘16 • https://dl.acm.org/doi/10.1145/2934872.2934874 3

֓ཁͱಡ΋͏ͱͨ͠ཧ༝ •֓ཁ • ωοτϫʔΫ؅ཧ͸ඇৗʹॏཁͳͷͰɺ͏·͘΍Δඞཁ͕͋Δ • Robotron: high-level design intent -> low-level device con fi g • FBͰ͸2008೥͔Β਺ສ୆ͷNWػثʢ਺ेສ୆ͷαʔόऩ༰ʣʹ࢖ΘΕ͍ͯΔ •ಡ΋͏ͱͨ͠ཧ༝ • ࠷ۙಡΜͰΔ࿦จͰҾ༻͞ΕΔ͜ͱ͕ଟͯ͘ؾʹͳͬͨɻ • େن໛NW؅ཧͱӡ༻ɺͲ͏΍ͬͯΔͷʁ 4 ”ଟ͘ͷਓʑ͸ѱ͍ܦӦͱӡ໋Λࠞಉ͍ͯ͠Δ” ʢӡ༻ʣ

1. INTRODUCTION (1/2) 5 •େن໛ͰμΠφϛοΫͳNWӡ༻͸େมͳΜͩΑʂ • ճઢ઀ଓɾަ׵ɺσόΠεઃఆɺOS upgrade, ACLઃఆɺϓϩτίϧνϡʔχϯάɺNWΠϕϯτɾ౷ܭͷ؂ࢹ • ద੾ͳNW؅ཧॏཁɺྫ: ϓϩτίϧػೳͷ׆༻ɾਓ͕བྷΉ࡞ۀͱϦεΫɺΞδϟΠϧʢϢʔβཁ๬΍༰ྔΞοϓάϨʔυʣ •ͦΕ͕“ӡ༻”ͳΜͰ͠ΐʁ -> NO! Ͱ΋ެ։৘ใ͸͋·Γͳ͍ঢ়گɻ •؅ཧܥͷ՝୊ྫ • Distributed Con fi gurations: ΠϯςϯτΛ࣮ࡍͷcon fi g΁ɻྫɿճઢมߋʢIPΞυϨεɺBGP,ηογϣϯ I/Fઃఆ, etcʣ • Multiple domains: NW of NW: POPs, BB, DCs, ͦΕͧΕτϙϩδ΍ϙϦγ΋ҧ͏͕ɺಉҰ૊৫಺Ͱ઀ଓɻ • Versioning: NWτϙϩδ΍ϧʔςΟϯάઃܭɺϦϯΫ଎౓͸࣌ؒͱڞʹେ͖͘มԽɻෳ਺όʔδϣϯͷ௕ظؒ؅ཧ͕ඞཁɻ • Dependency: ઃఆґଘ͕͋Δ΋ͷ΋ɻྫ: iBGPߏ੒ʹϧʔλ௥Ճ͢ΔͱશϧʔλͰ௥Ճઃఆ͕ඞཁɻ • Vendor di ff erences: ϕϯμʔݻ༗ͷίϚϯυ͸ศར͕ͩඪ४Խ͞Εͯ͸͍ͳ͍ɻϓϩτίϧ࣮૷΋ϕϯμɾOSຖʹඍົʹҧ͏ͨΊอकࠔ೉ɻ

1. INTRODUCTION (2/2) •Robotronઃܭͱಋೖ • Con fi guration-as-code: ίʔυԽɻґଘੑɾ࠶ݱੑɾҰ؏ੑͷ୲อ • Validation: ࣗಈݕূػೳɻࠩ෼ఏࣔ֬ೝɻσϓϩΠޙͷৄࡉϞχλϦϯάτϥοΩϯά • Extensibility: ςϯϓϨɾπʔϧɾίʔυͰػೳ֦ு • ྫɿσόΠελΠϓɾճઢλΠϓɾτϙϩδ •2008೥͔ΒϓϩμΫγϣϯʢ਺ສ୆ن໛ͷNWػثʣͰ҆ఆՔಇ •ຊ࿦จɿେن໛NWͷ՝୊ఏࣔɾRobotronઃܭͱ࣮૷ɾར༻ঢ়گͱ஌ݟڞ༗ 6

2. THE NETWORK AND USE CASES •POP (Point-of-Presence) • ϐΞϦϯάɺlink༰ྔௐ੔ɺBGPઃఆมߋ •DC (Data Center) • ൺֱతNWมߋ͸গͳ͍ • αʔϏε༻αʔό΁ͷ഑ઢɾIPׂ౰ͳͲ •BB (Backbone) • MPLS/BGPͰPOP-DCؒτϥϯεϙʔτΛఏڙ • ճઢ૿ઃ࡞ۀ΍iBGPઃఆ࡞ۀ 7 PeeringRouter POP servers (LB, cache)΁ ࣗࣾDC໢΁

3. ROBOTRON OVERVIEW •खಈઃఆͱΞυϗοΫͳεΫϦϓτ͔Βͷ୤٫΁ • 2008೥Ҏ߱͸FBNetʢΦϒδΣΫτετΞʣΛ༻͍ͨtop-down network management • NWઃܭɺઃఆੜ੒ɺσϓϩΠɺ؂ࢹΛαϙʔτ •FBNet • SSOT (Single Source Of Truth)ͱͯ͠ͷΦϒδΣΫτετΞ • ֤ίϯϙʔωϯτ͸ΦϒδΣΫτͱͯ͠ϞσϧԽ͠ɺObjಉ࢜͸ଐੑ (attribute)ͱͯ͠දݱ •Network Design • ઃܭ (intent)͔Β֤ΦϒδΣΫτʹม׵ɾอ࣋ • ྫɿΫϥελߏஙʢϥοΫ਺ɺuplink਺ʣ-> ToR, ഑ઢ, I/F, IPΞυϨε •Con fi g Generation: ϕϯμʔݻ༗ͷઃఆΛੜ੒ɻςϯϓϨ֦ுͰ৽ػثαϙʔτɻ •Deployment: ࣮ػʹઃఆ౤ೖɻΧφϦʔϦϦʔε •Monitoring: ๬·͍͠ঢ়ଶ͔Βҳ୤͠ͳ͍͜ͱΛ֬ೝ 8

4. FBNET: MODELING THE NETWORK Data Model •Object, Value, and Relationship • ܕ෇͖ΦϒδΣΫτ: ෺ཧʢσόΠεɾϥΠϯΧʔυɾI/Fɾճઢʣͱ࿦ཧʢBGPηογϣϯɾIPʣ • ΦϒδΣΫτͷ஋ͱͦΕΒΛؔ࿈෇͚͢Δattributes৘ใ •Desired V.S. Derived • Desired: ObjσʔλΛมߋ࣮ͯ͠σόΠεʹ൓ө • Derived: ࣮σόΠεͷϦΞϧλΠϜ৘ใ͔ΒobjΛੜ੒ • DesiredͱDerivedͷσʔλࠩͰҟৗݕग़ • ઃఆະద༻ɺઃఆϛεɺHW/ fi berো֐ •ઃܭͷ޻෉ɿ • ඞཁͳϑΟʔϧυɾσʔλ͚ͩूΊΔ • ҟͳΔଐੑͰ΋ɺͳΔ΂͘ൺֱ͠΍͍͢Α͏ʹ͢Δ • σʔλιʔε͸SSOT (Single Source Of Truth)ʹ͢Δ 9 NWਤͰॻ͘ͱ͜͏͕ͩɺ Object/RelationͰॻ͘ͱ͜Εɻ

4. FBNET: MODELING THE NETWORK API •Read API • Obj୯ҐͰऔಘՄೳ • fi eld: Objͷ஋ϑΟʔϧυͷϦετ • query: ԋࢉࢠ΍regexΛ༻͍ͨϑΟϧλࣜ •Write API • ϋΠϨϕϧAPI • σʔλ੔߹ੑͷͨΊɺෳ਺ObjΛಉ࣌ʹมߋ͢Δ • ৽چͷObjΛೖྗͱͯ͠ɺ࡞੒ɾҠߦɾߋ৽ɾ࡟আΛߦ͏ 10 ྫ: get (, device.name == “hoge”)

4. FBNET: MODELING THE NETWORK Architecture and Implementation •෼ࢄΞʔΩςΫνϟɺ࣮૷ɺσʔληϯλͰͷεέʔϧɺରো֐ੑ •Storage Layer • Obj + attribute (relationship) -> MySQLͰอ࣋ɻORMར༻ •Service Layer • Thrift API (RPC)ͳͷͰݴޠඇґଘ • ॻ͖ࠐΈAPI͸Ϣʔεέʔεຖʹࣄલఆٛ͞Ε͍ͯΔ •Scalability and Availability • DCຖʹ1Ϛελʔɺෳ਺εϨʔϒ • Ϛελʔμ΢ϯ࣌͸Ұ൪ۙ͘ͷDCͷεϨʔϒ͕Ϛελʔʹঢ֨ 11

5. MANAGEMENT LIFE CYCLE Network Design •ਓ͕ؒࢦఆͨ͠high-levelNW ઃܭΛdesired Objʹม׵ͯ͠࢖༻ •POPs and DCs • FatTree, ߏ੒มԽͤͣɻςϯϓϨʔτྫɻߏ੒มߋ/upgradeͷݕূʹ΋࢖͏ •Backbone • มߋଟ͍ɿಈతͳ༰ྔɾճઢͷ૿ݮɺϧʔλͷ௥Ճɾ࡟আ • Objґଘؔ܎ɿBBϧʔλ௥Ճ->શϧʔλͷiBGPઃఆߋ৽ • -> relationship fi eld͔ΒશͯΛḷͬͯߋ৽Λ͔͚Δ •Design Validation • σʔλ੔߹ੑνΣοΫɺ݁ՌͷՄࢹԽνΣοΫɺࣾһID/νέοτID, ઃܭมߋϩά 12 ྫ: HWσόΠεͱ਺ɾ֤λΠϓͷ਺ɾ ઀ଓํ๏ɾΞυϨεํࣜ -> 94ݸͷObjੜ੒ɾؔ࿈෇͚

5. MANAGEMENT LIFE CYCLE Con fi g Generation •FBNet Obj͔Βϕϯμʔݻ༗ͷcon fi gΛੜ੒ • ϕϯμʔඇґଘσʔλ͸FBNet͔ΒThrift Objʹม׵ • ϕϯμʔݻ༗ߏจ͸ςϯϓϨΛ಺෦ར༻ •ੜ੒͞Εͨcon fi g • ϨϙδτϦʹอଘ͞ΕͯϐΞϨϏϡʔͱunittest • ઃఆલʹػثઃఆͷόοΫΞοϓ • ઃఆલޙͰػثΛϞχλϦϯά 13 CࣾͱJࣾʁ

5. MANAGEMENT LIFE CYCLE Deployment •Init Provisioning • ॳճߏங࣌ͳͲɻτϥϑΟοΫᷖճ͸ඞཁɻ •Incremental Updates • ӡ༻தͷػثͷઃఆͷҰ෦ͷΈมߋ • Dryrun Mode: ৽چcon fi gࠩ෼ൺֱ • Atomic Mode: ෳ਺σόΠεಉ࣌σϓϩΠʢྫ: iBGPϝογϡʣͱτϥϯβΫγϣϯϩʔϧόοΫ • Phased Mode: ஈ֊తσϓϩΠɻ͖͍͠஋ʢσόΠε%, Ϧʔδϣϯɺroleʣຖʹgo/no-go͢Δ • Human Con fi rmation: ਓ్͕ؒதͰಈ࡞֬ೝ࡞ۀΛߦ͏ɻ 14

5. MANAGEMENT LIFE CYCLE Monitoring •Passive Monitoring • SyslogͰ༷ʑͳΠϕϯτʢϑϥοϐϯάɺϦϒʔτ౳ʣΛัଊ • BGP AnycastѼʹૹΔɻcollecterଆͰ͸ਖ਼نදݱϧʔϧʹै͍Ξϥʔτ্͛Δ •Active Monitoring • CPU/Memory/Link࢖༻཰ͷ؂ࢹjobͷੜ੒ •Con fi g Monitoring • Active/Passive MonitoringΛ྆ํ࢖͏ • ੜ੒con fi gͱσϓϩΠ͞Εͨcon fi gΛൺֱɻϠόͦ͏ͳΒcon fi gΛ໭͢ɻ 15

6. USAGE STATISTICS •POP: εϖʔε΍ిྗ੍ݶͷͨΊupgradeͷͨΊΨϥοͱมΘΔ •DC: ৽Ϋϥελ௥Ճͱچੈ୅ഇࢭ·Ͱซ༻ •γεςϜύϑΥʔϚϯεʢtask׬ྃ࣌ؒͳͲʣͰ͸ͳ͘ɺRobotronͷ࢖༻౷ܭΛݟ͍ͯ͘ •FBNet Models • Ϟσϧؔ࿈෇͚਺ʢcircuit -> physical interface -> aggregated interfaceʣ • Ϟσϧมߋճ਺ʢ3೥ؒͷ#/weekʣɺݱ࣮తʹ͸Ϟσϧ͸҆ఆ͠ͳ͍ • ৽͍͠ίϯϙʔωϯτ • HWίϯϙʔωϯτ͚ͩͰ͸ͳ͍ɻྫɿGen1 (L2) -> Gen2 (L3 BGP)ͰBGPv4ηογϣϯϞσϧ • ৽͍͠ΞτϦϏϡʔτ • ඞཁʹԠ͍ͯ͡௥ՃɻྫɿBBϧʔλͷᷖճ༻ʹdrain_state௥Ճ • ϩδοΫͷมߋ • on-the- fl yͰੜ੒ʢnot อ࣋ʣɻྫɿৄࡉදࣔ༻ͷasset_urlଐੑ 16 ͜͜2೥ͷมભ -POP͸gen1->gen2 -DC͸3ੈ୅ซ༻ -Private v4ރׇ -> v6΁

7. ROBOTRON EVOLUTION •2008೥͔Βελʔτɻ࠷ॳ͸top-downͰ͸ͳ͔ͬͨɻ • ࠷ॳ͸active/passive monitoringͱՄࢹԽ͔Βελʔτ • FBNet͸্هνΣοΫ༻ͷloopback IPอ࣋ʢੜσʔλʣͷͨΊ࡞ΒΕͨ • ϞσϧԽͰϕϯμʔඇґଘ΁ • ઃܭϛεɺઃఆϛεɺHWো֐ΛϞσϧԽʹରͯ͠ΞυϗοΫ؂ࠪՄೳʹ • σϓϩΠΛखಈ͔Βࣗಈ+֤छ҆શରࡦ෇͖ • BBճઢ૿ઃ༻ʹDesired Model௥Ճʢະ࢖༻IPʹping͢Δ౳͸େมͩ͠Τϥʔ͕ଟ͍ͨΊɻʣ • ϕϯμʔݻ༗ςϯϓϨͷ֦ॆ 17

8. EXPERIENCE AND FUTURE WORK •ϞσϦϯάͷෳࡶ͞ • ֎෦ISP༻ͷBGPΧελϜimport policy (pre fi x list)͕ඞཁͳέʔεͰࣄނ • ͙͢ʹ؂ࢹͰؾ͍ͮͯroll backͨ͠ɻϞσϧσβΠϯमਖ਼ͰରԠ •ݹ͍ (stale)ઃఆ • ઃఆมߋޙɺ࣌ؒΛ͓͍͔ͯΒσϓϩΠͨ͠ɻͦͷؒʹઃఆ͕มΘ͍ͬͯͨ • ෺ཧ࡞ۀʢ഑ઢมߋͳͲʣ͕ඞཁͳ৔߹΋͋Γɻ࣌ܥྻ΋ߟྀ͢Δʁ͜ͷن໛ͰϩοΫ࢖༻͸ແཧ •ࣗಈfallback • Robotronͷόάɾ࢖͍ͨ͘ͳ͍ɾະରԠۓٸରԠͰखಈΦϖͰ໰୊Ҿ͖ى͜͢ • ӡ༻ܦݧ্ɺ৴པͰ͖Δfallbackػߏ͕ඞཁɻ • ؂ࢹΞϥʔτͰʢਫ਼ਆతʹʁʣ཈੍ɻۓٸ࣌windowҎ֎͸Robotron͕࣮ػઃఆΛ΋ͱʹ໭͢ 18

9. RELATED WORKS •ϓϩόΠμNW΍ΤϯϓϥNWͷbottom-up੩తߏ੒෼ੳ͕ଟ͍ + con fi gղੳͱτϥ γϡʔͷҰൠԽ • ຊݚڀɿBB/POP/DC͔ͭϚϧνυϝΠϯΛTop-downͰܧଓ࣮ࢪ •ந৅ԽͰ͏·͘΍Δܥ • Robotron͸ந৅Խͷൣғ͕΋ͬͱ޿͍ • SW։ൃख๏Λద༻ • ॳظηοτΞοϓ͚ͩͰͳ͘ɺՔಇࡁΈ؀ڥ΋Χόʔ 19

10. CONCLUSION •Robotron • FacebookͷDC/BB/POP؅ཧΛաڈ8೥ؒӡ༻ • ઃܭɾ࣮૷ɾӡ༻஌ݟΛ঺հ • ϓϩμΫγϣϯNWӡ༻ͷ঺հ •ݚڀɾNWӡ༻ͷ஌ݟʹͳΕ͹޾͍ 20

ࡾߦ·ͱΊ • Robotron: FacebookͷσʔληϯλNW؅ཧγεςϜ • IPAMʹCI/CDΛ͚ͭͨΑ͏ͳΠϝʔδ • ϓϩμΫγϣϯͰ8೥ؒӡ༻ • ઃܭ΍ӡ༻஌ݟΛ঺հ 21

EoP 22

ิ଍εϥΠυ 23

6. USAGE STATISTICS Design/Con fi g Change, Monitoring Usage •ઃܭมߋͱؔ࿈Obj਺ • 120, 20 @ 50%ile • I/F Objଟ͍ɺv6-onlyΫϥελͩͱv6ߋ৽ස౓ͷํ͕ଟ͍ •ઃఆมߋ • PR/BB: ߋ৽ଟ͍(12/week)ɺ಺༰গͳ͍(150ߦ) • DC: ߋ৽গͳ͍ʢ2/weekʣɺ಺༰ଟ͍(700ߦ) •؂ࢹ • SNMP, Syslog, CLI ! • Syslog 95%͸ແࢹର৅ɻຊ౰ʹϠό͍ͷ͸0.01%ͷ΍ͭ 24 3ϲ݄ͷมߋ෼