Share
Embed
Leveraging SDN Layering to Systematically Troubleshoot Networks
Talk I gave jointly with Brandon Heller at HotSDN 2013.
Paper: http://www.eecs.berkeley.edu/~rcs/research/vision-paper.pdf
Colin Scott
November 02, 2015
Transcript
19 slides
-
Leveraging SDN Layering to Systema2cally Troubleshoot Networks Brandon
Heller★ Colin Sco/u Nick McKeown⌘ Sco= Shenker u¤ Andreas Wundsam § Hongyi Zeng⌘ Sam Whitlocku Vimalkumar Jeyakumar⌘ Nikhil Handigol★ -
Admin Network skills + tools + knowledge
Protocols Configura2on Topology Policy • connect hosts A + B • quaran2ne virus-‐ infected hosts • route guest traffic to an HTTP proxy • priori2ze SSH + à 1: Configure Ethane, overlays, consistency primi2ves, network programming languages, … 3: Fix Stuff! 2: Troubleshoot This Talk -
Admin Network skills + tools + knowledge
Protocols Configura2on Topology Policy • connect hosts A + B • quaran2ne virus-‐ infected hosts • route guest traffic to an HTTP proxy • priori2ze SSH + à 1: Configure Ethane, overlays, consistency primi2ves, network programming languages, … 3: Fix Stuff! 2: Troubleshoot #1 request from network admins: Automa<c Troubleshoo<ng Source: “Automa2c Test Packet Genera2on”, CoNEXT ‘12, Zeng et al. This Talk -
How to automate troubleshoo2ng? Network Policy •
isolate groups A + B • route guest traffic to an HTTP proxy • block a list of virus-‐ infected hosts Challenging in tradi2onal networks. ~ ? (2) Check behavior against policy: • confusing: don’t know lowest-‐level forwarding behavior • distributed: hard to get a meaningful snapshot Two requirements. (1) Know the intended policy: • confusing: different config format for each protocol • distributed: configura2on spread among all nodes • hard: must understand all protocols & their interac2ons difficult to check imprac<cal to infer -
Control-‐Plane Layering in SDN Firmware Firmware Firmware
Network Hypervisor App App App State Layers Logical View Physical View Device State Hardware Policy Code Layers Network OS HW HW HW -
Firmware Firmware Firmware HW HW
HW Systema<cally Troubleshoo<ng an SDN Network OS Network Hypervisor App App App State Layers Logical View Physical View Device State Hardware Policy Code Layers Observa<on: Each state layer fully specifies network behavior. Insight: Bugs manifest as mistransla2ons between layers. Systema<c Approach: (1) Binary search to isolate to a code layer. (2) Leverage state to isolate within the code layer. -
Phase 1: Localizing to a code layer [Operator Intent]
Logical View Physical View Device State Hardware Policy ? ~ Apps NetHyperV NetOS Firmware [Actual Behavior] Cause: Firmware Bug Yes No ? ~ Yes No ? ~ Yes No SOFT [CONEXT ‘12] Anteater [SIGCOMM ‘11] Symptom: Hosts unable to communicate -
Phase 1: Localizing to a code layer [Operator Intent]
Logical View Physical View Device State Hardware Policy ? ~ Apps NetHyperV NetOS Firmware [Actual Behavior] Yes No ? ~ Yes No Symptom: Tenant Isola2on Breach HSA [NSDI ’12] OFRewind [ATC ‘11] Yes No ? ~ ? ~ Yes No Correspondence Checking Cause: NetHypervisor Bug -
How to automate troubleshoo2ng? Network Policy •
isolate groups A + B • route guest traffic to an HTTP proxy • block a list of virus-‐ infected hosts Possible in Sonware-‐Defined Networks ~ ? (2) Check behavior against policy: • confusing: don’t know lowest-‐level forwarding behavior • distributed: hard to get a meaningful snapshot Two requirements. (1) Know the intended policy: • confusing: different config format for each protocol • distributed: configura2on spread among all nodes • hard: must understand all protocols & their interac2ons directly accessible directly provided app fewer nodes -
Takeways • Control plane layering enables systema2c troubleshoo2ng
• Thinking about troubleshoo2ng in terms of layers shows us where tools fit in – Reveals missing tools – Highlights choices between tools, with tradeoffs • Plenty of opportuni2es len. Opera2onalize! -
Leverage the layers in SDN. Brandon Heller★
Colin Sco/u Nick McKeown⌘ Sco= Shenker u¤ Andreas Wundsam § Hongyi Zeng⌘ Sam Whitlocku Vimalkumar Jeyakumar⌘ Nikhil Handigol★ -
How is this different than general distributed systems debugging?
• Simple answer: it’s not! SDN is an excellent opportunity to draw upon ideas from other distributed systems • Subtlety: networks are solving a much more constrained problem than general distributed systems -
Limita2ons • Correctness only, not performance • Side
effects not reflected in state • No guarantee of finding single code layer • No guarantee of individual layer correctness • No guarantee of future correctness • Layer visibility may be imperfect -
Plenty of Opportuni2es Remain • Automa2c Troubleshoo2ng à
Ac2onable Bug Reports – Filtering the signal from the noise – Crea2ng consistent views of state • Improving Invariant Checkers – Scale – Flexible Policy Input • Hybrid Tradi2onal + SDN Debugging -
Plenty of Opportuni2es Remain • Automa2c Troubleshoo2ng à
Ac2onable Bug Reports – Filtering the signal from the noise – Crea2ng consistent views of state Packet History: Path + Headers + Forwarding State Forwarding State Forwarding State Forwarding State Forwarding State [HotSDN 2012: Where is the Debugger for My Sonware-‐Defined Network?] -
Plenty of Opportuni2es Remain • Automa2c Troubleshoo2ng à
Ac2onable Bug Reports – Filtering the signal from the noise Controller A Controller B Controller C Switch 1 Switch 2 Switch3 Switch 4 Switch 5 Switch 6 Switch 7 Switch 8 Switch 9 [Berkeley Tech Report: How Did We Get Into This Mess? Isola2ng Fault-‐Inducing Inputs to SDN Control Sonware] Minimal Causal Sequence -
Isn’t this unnecessary with consistency primi2ves/languages/etc? • No
• Catch/rule out bugs outside the framework • Catch instances where the framework pushes config that breaks the policy -
Control-‐Plane Layering in SDN Firmware Firmware Firmware
Network Hypervisor App App App State Layers Logical View Physical View Device State Hardware Policy Example Errors Configuration Parsing Error Tenant isolation breach (policy mistranslation) Failover logic error, synchronization bug Register misconfiguration, Router memory corruption Code Layers [Unintended Config] [External Connectivity Error] Network OS HW HW HW