Understanding and applying network troubleshooting methodology #edgecore
Network Troubleshooting MethodologyPhil Huang SDN Solution Engineer, Open Networking Division
View Slide
About Me• Phil Huang• Edgecore SDN Solution Engineer• Big Switch Networks• Cumulus Networks• Pica8• Open Networking Linux & OF-DPA• ON.Lab ONOS/CORD Ambassdor• blog.pichuang.com.tw© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
Facebook Datacenter© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://www.zdnet.com/pictures/facebooks-data-centers-worldwide-by-the-numbers-and-in-pictures/
Google Datacenter© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: https://datacentermurals.withgoogle.com/
Amazon Datacenter© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://datacenterfrontier.com/inside-amazon-cloud-computing-infrastructure/
Open Networking Evolution© 2017 Edgecore Networks. All rights reserved | www.edge-core.comDC CoreData Center Clos FabricCloud Service ProvidersTelecom Service ProvidersEnterprise & CampusDC EdgePE Edge
Underlay Network Evolution for Data Center© 2017 Edgecore Networks. All rights reserved | www.edge-core.comThree-Tier ArchitectureRef: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/Leaf-Spine Architecture1 32 4Facebook Fabric
Overlay Network Evolution for Data Center© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: https://www.linkedin.com/pulse/network-virtualization-101-nve-overlay-sdn-dhiman-chowdhury
Network Troubleshooting Methodology1. Identify the problem2. Establish a theory of probable cause3. Test a theory of probable cause4. Establish a plan of action and identify potential effects5. Implement the plan or escalate6. Verify full system functionally7. Document findings, actions, and outcomes© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: CompTIA Network+ Troubleshooting ModelReproduceFixReport
Network Troubleshooting Methodology1. Identify the problem2. Establish a theory of probable cause3. Test a theory of probable cause4. Establish a plan of action and identify potential effects5. Implement the plan or escalate6. Verify full system functionally7. Document findings, actions, and outcomes© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: CompTIA Network+ Troubleshooting ModelReproduce
Reproduce The Problem• Collect useful information• Identify symptoms• Divide and Conquer• Right question help fault seeking© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
Truth TableP Q P→Q ~P->~QF F T TF T T FT F F TT T T T© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: https://youtu.be/hyaA__bLzSQ?list=PLj6E8qlqmkFuqJuqa8y7i2mh3KjetbexW
Truth Table Example• 若網路線沒插 -> Ping 不到• P → Q• 若 Ping 的到,則網路線有插• ~Q → ~P ≡ P → Q• 若 Ping 不到,則網路線沒插• Q → P ≠ P → Q• 若網路線沒插且網路設定錯誤,則 Ping 不到• (P1 ∧ P2) → Q• 若 Ping 不到,則網路線沒插或網路設定錯誤• ~Q -> (~P1 ∨ ~P2) ≡ (P1 ∧ P2) → Q• Too many interfering factor© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
Bayes’s Theorem• The probability of an event, based on prior knowledge of conditionsthat might be related to the event© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: https://en.wikipedia.org/wiki/Bayes%27_theorem
Normal Distribution© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: https://en.wikipedia.org/wiki/Normal_distribution
Network Troubleshooting Methodology1. Identify the problem2. Establish a theory of probable cause3. Test a theory of probable cause4. Establish a plan of action and identify potential effects5. Implement the plan or escalate6. Verify full system functionally7. Document findings, actions, and outcomes© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: CompTIA Network+ Troubleshooting ModelReport
8D Report• 8 Disciplines Problem Solving• Problem-Solving process with eight objective• Team-Oriented• Follow the logic of the PDCA (Plan, Do, Check, Act) cycle• Procedure• D0: Plan• D1: Form the team• D2: Describe the problem• D3: Develop Interim Containment Plan• D4: Determine and Verify Root Causes and Escape Points:• D5: Verify Permanent Corrections (PCs) for Problem will resolve problem for the customer• D6: Define and Implement Corrective Actions• D7: Prevent Recurrence / System Problems• D8: Congratulate Your Team© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
Network Troubleshooting Methodology1. Identify the problem2. Establish a theory of probable cause3. Test a theory of probable cause4. Establish a plan of action and identify potential effects5. Implement the plan or escalate6. Verify full system functionally7. Document findings, actions, and outcomes© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: CompTIA Network+ Troubleshooting ModelFix
OSI Model v.s TCP/IP Model© 2017 Edgecore Networks. All rights reserved | www.edge-core.comPhysicalData linkNetworkTransportSessionPresentationApplicationApplicationTransportInternetNetworkInterfaceIP Protocol BGP, OSPF, RIP ICMPTCP UDPARP Internet DriverTunnel Protocols(GRE, MPLS, L2TP)SSL / TLSDNS HTTP DHCPRPC LDAPSSH TELNET NFS
Top-Down and Bottom-Up Approach© 2017 Edgecore Networks. All rights reserved | www.edge-core.comApplicationTransportInternetNetworkInterfaceARPIP Protocol BGP, OSPF, RIPInternet DriverICMPTCP UDPSSL / TLSDNS HTTP DHCPTunnel Protocols(GRE, MPLS, L2TP)RPC LDAPSSH TELNET NFSBottom-Up Top-Down
Divide and Conquer Approach© 2017 Edgecore Networks. All rights reserved | www.edge-core.comApplicationTransportInternetNetworkInterfaceARPIP Protocol BGP, OSPF, RIPInternet DriverICMPTCP UDPSSL / TLSDNS HTTP DHCPTunnel Protocols(GRE, MPLS, L2TP)RPC LDAPSSH TELNET NFS
OpenStack Network Troubleshooting© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: https://docs.openstack.org/ops-guide/ops-network-troubleshooting.html
Ping Command• ping www.edge-core.com• ping 8.8.8.8© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://www.thegeekstuff.com/2009/11/ping-tutorial-13-effective-ping-command-examples/28 bytes = IP Header (20 bytes)+ ICMP Header (8 bytes)
Action: Ping Domain Name© 2017 Edgecore Networks. All rights reserved | www.edge-core.comHost 1100.10Host 2100.12192.168.100.0/24192.168.100.254Google DNS8.8.8.8Edgecorewww.edge-core.com
Action: Ping DNS IP© 2017 Edgecore Networks. All rights reserved | www.edge-core.comHost 1100.10Host 2100.12192.168.100.0/24192.168.100.254Google DNS8.8.8.8Edgecorewww.edge-core.com
Action: Ping Gateway© 2017 Edgecore Networks. All rights reserved | www.edge-core.comHost 1100.10Host 2100.12192.168.100.0/24192.168.100.254Google DNS8.8.8.8Edgecorewww.edge-core.com
Action: Ping Host in Same LAN© 2017 Edgecore Networks. All rights reserved | www.edge-core.comHost 1100.10Host 2100.12192.168.100.0/24192.168.100.254Google DNS8.8.8.8Edgecorewww.edge-core.comping 192.168.100.12
My Traceroute (mtr) Command• Combines the functionally of the traceroute and ping programs© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
iptables• Total of 4 Tables• raw > mangle > nat > filter (default table)• Total of 5 Chains• PREROUTING, INPUT, FORWARDING, OUTPUT, POSTROUTING© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://inai.de/images/nf-packet-flow.png• Network Layer• Iptables, ip6tables• Link Layer• ebtables
NAT Internal Network to Internet• iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
Port Forwarding• iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 3389 -jDNAT --to-destination :3389© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
Drop ICMP Packet• iptables -t filter -A FORWARD -p ICMP -j DROP© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
ip link command• Command• ip link show• ip link set dev eth0 up• ip link set dev eth0 mtu 9000© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://baturin.org/docs/iproute2/#Link management
ip addr command• Command• ip addr show• ip addr add 192.168.101.12/24 dev eth0© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://baturin.org/docs/iproute2/#Address management
ip route command• Management connected routes• Command• ip route add default via 192.168.100.254 dev eth0• ip route add 172.17.0.0/16 dev docker0© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://baturin.org/docs/iproute2/#Route management
ip monitor command• Network event monitoring• Command• ip monitor all© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: http://baturin.org/docs/iproute2/#Network event monitoring
Advanced Command• ip tuntap• Allow userspace programs to emulate a network device• TUN sends and receives IP packets (L3)• TAP sends and receives Ethernet frames (L2)• ip netns• ip rule© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
dig command• Use dig command for DNS lookup and query DNS name servers forvarious resource record• Command• dig edge-core.com ANY +noall +answer• dig @8.8.8.8 www.edge-core.com• dig –x 140.113.235.1© 2017 Edgecore Networks. All rights reserved | www.edge-core.comRef: https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/
lsof command• List open files• Command• lsof -i:80• lsof /mnt/usb© 2017 Edgecore Networks. All rights reserved | www.edge-core.com
3/16/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com41Open NetworkingfromFreedomControlInnovation