Network Troubleshooting Methodology

Understanding and applying network troubleshooting methodology
#edgecore

1. Network Troubleshooting Methodology Phil Huang <phil_huang@edge-core.com> SDN Solution Engineer, Open Networking Division
2. About Me • Phil Huang • Edgecore SDN Solution Engineer • Big Switch Networks • Cumulus Networks • Pica8 • Open Networking Linux & OF-DPA • ON.Lab ONOS/CORD Ambassdor • blog.pichuang.com.tw © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
3. Facebook Datacenter © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://www.zdnet.com/pictures/facebooks-data-centers-worldwide-by-the-numbers-and-in-pictures/
4. Google Datacenter © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://datacentermurals.withgoogle.com/
5. Amazon Datacenter © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://datacenterfrontier.com/inside-amazon-cloud-computing-infrastructure/
6. Open Networking Evolution © 2017 Edgecore Networks. All rights reserved | www.edge-core.com DC Core Data Center Clos Fabric Cloud Service Providers Telecom Service Providers Enterprise & Campus DC Edge PE Edge
7. Underlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Three-Tier Architecture Ref: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ Leaf-Spine Architecture 1 3 2 4 Facebook Fabric
8. Overlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://www.linkedin.com/pulse/network-virtualization-101-nve-overlay-sdn-dhiman-chowdhury
9. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Reproduce Fix Report
10. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Reproduce
11. Reproduce The Problem • Collect useful information • Identify symptoms • Divide and Conquer • Right question help fault seeking © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
12. Truth Table P Q P→Q ~P->~Q F F T T F T T F T F F T T T T T © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://youtu.be/hyaA__bLzSQ?list=PLj6E8qlqmkFuqJuqa8y7i2mh3KjetbexW
13. Truth Table Example • 若網路線沒插 -> Ping 不到 • P → Q • 若 Ping 的到，則網路線有插 • ~Q → ~P ≡ P → Q • 若 Ping 不到，則網路線沒插 • Q → P ≠ P → Q • 若網路線沒插且網路設定錯誤，則 Ping 不到 • (P1 ∧ P2) → Q • 若 Ping 不到，則網路線沒插或網路設定錯誤 • ~Q -> (~P1 ∨ ~P2) ≡ (P1 ∧ P2) → Q • Too many interfering factor © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
14. Bayes’s Theorem • The probability of an event, based on prior knowledge of conditions that might be related to the event © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://en.wikipedia.org/wiki/Bayes%27_theorem
15. Normal Distribution © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://en.wikipedia.org/wiki/Normal_distribution
16. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Report
17. 8D Report • 8 Disciplines Problem Solving • Problem-Solving process with eight objective • Team-Oriented • Follow the logic of the PDCA (Plan, Do, Check, Act) cycle • Procedure • D0: Plan • D1: Form the team • D2: Describe the problem • D3: Develop Interim Containment Plan • D4: Determine and Verify Root Causes and Escape Points: • D5: Verify Permanent Corrections (PCs) for Problem will resolve problem for the customer • D6: Define and Implement Corrective Actions • D7: Prevent Recurrence / System Problems • D8: Congratulate Your Team © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
18. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Fix
19. OSI Model v.s TCP/IP Model © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Physical Data link Network Transport Session Presentation Application Application Transport Internet Network Interface IP Protocol BGP, OSPF, RIP ICMP TCP UDP ARP Internet Driver Tunnel Protocols (GRE, MPLS, L2TP) SSL / TLS DNS HTTP DHCP RPC LDAP SSH TELNET NFS
20. Top-Down and Bottom-Up Approach © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS Bottom-Up Top-Down
21. Divide and Conquer Approach © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS
22. OpenStack Network Troubleshooting © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://docs.openstack.org/ops-guide/ops-network-troubleshooting.html
23. Ping Command • ping www.edge-core.com • ping 8.8.8.8 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://www.thegeekstuff.com/2009/11/ping-tutorial-13-effective-ping-command-examples/ 28 bytes = IP Header (20 bytes)+ ICMP Header (8 bytes)
24. Action: Ping Domain Name © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com
25. Action: Ping DNS IP © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com
26. Action: Ping Gateway © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com
27. Action: Ping Host in Same LAN © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com ping 192.168.100.12
28. My Traceroute (mtr) Command • Combines the functionally of the traceroute and ping programs © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
29. iptables • Total of 4 Tables • raw > mangle > nat > filter (default table) • Total of 5 Chains • PREROUTING, INPUT, FORWARDING, OUTPUT, POSTROUTING © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
30. © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://inai.de/images/nf-packet-flow.png • Network Layer • Iptables, ip6tables • Link Layer • ebtables
31. NAT Internal Network to Internet • iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
32. Port Forwarding • iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 3389 -j DNAT --to-destination <TARGET_IP>:3389 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
33. Drop ICMP Packet • iptables -t filter -A FORWARD -p ICMP -j DROP © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
34. ip link command • Command • ip link show • ip link set dev eth0 up • ip link set dev eth0 mtu 9000 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Link management
35. ip addr command • Command • ip addr show • ip addr add 192.168.101.12/24 dev eth0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Address management
36. ip route command • Management connected routes • Command • ip route add default via 192.168.100.254 dev eth0 • ip route add 172.17.0.0/16 dev docker0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Route management
37. ip monitor command • Network event monitoring • Command • ip monitor all © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Network event monitoring
38. Advanced Command • ip tuntap • Allow userspace programs to emulate a network device • TUN sends and receives IP packets (L3) • TAP sends and receives Ethernet frames (L2) • ip netns • ip rule © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
39. dig command • Use dig command for DNS lookup and query DNS name servers for various resource record • Command • dig edge-core.com ANY +noall +answer • dig @8.8.8.8 www.edge-core.com • dig –x 140.113.235.1 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/
40. lsof command • List open files • Command • lsof -i:80 • lsof /mnt/usb © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
41. 3/16/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 41 Open Networking from Freedom Control Innovation