Network Troubleshooting Methodology

Network Troubleshooting Methodology

Understanding and applying network troubleshooting methodology
#edgecore

D907136acebc72f1df878541b26f271a?s=128

Phil Huang

March 16, 2017
Tweet

Transcript

  1. Network Troubleshooting Methodology Phil Huang <phil_huang@edge-core.com> SDN Solution Engineer, Open

    Networking Division
  2. About Me • Phil Huang • Edgecore SDN Solution Engineer

    • Big Switch Networks • Cumulus Networks • Pica8 • Open Networking Linux & OF-DPA • ON.Lab ONOS/CORD Ambassdor • blog.pichuang.com.tw © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  3. Facebook Datacenter © 2017 Edgecore Networks. All rights reserved |

    www.edge-core.com Ref: http://www.zdnet.com/pictures/facebooks-data-centers-worldwide-by-the-numbers-and-in-pictures/
  4. Google Datacenter © 2017 Edgecore Networks. All rights reserved |

    www.edge-core.com Ref: https://datacentermurals.withgoogle.com/
  5. Amazon Datacenter © 2017 Edgecore Networks. All rights reserved |

    www.edge-core.com Ref: http://datacenterfrontier.com/inside-amazon-cloud-computing-infrastructure/
  6. Open Networking Evolution © 2017 Edgecore Networks. All rights reserved

    | www.edge-core.com DC Core Data Center Clos Fabric Cloud Service Providers Telecom Service Providers Enterprise & Campus DC Edge PE Edge
  7. Underlay Network Evolution for Data Center © 2017 Edgecore Networks.

    All rights reserved | www.edge-core.com Three-Tier Architecture Ref: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ Leaf-Spine Architecture 1 3 2 4 Facebook Fabric
  8. Overlay Network Evolution for Data Center © 2017 Edgecore Networks.

    All rights reserved | www.edge-core.com Ref: https://www.linkedin.com/pulse/network-virtualization-101-nve-overlay-sdn-dhiman-chowdhury
  9. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a

    theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Reproduce Fix Report
  10. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a

    theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Reproduce
  11. Reproduce The Problem • Collect useful information • Identify symptoms

    • Divide and Conquer • Right question help fault seeking © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  12. Truth Table P Q P→Q ~P->~Q F F T T

    F T T F T F F T T T T T © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://youtu.be/hyaA__bLzSQ?list=PLj6E8qlqmkFuqJuqa8y7i2mh3KjetbexW
  13. Truth Table Example • 若網路線沒插 -> Ping 不到 • P

    → Q • 若 Ping 的到,則網路線有插 • ~Q → ~P ≡ P → Q • 若 Ping 不到,則網路線沒插 • Q → P ≠ P → Q • 若網路線沒插且網路設定錯誤,則 Ping 不到 • (P1 ∧ P2) → Q • 若 Ping 不到,則網路線沒插或網路設定錯誤 • ~Q -> (~P1 ∨ ~P2) ≡ (P1 ∧ P2) → Q • Too many interfering factor © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  14. Bayes’s Theorem • The probability of an event, based on

    prior knowledge of conditions that might be related to the event © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://en.wikipedia.org/wiki/Bayes%27_theorem
  15. Normal Distribution © 2017 Edgecore Networks. All rights reserved |

    www.edge-core.com Ref: https://en.wikipedia.org/wiki/Normal_distribution
  16. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a

    theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Report
  17. 8D Report • 8 Disciplines Problem Solving • Problem-Solving process

    with eight objective • Team-Oriented • Follow the logic of the PDCA (Plan, Do, Check, Act) cycle • Procedure • D0: Plan • D1: Form the team • D2: Describe the problem • D3: Develop Interim Containment Plan • D4: Determine and Verify Root Causes and Escape Points: • D5: Verify Permanent Corrections (PCs) for Problem will resolve problem for the customer • D6: Define and Implement Corrective Actions • D7: Prevent Recurrence / System Problems • D8: Congratulate Your Team © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  18. Network Troubleshooting Methodology 1. Identify the problem 2. Establish a

    theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Fix
  19. OSI Model v.s TCP/IP Model © 2017 Edgecore Networks. All

    rights reserved | www.edge-core.com Physical Data link Network Transport Session Presentation Application Application Transport Internet Network Interface IP Protocol BGP, OSPF, RIP ICMP TCP UDP ARP Internet Driver Tunnel Protocols (GRE, MPLS, L2TP) SSL / TLS DNS HTTP DHCP RPC LDAP SSH TELNET NFS
  20. Top-Down and Bottom-Up Approach © 2017 Edgecore Networks. All rights

    reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS Bottom-Up Top-Down
  21. Divide and Conquer Approach © 2017 Edgecore Networks. All rights

    reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS
  22. OpenStack Network Troubleshooting © 2017 Edgecore Networks. All rights reserved

    | www.edge-core.com Ref: https://docs.openstack.org/ops-guide/ops-network-troubleshooting.html
  23. Ping Command • ping www.edge-core.com • ping 8.8.8.8 © 2017

    Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://www.thegeekstuff.com/2009/11/ping-tutorial-13-effective-ping-command-examples/ 28 bytes = IP Header (20 bytes)+ ICMP Header (8 bytes)
  24. Action: Ping Domain Name © 2017 Edgecore Networks. All rights

    reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com
  25. Action: Ping DNS IP © 2017 Edgecore Networks. All rights

    reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com
  26. Action: Ping Gateway © 2017 Edgecore Networks. All rights reserved

    | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com
  27. Action: Ping Host in Same LAN © 2017 Edgecore Networks.

    All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com ping 192.168.100.12
  28. My Traceroute (mtr) Command • Combines the functionally of the

    traceroute and ping programs © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  29. iptables • Total of 4 Tables • raw > mangle

    > nat > filter (default table) • Total of 5 Chains • PREROUTING, INPUT, FORWARDING, OUTPUT, POSTROUTING © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  30. © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref:

    http://inai.de/images/nf-packet-flow.png • Network Layer • Iptables, ip6tables • Link Layer • ebtables
  31. NAT Internal Network to Internet • iptables -t nat -A

    POSTROUTING -o eth0 -j MASQUERADE © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  32. Port Forwarding • iptables -t nat -A PREROUTING -p tcp

    -i eth0 --dport 3389 -j DNAT --to-destination <TARGET_IP>:3389 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  33. Drop ICMP Packet • iptables -t filter -A FORWARD -p

    ICMP -j DROP © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  34. ip link command • Command • ip link show •

    ip link set dev eth0 up • ip link set dev eth0 mtu 9000 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Link management
  35. ip addr command • Command • ip addr show •

    ip addr add 192.168.101.12/24 dev eth0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Address management
  36. ip route command • Management connected routes • Command •

    ip route add default via 192.168.100.254 dev eth0 • ip route add 172.17.0.0/16 dev docker0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Route management
  37. ip monitor command • Network event monitoring • Command •

    ip monitor all © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Network event monitoring
  38. Advanced Command • ip tuntap • Allow userspace programs to

    emulate a network device • TUN sends and receives IP packets (L3) • TAP sends and receives Ethernet frames (L2) • ip netns • ip rule © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  39. dig command • Use dig command for DNS lookup and

    query DNS name servers for various resource record • Command • dig edge-core.com ANY +noall +answer • dig @8.8.8.8 www.edge-core.com • dig –x 140.113.235.1 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/
  40. lsof command • List open files • Command • lsof

    -i:80 • lsof /mnt/usb © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
  41. 3/16/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    41 Open Networking from Freedom Control Innovation