Network Troubleshooting Methodology

Understanding and applying network troubleshooting methodology
#edgecore

Transcript

1. Network Troubleshooting Methodology
   Phil Huang
   SDN Solution Engineer, Open Networking Division
   

   View Slide

2. About Me
   • Phil Huang
   • Edgecore SDN Solution Engineer
   • Big Switch Networks
   • Cumulus Networks
   • Pica8
   • Open Networking Linux & OF-DPA
   • ON.Lab ONOS/CORD Ambassdor
   • blog.pichuang.com.tw
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   

   View Slide

3. Facebook Datacenter
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   Ref: http://www.zdnet.com/pictures/facebooks-data-centers-worldwide-by-the-numbers-and-in-pictures/
   

   View Slide

4. Google Datacenter
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   Ref: https://datacentermurals.withgoogle.com/
   

   View Slide

5. Amazon Datacenter
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   Ref: http://datacenterfrontier.com/inside-amazon-cloud-computing-infrastructure/
   

   View Slide

6. Open Networking Evolution
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   DC Core
   Data Center Clos Fabric
   Cloud Service Providers
   Telecom Service Providers
   Enterprise & Campus
   DC Edge
   PE Edge
   

   View Slide

7. Underlay Network Evolution for Data Center
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   Three-Tier Architecture
   Ref: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/
   Leaf-Spine Architecture
   1 3
   2 4
   Facebook Fabric
   

   View Slide

8. Overlay Network Evolution for Data Center
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   Ref: https://www.linkedin.com/pulse/network-virtualization-101-nve-overlay-sdn-dhiman-chowdhury
   

   View Slide

9. Network Troubleshooting Methodology
   1. Identify the problem
   2. Establish a theory of probable cause
   3. Test a theory of probable cause
   4. Establish a plan of action and identify potential effects
   5. Implement the plan or escalate
   6. Verify full system functionally
   7. Document findings, actions, and outcomes
   © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
   Ref: CompTIA Network+ Troubleshooting Model
   Reproduce
   Fix
   Report
   

   View Slide

10. Network Troubleshooting Methodology
    1. Identify the problem
    2. Establish a theory of probable cause
    3. Test a theory of probable cause
    4. Establish a plan of action and identify potential effects
    5. Implement the plan or escalate
    6. Verify full system functionally
    7. Document findings, actions, and outcomes
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: CompTIA Network+ Troubleshooting Model
    Reproduce
    

    View Slide

11. Reproduce The Problem
    • Collect useful information
    • Identify symptoms
    • Divide and Conquer
    • Right question help fault seeking
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

12. Truth Table
    P Q P→Q ~P->~Q
    F F T T
    F T T F
    T F F T
    T T T T
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://youtu.be/hyaA__bLzSQ?list=PLj6E8qlqmkFuqJuqa8y7i2mh3KjetbexW
    

    View Slide

13. Truth Table Example
    • 若網路線沒插 -> Ping 不到
    • P → Q
    • 若 Ping 的到，則網路線有插
    • ~Q → ~P ≡ P → Q
    • 若 Ping 不到，則網路線沒插
    • Q → P ≠ P → Q
    • 若網路線沒插且網路設定錯誤，則 Ping 不到
    • (P1 ∧ P2) → Q
    • 若 Ping 不到，則網路線沒插或網路設定錯誤
    • ~Q -> (~P1 ∨ ~P2) ≡ (P1 ∧ P2) → Q
    • Too many interfering factor
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

14. Bayes’s Theorem
    • The probability of an event, based on prior knowledge of conditions
    that might be related to the event
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://en.wikipedia.org/wiki/Bayes%27_theorem
    

    View Slide

15. Normal Distribution
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://en.wikipedia.org/wiki/Normal_distribution
    

    View Slide

16. Network Troubleshooting Methodology
    1. Identify the problem
    2. Establish a theory of probable cause
    3. Test a theory of probable cause
    4. Establish a plan of action and identify potential effects
    5. Implement the plan or escalate
    6. Verify full system functionally
    7. Document findings, actions, and outcomes
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: CompTIA Network+ Troubleshooting Model
    Report
    

    View Slide

17. 8D Report
    • 8 Disciplines Problem Solving
    • Problem-Solving process with eight objective
    • Team-Oriented
    • Follow the logic of the PDCA (Plan, Do, Check, Act) cycle
    • Procedure
    • D0: Plan
    • D1: Form the team
    • D2: Describe the problem
    • D3: Develop Interim Containment Plan
    • D4: Determine and Verify Root Causes and Escape Points:
    • D5: Verify Permanent Corrections (PCs) for Problem will resolve problem for the customer
    • D6: Define and Implement Corrective Actions
    • D7: Prevent Recurrence / System Problems
    • D8: Congratulate Your Team
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

18. Network Troubleshooting Methodology
    1. Identify the problem
    2. Establish a theory of probable cause
    3. Test a theory of probable cause
    4. Establish a plan of action and identify potential effects
    5. Implement the plan or escalate
    6. Verify full system functionally
    7. Document findings, actions, and outcomes
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: CompTIA Network+ Troubleshooting Model
    Fix
    

    View Slide

19. OSI Model v.s TCP/IP Model
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Physical
    Data link
    Network
    Transport
    Session
    Presentation
    Application
    Application
    Transport
    Internet
    Network
    Interface
    IP Protocol BGP, OSPF, RIP ICMP
    TCP UDP
    ARP Internet Driver
    Tunnel Protocols
    (GRE, MPLS, L2TP)
    SSL / TLS
    DNS HTTP DHCP
    RPC LDAP
    SSH TELNET NFS
    

    View Slide

20. Top-Down and Bottom-Up Approach
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Application
    Transport
    Internet
    Network
    Interface
    ARP
    IP Protocol BGP, OSPF, RIP
    Internet Driver
    ICMP
    TCP UDP
    SSL / TLS
    DNS HTTP DHCP
    Tunnel Protocols
    (GRE, MPLS, L2TP)
    RPC LDAP
    SSH TELNET NFS
    Bottom-Up Top-Down
    

    View Slide

21. Divide and Conquer Approach
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Application
    Transport
    Internet
    Network
    Interface
    ARP
    IP Protocol BGP, OSPF, RIP
    Internet Driver
    ICMP
    TCP UDP
    SSL / TLS
    DNS HTTP DHCP
    Tunnel Protocols
    (GRE, MPLS, L2TP)
    RPC LDAP
    SSH TELNET NFS
    

    View Slide

22. OpenStack Network Troubleshooting
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://docs.openstack.org/ops-guide/ops-network-troubleshooting.html
    

    View Slide

23. Ping Command
    • ping www.edge-core.com
    • ping 8.8.8.8
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://www.thegeekstuff.com/2009/11/ping-tutorial-13-effective-ping-command-examples/
    28 bytes = IP Header (20 bytes)+ ICMP Header (8 bytes)
    

    View Slide

24. Action: Ping Domain Name
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com
    

    View Slide

25. Action: Ping DNS IP
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com
    

    View Slide

26. Action: Ping Gateway
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com
    

    View Slide

27. Action: Ping Host in Same LAN
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com
    ping 192.168.100.12
    

    View Slide

28. My Traceroute (mtr) Command
    • Combines the functionally of the traceroute and ping programs
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

29. iptables
    • Total of 4 Tables
    • raw > mangle > nat > filter (default table)
    • Total of 5 Chains
    • PREROUTING, INPUT, FORWARDING, OUTPUT, POSTROUTING
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

30. © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://inai.de/images/nf-packet-flow.png
    • Network Layer
    • Iptables, ip6tables
    • Link Layer
    • ebtables
    

    View Slide

31. NAT Internal Network to Internet
    • iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

32. Port Forwarding
    • iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 3389 -j
    DNAT --to-destination :3389
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

33. Drop ICMP Packet
    • iptables -t filter -A FORWARD -p ICMP -j DROP
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

34. ip link command
    • Command
    • ip link show
    • ip link set dev eth0 up
    • ip link set dev eth0 mtu 9000
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Link management
    

    View Slide

35. ip addr command
    • Command
    • ip addr show
    • ip addr add 192.168.101.12/24 dev eth0
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Address management
    

    View Slide

36. ip route command
    • Management connected routes
    • Command
    • ip route add default via 192.168.100.254 dev eth0
    • ip route add 172.17.0.0/16 dev docker0
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Route management
    

    View Slide

37. ip monitor command
    • Network event monitoring
    • Command
    • ip monitor all
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Network event monitoring
    

    View Slide

38. Advanced Command
    • ip tuntap
    • Allow userspace programs to emulate a network device
    • TUN sends and receives IP packets (L3)
    • TAP sends and receives Ethernet frames (L2)
    • ip netns
    • ip rule
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

39. dig command
    • Use dig command for DNS lookup and query DNS name servers for
    various resource record
    • Command
    • dig edge-core.com ANY +noall +answer
    • dig @8.8.8.8 www.edge-core.com
    • dig –x 140.113.235.1
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/
    

    View Slide

40. lsof command
    • List open files
    • Command
    • lsof -i:80
    • lsof /mnt/usb
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    

    View Slide

41. 3/16/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    41
    Open Networking
    from
    Freedom
    Control
    Innovation
    

    View Slide