$30 off During Our Annual Pro Sale. View Details »

Network Troubleshooting Methodology

Network Troubleshooting Methodology

Understanding and applying network troubleshooting methodology
#edgecore

Phil Huang

March 16, 2017
Tweet

More Decks by Phil Huang

Other Decks in Technology

Transcript

  1. Network Troubleshooting Methodology
    Phil Huang
    SDN Solution Engineer, Open Networking Division

    View Slide

  2. About Me
    • Phil Huang
    • Edgecore SDN Solution Engineer
    • Big Switch Networks
    • Cumulus Networks
    • Pica8
    • Open Networking Linux & OF-DPA
    • ON.Lab ONOS/CORD Ambassdor
    • blog.pichuang.com.tw
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  3. Facebook Datacenter
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://www.zdnet.com/pictures/facebooks-data-centers-worldwide-by-the-numbers-and-in-pictures/

    View Slide

  4. Google Datacenter
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://datacentermurals.withgoogle.com/

    View Slide

  5. Amazon Datacenter
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://datacenterfrontier.com/inside-amazon-cloud-computing-infrastructure/

    View Slide

  6. Open Networking Evolution
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    DC Core
    Data Center Clos Fabric
    Cloud Service Providers
    Telecom Service Providers
    Enterprise & Campus
    DC Edge
    PE Edge

    View Slide

  7. Underlay Network Evolution for Data Center
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Three-Tier Architecture
    Ref: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/
    Leaf-Spine Architecture
    1 3
    2 4
    Facebook Fabric

    View Slide

  8. Overlay Network Evolution for Data Center
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://www.linkedin.com/pulse/network-virtualization-101-nve-overlay-sdn-dhiman-chowdhury

    View Slide

  9. Network Troubleshooting Methodology
    1. Identify the problem
    2. Establish a theory of probable cause
    3. Test a theory of probable cause
    4. Establish a plan of action and identify potential effects
    5. Implement the plan or escalate
    6. Verify full system functionally
    7. Document findings, actions, and outcomes
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: CompTIA Network+ Troubleshooting Model
    Reproduce
    Fix
    Report

    View Slide

  10. Network Troubleshooting Methodology
    1. Identify the problem
    2. Establish a theory of probable cause
    3. Test a theory of probable cause
    4. Establish a plan of action and identify potential effects
    5. Implement the plan or escalate
    6. Verify full system functionally
    7. Document findings, actions, and outcomes
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: CompTIA Network+ Troubleshooting Model
    Reproduce

    View Slide

  11. Reproduce The Problem
    • Collect useful information
    • Identify symptoms
    • Divide and Conquer
    • Right question help fault seeking
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  12. Truth Table
    P Q P→Q ~P->~Q
    F F T T
    F T T F
    T F F T
    T T T T
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://youtu.be/hyaA__bLzSQ?list=PLj6E8qlqmkFuqJuqa8y7i2mh3KjetbexW

    View Slide

  13. Truth Table Example
    • 若網路線沒插 -> Ping 不到
    • P → Q
    • 若 Ping 的到,則網路線有插
    • ~Q → ~P ≡ P → Q
    • 若 Ping 不到,則網路線沒插
    • Q → P ≠ P → Q
    • 若網路線沒插且網路設定錯誤,則 Ping 不到
    • (P1 ∧ P2) → Q
    • 若 Ping 不到,則網路線沒插或網路設定錯誤
    • ~Q -> (~P1 ∨ ~P2) ≡ (P1 ∧ P2) → Q
    • Too many interfering factor
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  14. Bayes’s Theorem
    • The probability of an event, based on prior knowledge of conditions
    that might be related to the event
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://en.wikipedia.org/wiki/Bayes%27_theorem

    View Slide

  15. Normal Distribution
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://en.wikipedia.org/wiki/Normal_distribution

    View Slide

  16. Network Troubleshooting Methodology
    1. Identify the problem
    2. Establish a theory of probable cause
    3. Test a theory of probable cause
    4. Establish a plan of action and identify potential effects
    5. Implement the plan or escalate
    6. Verify full system functionally
    7. Document findings, actions, and outcomes
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: CompTIA Network+ Troubleshooting Model
    Report

    View Slide

  17. 8D Report
    • 8 Disciplines Problem Solving
    • Problem-Solving process with eight objective
    • Team-Oriented
    • Follow the logic of the PDCA (Plan, Do, Check, Act) cycle
    • Procedure
    • D0: Plan
    • D1: Form the team
    • D2: Describe the problem
    • D3: Develop Interim Containment Plan
    • D4: Determine and Verify Root Causes and Escape Points:
    • D5: Verify Permanent Corrections (PCs) for Problem will resolve problem for the customer
    • D6: Define and Implement Corrective Actions
    • D7: Prevent Recurrence / System Problems
    • D8: Congratulate Your Team
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  18. Network Troubleshooting Methodology
    1. Identify the problem
    2. Establish a theory of probable cause
    3. Test a theory of probable cause
    4. Establish a plan of action and identify potential effects
    5. Implement the plan or escalate
    6. Verify full system functionally
    7. Document findings, actions, and outcomes
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: CompTIA Network+ Troubleshooting Model
    Fix

    View Slide

  19. OSI Model v.s TCP/IP Model
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Physical
    Data link
    Network
    Transport
    Session
    Presentation
    Application
    Application
    Transport
    Internet
    Network
    Interface
    IP Protocol BGP, OSPF, RIP ICMP
    TCP UDP
    ARP Internet Driver
    Tunnel Protocols
    (GRE, MPLS, L2TP)
    SSL / TLS
    DNS HTTP DHCP
    RPC LDAP
    SSH TELNET NFS

    View Slide

  20. Top-Down and Bottom-Up Approach
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Application
    Transport
    Internet
    Network
    Interface
    ARP
    IP Protocol BGP, OSPF, RIP
    Internet Driver
    ICMP
    TCP UDP
    SSL / TLS
    DNS HTTP DHCP
    Tunnel Protocols
    (GRE, MPLS, L2TP)
    RPC LDAP
    SSH TELNET NFS
    Bottom-Up Top-Down

    View Slide

  21. Divide and Conquer Approach
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Application
    Transport
    Internet
    Network
    Interface
    ARP
    IP Protocol BGP, OSPF, RIP
    Internet Driver
    ICMP
    TCP UDP
    SSL / TLS
    DNS HTTP DHCP
    Tunnel Protocols
    (GRE, MPLS, L2TP)
    RPC LDAP
    SSH TELNET NFS

    View Slide

  22. OpenStack Network Troubleshooting
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://docs.openstack.org/ops-guide/ops-network-troubleshooting.html

    View Slide

  23. Ping Command
    • ping www.edge-core.com
    • ping 8.8.8.8
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://www.thegeekstuff.com/2009/11/ping-tutorial-13-effective-ping-command-examples/
    28 bytes = IP Header (20 bytes)+ ICMP Header (8 bytes)

    View Slide

  24. Action: Ping Domain Name
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com

    View Slide

  25. Action: Ping DNS IP
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com

    View Slide

  26. Action: Ping Gateway
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com

    View Slide

  27. Action: Ping Host in Same LAN
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Host 1
    100.10
    Host 2
    100.12
    192.168.100.0/24
    192.168.100.254
    Google DNS
    8.8.8.8
    Edgecore
    www.edge-core.com
    ping 192.168.100.12

    View Slide

  28. My Traceroute (mtr) Command
    • Combines the functionally of the traceroute and ping programs
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  29. iptables
    • Total of 4 Tables
    • raw > mangle > nat > filter (default table)
    • Total of 5 Chains
    • PREROUTING, INPUT, FORWARDING, OUTPUT, POSTROUTING
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  30. © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://inai.de/images/nf-packet-flow.png
    • Network Layer
    • Iptables, ip6tables
    • Link Layer
    • ebtables

    View Slide

  31. NAT Internal Network to Internet
    • iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  32. Port Forwarding
    • iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 3389 -j
    DNAT --to-destination :3389
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  33. Drop ICMP Packet
    • iptables -t filter -A FORWARD -p ICMP -j DROP
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  34. ip link command
    • Command
    • ip link show
    • ip link set dev eth0 up
    • ip link set dev eth0 mtu 9000
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Link management

    View Slide

  35. ip addr command
    • Command
    • ip addr show
    • ip addr add 192.168.101.12/24 dev eth0
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Address management

    View Slide

  36. ip route command
    • Management connected routes
    • Command
    • ip route add default via 192.168.100.254 dev eth0
    • ip route add 172.17.0.0/16 dev docker0
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Route management

    View Slide

  37. ip monitor command
    • Network event monitoring
    • Command
    • ip monitor all
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: http://baturin.org/docs/iproute2/#Network event monitoring

    View Slide

  38. Advanced Command
    • ip tuntap
    • Allow userspace programs to emulate a network device
    • TUN sends and receives IP packets (L3)
    • TAP sends and receives Ethernet frames (L2)
    • ip netns
    • ip rule
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  39. dig command
    • Use dig command for DNS lookup and query DNS name servers for
    various resource record
    • Command
    • dig edge-core.com ANY +noall +answer
    • dig @8.8.8.8 www.edge-core.com
    • dig –x 140.113.235.1
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    Ref: https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/

    View Slide

  40. lsof command
    • List open files
    • Command
    • lsof -i:80
    • lsof /mnt/usb
    © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

    View Slide

  41. 3/16/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com
    41
    Open Networking
    from
    Freedom
    Control
    Innovation

    View Slide