Network Troubleshooting Methodology

Slide 1

Slide 1 text

Network Troubleshooting Methodology Phil Huang SDN Solution Engineer, Open Networking Division

Slide 2

Slide 2 text

About Me • Phil Huang • Edgecore SDN Solution Engineer • Big Switch Networks • Cumulus Networks • Pica8 • Open Networking Linux & OF-DPA • ON.Lab ONOS/CORD Ambassdor • blog.pichuang.com.tw © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Open Networking Evolution © 2017 Edgecore Networks. All rights reserved | www.edge-core.com DC Core Data Center Clos Fabric Cloud Service Providers Telecom Service Providers Enterprise & Campus DC Edge PE Edge

Slide 7

Slide 7 text

Underlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Three-Tier Architecture Ref: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ Leaf-Spine Architecture 1 3 2 4 Facebook Fabric

Slide 8

Slide 8 text

Overlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://www.linkedin.com/pulse/network-virtualization-101-nve-overlay-sdn-dhiman-chowdhury

Slide 9

Slide 9 text

Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Reproduce Fix Report

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Truth Table Example • 若網路線沒插 -> Ping 不到 • P → Q • 若 Ping 的到，則網路線有插 • ~Q → ~P ≡ P → Q • 若 Ping 不到，則網路線沒插 • Q → P ≠ P → Q • 若網路線沒插且網路設定錯誤，則 Ping 不到 • (P1 ∧ P2) → Q • 若 Ping 不到，則網路線沒插或網路設定錯誤 • ~Q -> (~P1 ∨ ~P2) ≡ (P1 ∧ P2) → Q • Too many interfering factor © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 14

Slide 14 text

Bayes’s Theorem • The probability of an event, based on prior knowledge of conditions that might be related to the event © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://en.wikipedia.org/wiki/Bayes%27_theorem

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

8D Report • 8 Disciplines Problem Solving • Problem-Solving process with eight objective • Team-Oriented • Follow the logic of the PDCA (Plan, Do, Check, Act) cycle • Procedure • D0: Plan • D1: Form the team • D2: Describe the problem • D3: Develop Interim Containment Plan • D4: Determine and Verify Root Causes and Escape Points: • D5: Verify Permanent Corrections (PCs) for Problem will resolve problem for the customer • D6: Define and Implement Corrective Actions • D7: Prevent Recurrence / System Problems • D8: Congratulate Your Team © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 18

Slide 18 text

Slide 19

Slide 19 text

OSI Model v.s TCP/IP Model © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Physical Data link Network Transport Session Presentation Application Application Transport Internet Network Interface IP Protocol BGP, OSPF, RIP ICMP TCP UDP ARP Internet Driver Tunnel Protocols (GRE, MPLS, L2TP) SSL / TLS DNS HTTP DHCP RPC LDAP SSH TELNET NFS

Slide 20

Slide 20 text

Top-Down and Bottom-Up Approach © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS Bottom-Up Top-Down

Slide 21

Slide 21 text

Divide and Conquer Approach © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Ping Command • ping www.edge-core.com • ping 8.8.8.8 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://www.thegeekstuff.com/2009/11/ping-tutorial-13-effective-ping-command-examples/ 28 bytes = IP Header (20 bytes)+ ICMP Header (8 bytes)

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Action: Ping Host in Same LAN © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com ping 192.168.100.12

Slide 28

Slide 28 text

Slide 29

Slide 29 text

iptables • Total of 4 Tables • raw > mangle > nat > filter (default table) • Total of 5 Chains • PREROUTING, INPUT, FORWARDING, OUTPUT, POSTROUTING © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

ip link command • Command • ip link show • ip link set dev eth0 up • ip link set dev eth0 mtu 9000 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Link management

Slide 35

Slide 35 text

ip addr command • Command • ip addr show • ip addr add 192.168.101.12/24 dev eth0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Address management

Slide 36

Slide 36 text

ip route command • Management connected routes • Command • ip route add default via 192.168.100.254 dev eth0 • ip route add 172.17.0.0/16 dev docker0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Route management

Slide 37

Slide 37 text

ip monitor command • Network event monitoring • Command • ip monitor all © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Network event monitoring

Slide 38

Slide 38 text

Advanced Command • ip tuntap • Allow userspace programs to emulate a network device • TUN sends and receives IP packets (L3) • TAP sends and receives Ethernet frames (L2) • ip netns • ip rule © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 39

Slide 39 text

dig command • Use dig command for DNS lookup and query DNS name servers for various resource record • Command • dig edge-core.com ANY +noall +answer • dig @8.8.8.8 www.edge-core.com • dig –x 140.113.235.1 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/