Slide 1

Slide 1 text

Network Troubleshooting Methodology Phil Huang SDN Solution Engineer, Open Networking Division

Slide 2

Slide 2 text

About Me • Phil Huang • Edgecore SDN Solution Engineer • Big Switch Networks • Cumulus Networks • Pica8 • Open Networking Linux & OF-DPA • ON.Lab ONOS/CORD Ambassdor • blog.pichuang.com.tw © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 3

Slide 3 text

Facebook Datacenter © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://www.zdnet.com/pictures/facebooks-data-centers-worldwide-by-the-numbers-and-in-pictures/

Slide 4

Slide 4 text

Google Datacenter © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://datacentermurals.withgoogle.com/

Slide 5

Slide 5 text

Amazon Datacenter © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://datacenterfrontier.com/inside-amazon-cloud-computing-infrastructure/

Slide 6

Slide 6 text

Open Networking Evolution © 2017 Edgecore Networks. All rights reserved | www.edge-core.com DC Core Data Center Clos Fabric Cloud Service Providers Telecom Service Providers Enterprise & Campus DC Edge PE Edge

Slide 7

Slide 7 text

Underlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Three-Tier Architecture Ref: https://code.facebook.com/posts/360346274145943/introducing-data-center-fabric-the-next-generation-facebook-data-center-network/ Leaf-Spine Architecture 1 3 2 4 Facebook Fabric

Slide 8

Slide 8 text

Overlay Network Evolution for Data Center © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://www.linkedin.com/pulse/network-virtualization-101-nve-overlay-sdn-dhiman-chowdhury

Slide 9

Slide 9 text

Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Reproduce Fix Report

Slide 10

Slide 10 text

Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Reproduce

Slide 11

Slide 11 text

Reproduce The Problem • Collect useful information • Identify symptoms • Divide and Conquer • Right question help fault seeking © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 12

Slide 12 text

Truth Table P Q P→Q ~P->~Q F F T T F T T F T F F T T T T T © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://youtu.be/hyaA__bLzSQ?list=PLj6E8qlqmkFuqJuqa8y7i2mh3KjetbexW

Slide 13

Slide 13 text

Truth Table Example • 若網路線沒插 -> Ping 不到 • P → Q • 若 Ping 的到,則網路線有插 • ~Q → ~P ≡ P → Q • 若 Ping 不到,則網路線沒插 • Q → P ≠ P → Q • 若網路線沒插且網路設定錯誤,則 Ping 不到 • (P1 ∧ P2) → Q • 若 Ping 不到,則網路線沒插或網路設定錯誤 • ~Q -> (~P1 ∨ ~P2) ≡ (P1 ∧ P2) → Q • Too many interfering factor © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 14

Slide 14 text

Bayes’s Theorem • The probability of an event, based on prior knowledge of conditions that might be related to the event © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://en.wikipedia.org/wiki/Bayes%27_theorem

Slide 15

Slide 15 text

Normal Distribution © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://en.wikipedia.org/wiki/Normal_distribution

Slide 16

Slide 16 text

Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Report

Slide 17

Slide 17 text

8D Report • 8 Disciplines Problem Solving • Problem-Solving process with eight objective • Team-Oriented • Follow the logic of the PDCA (Plan, Do, Check, Act) cycle • Procedure • D0: Plan • D1: Form the team • D2: Describe the problem • D3: Develop Interim Containment Plan • D4: Determine and Verify Root Causes and Escape Points: • D5: Verify Permanent Corrections (PCs) for Problem will resolve problem for the customer • D6: Define and Implement Corrective Actions • D7: Prevent Recurrence / System Problems • D8: Congratulate Your Team © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 18

Slide 18 text

Network Troubleshooting Methodology 1. Identify the problem 2. Establish a theory of probable cause 3. Test a theory of probable cause 4. Establish a plan of action and identify potential effects 5. Implement the plan or escalate 6. Verify full system functionally 7. Document findings, actions, and outcomes © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: CompTIA Network+ Troubleshooting Model Fix

Slide 19

Slide 19 text

OSI Model v.s TCP/IP Model © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Physical Data link Network Transport Session Presentation Application Application Transport Internet Network Interface IP Protocol BGP, OSPF, RIP ICMP TCP UDP ARP Internet Driver Tunnel Protocols (GRE, MPLS, L2TP) SSL / TLS DNS HTTP DHCP RPC LDAP SSH TELNET NFS

Slide 20

Slide 20 text

Top-Down and Bottom-Up Approach © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS Bottom-Up Top-Down

Slide 21

Slide 21 text

Divide and Conquer Approach © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Application Transport Internet Network Interface ARP IP Protocol BGP, OSPF, RIP Internet Driver ICMP TCP UDP SSL / TLS DNS HTTP DHCP Tunnel Protocols (GRE, MPLS, L2TP) RPC LDAP SSH TELNET NFS

Slide 22

Slide 22 text

OpenStack Network Troubleshooting © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://docs.openstack.org/ops-guide/ops-network-troubleshooting.html

Slide 23

Slide 23 text

Ping Command • ping www.edge-core.com • ping 8.8.8.8 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://www.thegeekstuff.com/2009/11/ping-tutorial-13-effective-ping-command-examples/ 28 bytes = IP Header (20 bytes)+ ICMP Header (8 bytes)

Slide 24

Slide 24 text

Action: Ping Domain Name © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com

Slide 25

Slide 25 text

Action: Ping DNS IP © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com

Slide 26

Slide 26 text

Action: Ping Gateway © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com

Slide 27

Slide 27 text

Action: Ping Host in Same LAN © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Host 1 100.10 Host 2 100.12 192.168.100.0/24 192.168.100.254 Google DNS 8.8.8.8 Edgecore www.edge-core.com ping 192.168.100.12

Slide 28

Slide 28 text

My Traceroute (mtr) Command • Combines the functionally of the traceroute and ping programs © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 29

Slide 29 text

iptables • Total of 4 Tables • raw > mangle > nat > filter (default table) • Total of 5 Chains • PREROUTING, INPUT, FORWARDING, OUTPUT, POSTROUTING © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 30

Slide 30 text

© 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://inai.de/images/nf-packet-flow.png • Network Layer • Iptables, ip6tables • Link Layer • ebtables

Slide 31

Slide 31 text

NAT Internal Network to Internet • iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 32

Slide 32 text

Port Forwarding • iptables -t nat -A PREROUTING -p tcp -i eth0 --dport 3389 -j DNAT --to-destination :3389 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 33

Slide 33 text

Drop ICMP Packet • iptables -t filter -A FORWARD -p ICMP -j DROP © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 34

Slide 34 text

ip link command • Command • ip link show • ip link set dev eth0 up • ip link set dev eth0 mtu 9000 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Link management

Slide 35

Slide 35 text

ip addr command • Command • ip addr show • ip addr add 192.168.101.12/24 dev eth0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Address management

Slide 36

Slide 36 text

ip route command • Management connected routes • Command • ip route add default via 192.168.100.254 dev eth0 • ip route add 172.17.0.0/16 dev docker0 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Route management

Slide 37

Slide 37 text

ip monitor command • Network event monitoring • Command • ip monitor all © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: http://baturin.org/docs/iproute2/#Network event monitoring

Slide 38

Slide 38 text

Advanced Command • ip tuntap • Allow userspace programs to emulate a network device • TUN sends and receives IP packets (L3) • TAP sends and receives Ethernet frames (L2) • ip netns • ip rule © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 39

Slide 39 text

dig command • Use dig command for DNS lookup and query DNS name servers for various resource record • Command • dig edge-core.com ANY +noall +answer • dig @8.8.8.8 www.edge-core.com • dig –x 140.113.235.1 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com Ref: https://www.cyberciti.biz/faq/linux-unix-dig-command-examples-usage-syntax/

Slide 40

Slide 40 text

lsof command • List open files • Command • lsof -i:80 • lsof /mnt/usb © 2017 Edgecore Networks. All rights reserved | www.edge-core.com

Slide 41

Slide 41 text

3/16/17 © 2017 Edgecore Networks. All rights reserved | www.edge-core.com 41 Open Networking from Freedom Control Innovation