Can't ping my VM!

Talk about debugging common problems in Neutron

Transcript

1. I can't ping my VM! Rossella Sblendido Neutron Core Reviewer SUSE rsblendido@suse.com
2. 2 Can't reach my VM... Rebooting didn't fix it Can't find any solution online I will wipe eveything out!
3. 3 Breathe! • You will be able to fix most problems in 40 minutes from now :)
4. 4 Networking problems... • Misconfiguration ‒ Wrong config entries ‒ Underlying network infrastructure has problems/is not configured correctly (e.g. external network not reachable, some firewall rule is filtering the traffic) • Bug in the code
5. Can't ping/ssh VM private IP
6. 6 Problem: I can't ping a VM (private IP) • Easy first checks: ‒ Is the VM up and running? Nova list and if not check nova logs ‒ Grep TRACE in /var/log/nova/* and /var/log/neutron/* ‒ Check that security group allows ICMP ‒ Check that all the nodes can reach one another Let's dig further!
7. 7 How does a VM get an IP?
8. 8 DHCP agent • Provides DHCP services • Communication over RPC • Isolation through namespaces • dnsmasq
9. 9 IP address allocation
10. 10 OpenVSwitch
11. 11 Linuxbridge
12. 12 Digging further ‒ Did the VM receive an IP ? ‒ If not is the DHCP agent running? ‒ Is dnsmasq running in the dhcp namespace and is the lease file correctly filled? ‒ If using VLANs is the underlay switch allowing the VLAN ID used for the network? ‒ Tcpdump all the way to see where the packet gets lost
13. VM can't reach external world
14. 14 L3 Agent • Provides L3/NAT • Runs network node • Uses namespaces • External network access
15. 15 OpenVSwitch
16. 16 LinuxBridge
17. 17 • Is the instances on a network with a router to the external network? • Can the VM ping the router? • Is br-ext configured correctly? Ovs vsctl show • Can you reach the external world from the router namespace? • If using VLAN, is the switch allowing these VLAN IDs? Things to check
18. Can't ping/ssh VM floating IP
19. 19 OpenVSwitch
20. 20 LinuxBridge
21. 21 Things to check... • Did you configure security group properly? Ping and ssh need to be allowed explicitly • Pinging the private IP is working? • Can the VM ping the router? • Can you ping the VM from the router namespace using the floating IP? • ovs-vsctl show • tcpdump
22. VM can't reach the Metadata server
23. 23 Metadata Agent • Proxies Metadata requests to Nova • Routed Networks ‒ Process embedded in router • Non-routed Networks ‒ Static routes redirect traffic running in the DHCP namespace
24. 24 Routed networks Iptables redirects Metadata traffic to the proxy
25. 25 Isolated networks Option 121 is used to inject a route in the VM to set the Metadata proxy as next hop to reach the MD server
26. 26 Things to check • Is the Metadata agent up? • Is the Metadata proxy up? • Look at the logs of neutron-metadata-agent and nova- metadata-agent • Is the Metadata server reachable from the router namespace or the DHCP namespace? • Does the image you are using suppot Option121? • Tcpdump all the way
27. Vif plugging timeout
28. 28 L2 Agent • Runs on hypervisor (compute node) • Configure the local vswitch • Communicates with the server over RPC • Wires new devices • Security Group Rules
29. 29 Vif plugging
30. 30 • Check the neutron server and the l2 agent logs • If the system is loaded adjust vif_plugging_timeout, rpc_thread_pool_size and rpc_conn_pool_size Things to check
31. Tools
32. 32 General info • ip addr • route -n • iptables -L • arp
33. 33 tcpdump • tcpdump -i eth0 -n ip proto gre • tcpdump -envi br-int • ip netns exec qrouter-UUID tcpdump -i qr-63ea2815- b5 icmp • tcpdump -i eth0 -n arp or icmp • tcpdump -i any -n icmp
34. 34 Namespaces • ip netns list • ip netns exec qrouter-UUID ip link • ip netns exec qrouter-UUID route -n • ip netns exec qrouter-UUID iptables -L -t nat
35. 35 OpenVSwitch • ovs-vsctl show • ovs-dpctl show • ovs-dpctl dump-flows • ovs-ofctl dump-flows br-tun • ovs-ofctl dump-flows br-tun table=21
36. 36 LinuxBridge • brctl show • brctl show bridgename
37. 37 Useful links • http://docs.openstack.org/admin-guide- cloud/content/under_the_hood_openvswitch.html • https://www.rdoproject.org/Networking_in_too_much_ detail • https://kashyapc.fedorapeople.org/virt/openstack/neutr on/neutron-diagnostics.html • http://techbackground.blogspot.ie/2013/06/metadata- via-dhcp-namespace.html
38. Thank you. 38 Don't forget to give feedback :) Questions?
40. Unpublished Work of SUSE. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.