Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Can't ping my VM!

Can't ping my VM!

Talk about debugging common problems in Neutron

Rossella Sblendido

November 18, 2015
Tweet

More Decks by Rossella Sblendido

Other Decks in Programming

Transcript

  1. 2 Can't reach my VM... Rebooting didn't fix it Can't

    find any solution online I will wipe eveything out!
  2. 3 Breathe! • You will be able to fix most

    problems in 40 minutes from now :)
  3. 4 Networking problems... • Misconfiguration ‒ Wrong config entries ‒

    Underlying network infrastructure has problems/is not configured correctly (e.g. external network not reachable, some firewall rule is filtering the traffic) • Bug in the code
  4. 6 Problem: I can't ping a VM (private IP) •

    Easy first checks: ‒ Is the VM up and running? Nova list and if not check nova logs ‒ Grep TRACE in /var/log/nova/* and /var/log/neutron/* ‒ Check that security group allows ICMP ‒ Check that all the nodes can reach one another Let's dig further!
  5. 8 DHCP agent • Provides DHCP services • Communication over

    RPC • Isolation through namespaces • dnsmasq
  6. 12 Digging further ‒ Did the VM receive an IP

    ? ‒ If not is the DHCP agent running? ‒ Is dnsmasq running in the dhcp namespace and is the lease file correctly filled? ‒ If using VLANs is the underlay switch allowing the VLAN ID used for the network? ‒ Tcpdump all the way to see where the packet gets lost
  7. 14 L3 Agent • Provides L3/NAT • Runs network node

    • Uses namespaces • External network access
  8. 17 • Is the instances on a network with a

    router to the external network? • Can the VM ping the router? • Is br-ext configured correctly? Ovs vsctl show • Can you reach the external world from the router namespace? • If using VLAN, is the switch allowing these VLAN IDs? Things to check
  9. 21 Things to check... • Did you configure security group

    properly? Ping and ssh need to be allowed explicitly • Pinging the private IP is working? • Can the VM ping the router? • Can you ping the VM from the router namespace using the floating IP? • ovs-vsctl show • tcpdump
  10. 23 Metadata Agent • Proxies Metadata requests to Nova •

    Routed Networks ‒ Process embedded in router • Non-routed Networks ‒ Static routes redirect traffic running in the DHCP namespace
  11. 25 Isolated networks Option 121 is used to inject a

    route in the VM to set the Metadata proxy as next hop to reach the MD server
  12. 26 Things to check • Is the Metadata agent up?

    • Is the Metadata proxy up? • Look at the logs of neutron-metadata-agent and nova- metadata-agent • Is the Metadata server reachable from the router namespace or the DHCP namespace? • Does the image you are using suppot Option121? • Tcpdump all the way
  13. 28 L2 Agent • Runs on hypervisor (compute node) •

    Configure the local vswitch • Communicates with the server over RPC • Wires new devices • Security Group Rules
  14. 30 • Check the neutron server and the l2 agent

    logs • If the system is loaded adjust vif_plugging_timeout, rpc_thread_pool_size and rpc_conn_pool_size Things to check
  15. 33 tcpdump • tcpdump -i eth0 -n ip proto gre

    • tcpdump -envi br-int • ip netns exec qrouter-UUID tcpdump -i qr-63ea2815- b5 icmp • tcpdump -i eth0 -n arp or icmp • tcpdump -i any -n icmp
  16. 34 Namespaces • ip netns list • ip netns exec

    qrouter-UUID ip link • ip netns exec qrouter-UUID route -n • ip netns exec qrouter-UUID iptables -L -t nat
  17. 35 OpenVSwitch • ovs-vsctl show • ovs-dpctl show • ovs-dpctl

    dump-flows • ovs-ofctl dump-flows br-tun • ovs-ofctl dump-flows br-tun table=21
  18. 37 Useful links • http://docs.openstack.org/admin-guide- cloud/content/under_the_hood_openvswitch.html • https://www.rdoproject.org/Networking_in_too_much_ detail •

    https://kashyapc.fedorapeople.org/virt/openstack/neutr on/neutron-diagnostics.html • http://techbackground.blogspot.ie/2013/06/metadata- via-dhcp-namespace.html
  19. Unpublished Work of SUSE. All Rights Reserved. This work is

    an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.