Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Open Source Lan Party Gateway - LOADays 2018

Open Source Lan Party Gateway - LOADays 2018

Presentation on how to setup a Lan Party gateway server.

Used components:

Ubuntu/Centos
dnsmasq
bind9
isc-dhcp-server
LXC
openvSwitch
tc
iptables
squid
collectd
graphite
grafana

Avatar for Christophe Vanlancker

Christophe Vanlancker

April 21, 2018
Tweet

More Decks by Christophe Vanlancker

Other Decks in Technology

Transcript

  1. Some History • 80’s: “PCs invade the home” • 90’s:

    – Cheaper networking equipment – Cheaper PCs – Rise of popular games: Quake, Age of Empires, Warcraft, Unreal Tournament, … • 20xx: – Online gaming: League of Legends, World of Warcraft, PUBG, Fortnite, Runscape,...
  2. Advanced • Services – Virtualization/Containerization – Monitoring/Metrics – Game servers

    • HA • Caching • DNS overriding • Link Aggregation • Multiple UPLinks • QoS/Throttling • Network segregation (VLANs)
  3. Used Technology • OS: Centos/Ubuntu • DNS: dnsmasq / Bind

    • DHCP: dnsmasq / ISC DHCP • Containerization: LXC • Firewall: iptables • Traffic Shaping: tc • Forward Proxy: Squid • Metrics: Graphite, Collectd, Grafana • Networking: Open vSwitch
  4. Router Enable routing functionality • On the fly sysctl -w

    net.ipv4.ip_forward=1 or echo 1 > /proc/sys/net/ipv4/ip_forward • Permanent: /etc/sysctl.conf: net.ipv4.ip_forward = 1
  5. Router NAT Makes packets leaving through your uplink use the

    public IP of the router so that replies can return to it. – iptables -t nat -A POSTROUTING -j MASQUERADE -o <wan interface> or – iptables -t nat -A POSTROUTING -j SNAT -o <wan interface> --to <wan address>
  6. Router NAT SNAT – Requires you to pass the IP

    of your public interface – Keeps track of connections when interface is brought down and back up MASQUERADE – You just need to pass the name of the public interface – Handy when using DHCP and you don't know your public IP beforehand or if it might change over time – Overhead with respect to SNAT as the IP of the device needs to be looked up every packet
  7. Open vSwitch • Multilayer virtual switch • Network automation •

    Standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag) • Supports distribution across multiple physical servers
  8. Open vSwitch Create a bridge ovs-vsctl add-br internal_br Create fake

    bridges with vlan tags ovs-vsctl add-br management_br internal_br 10 ovs-vsctl add-br lan_br internal_br 20 ovs-vsctl add-br wifi_br internal_br 30 Create a bond device with physical interfaces and attach it to the bridge ovs-vsctl add-bond internal_br bond0 enp0s9 enp0s10 lacp=active Add interface to the bridge ovs-vsctl add-port interal_br enp0s3
  9. Open vSwitch Bonding – combining multiple nics • LACP –

    Requires LACP protocol on both ends – OVS falls back to active-backup • Active Backup – Only 1 NIC is active, fallback to secondary • SLB – Allows some limited form of loadbalancing – Use this with dumb switches • balance-tcp
  10. DHCP dnsmasq Pro: – Small and simple – DNS included

    Con: – More geard towards desktop (dns caching) and small networks isc-dhcp-server Pro: – The go-to industry standard – Highly configurable Con (?): – Dynamic DNS decoupled
  11. DHCP dnsmasq # Set domain domain=zanzi.lan # Upstream DNS servers

    server=8.8.8.8 server=8.8.4.4 # Prevent non-routable private addresses from being forwarded bogus-priv # Prepend domain to all hosts expand-hosts # Do not read /etc/hosts no-hosts # Read hosts.dnsmasq for hosts entries addn-hosts=/etc/hosts.dnsmasq # Read /etc/ethers for static mac to ip entries read-ethers # Only bind to interfaces that it's listening on bind-interfaces # Be authoritative and barge in when a machine wakes up and broadcasts's a dhcp request dhcp-authoritative
  12. DHCP dnsmasq # Set listening interface, dhcp range and lease

    time dhcp-range=lan_vlan,192.168.20.6,192.168.20.254,8h # Pass default gateway dhcp-option=lan_vlan,3,192.168.20.1 # Pass DNS server dhcp-option=lan_vlan,6,192.168.20.1,192.168.20.2 # Pass search domain dhcp-option=lan_vlan,119,zanzi.lan # Pass NTP server dhcp-option=lan_vlan,42,192.168.20.1,192.168.20.2
  13. DHCP dnsmasq # /etc/ethers 00:07:ef:00:00:03 192.168.10.3 00:07:ef:00:00:04 192.168.10.4 00:07:ef:00:00:07 192.168.10.7

    or # /etc/dnsmasq.conf dhcp-host=00:07:ef:00:00:03,192.168.10.3 dhcp-host=00:07:ef:00:00:04,192.168.10.4 dhcp-host=00:07:ef:00:00:07,192.168.10.7
  14. DHCP isc-dhcp-server # ---------- # dhcpd.conf # ---------- authoritative; #omapi-port

    7911; default-lease-time 43200; max-lease-time 86400; log-facility daemon; option domain-name "zanzi.lan"; option domain-name-servers 192.169.10.1, 192.169.20.1; option fqdn.no-client-update on; # set the "O" and "S" flag bits option fqdn.rcode2 255; option ntp-servers none; ddns-update-style none; include "/etc/dhcp/dhcpd.pools"; include "/etc/dhcp/dhcpd.hosts"; include "/etc/dhcp/dhcpd.ignoredsubnets";
  15. DHCP isc-dhcp-server ################################# # lan pool 192.169.20.0 255.255.255.0 ################################# subnet

    192.169.20.0 netmask 255.255.255.0 { pool { range 192.169.20.5 192.169.20.254; } option domain-name "zanzi.lan"; option subnet-mask 255.255.255.0; option routers 192.169.20.1; option domain-name-servers 192.169.20.1; }
  16. DHCP isc-dhcp-server subnet 192.168.10.0 netmask 255.255.255.0 { … host zanzi-web

    { hardware ethernet 00:07:ef:00:00:03; fixed-address 192.168.10.3; } host zanzi-share { hardware ethernet 00:07:ef:00:00:04; fixed-address 192.168.10.4; } host zanzi-cache { hardware ethernet 00:07:ef:00:00:07; fixed-address 192.168.10.7; } }
  17. DNS dnsmasq Pro: – Small and simple – DNS included

    with Dynamic Hosts Con: – More geard towards desktop (dns caching) and small networks bind9 Pro: – The go-to industry standard – Highly configurable Con (?): – Dynamic DNS decoupled – Harder to configure
  18. DNS dnsmasq # Set domain domain=zanzi.lan # Upstream DNS servers

    server=8.8.8.8 server=8.8.4.4 # Prevent non-routable private addresses from being forwarded bogus-priv # Prepend domain to all hosts expand-hosts # Do not read /etc/hosts no-hosts # Read hosts.dnsmasq for hosts entries addn-hosts=/etc/hosts.dnsmasq # Read /etc/ethers for static mac to ip entries read-ethers # Only bind to interfaces that it's listening on bind-interfaces # Be authoritative and barge in when a machine wakes up and broadcasts's a dhcp request dhcp-authoritative
  19. DNS dnsmasq # Steam address=/.cs.steampowered.com/192.168.10.3 address=/.steamcontent.com/192.168.10.3 address=/content1.steampowered.com/192.168.10.3 address=/content2.steampowered.com/192.168.10.3 address=/content3.steampowered.com/192.168.10.3 address=/content4.steampowered.com/192.168.10.3

    address=/content5.steampowered.com/192.168.10.3 address=/content6.steampowered.com/192.168.10.3 address=/content7.steampowered.com/192.168.10.3 address=/content8.steampowered.com/192.168.10.3 address=/clientconfig.akamai.steamstatic.com/192.168.10.3 address=/hsar.steampowered.com.edgesuite.net/192.168.10.3
  20. DNS bind9 options { listen-on port 53 { any; };

    forwarders { 1.1.1.1; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; allow-query { 192.169.10.0/24; 192.169.20.0/24; }; recursion no; }; zone "169.192.in-addr.arpa" IN { type master; file "169.192.in-addr.arpa"; allow-update { key DHCP_UPDATER }; }; zone "zanzi.lan" IN { type master; file "zanzi.lan"; allow-update { key DHCP_UPDATER }; };
  21. LXC • Full OS Containers • 10x greater density than

    KVM • 57% less latency than KVM • 94% faster instance launching than KVM
  22. LXC

  23. LXC • lxc profile list • lxc profile show default

    • lxc profile edit default • lxc profile apply <container> <profile1>,… • lxc init/launch ubuntu: webserver -t c<CPU>-m<RAM in GB> • lxc start/stop webserver • lxc exec webserver bash config: user.network_mode: dhcp description: Default LXD profile devices: eth0: name: eth0 nictype: bridged parent: management_vlan type: nic name: default used_by: []
  24. LXC Preconfigured Network • lxc init ubuntu: zanzi-web • lxc

    config device add zanzi-web eth0 nic nictype=bridged parent=management_vlan name=eth0 hwaddr=00:07:ef:00:00:03 • lxc start zanzi-web • lxc list
  25. Squid Allow access acl management_vlan src 192.168.10.0/24 acl lan_vlan src

    192.168.20.0/24 acl wifi_vlan src 192.168.30.0/24 http_access allow management_vlan http_access allow lan_vlan http_access allow wifi_vlan
  26. Squid Ports and packet redirection http_port 8080 transparent https_port 8443

    intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=16MB cert=/etc/squid/ssl_cert/server.pem
  27. Squid Chain PREROUTING (policy ACCEPT 1485 packets, 138K bytes) pkts

    bytes target prot opt in out source destination 0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443 0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443 0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443
  28. Squid HTTP caching cache_mem 4 GB maximum_object_size_in_memory 2048 KB cache_dir

    aufs /var/cache/squid 30000 16 256 maximum_object_size 10 MB minimum_object_size 4 KB
  29. Squid HTTPS caching Cannot be done unless you perform a

    MITM # Become a TCP tunnel without decrypting proxied traffic ssl_bump splice
  30. Squid HTTPS caching – but if you really want to….

    # Connect to server first, generate mimicked cert ssl_bump server-first all # sslcrtd sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db/ -M 4MB -b 4 KB sslcrtd_children 8 startup=1 idle=1 sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER
  31. Squid People will need to accept your CA Certificate so

    that the on- the-fly signed certs produced by Squid are accepted by the OS and browser. Captive Portal?
  32. Squid Blocking domains – SquidGuard dest porn { domainlist porn/domains

    urllist porn/urls } dest warez { domainlist warez/domains urllist warez/urls }
  33. Squid acl { default { pass !porn !warez all redirect

    http://<webserver>/block.html } } url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf
  34. Squid Rate Limiting acl netflix dstdomain .nflxvideo.net .netflix.com .nflximg.net .nflxext.com

    .llnwd.net .fast.com acl youtube dstdomain .googlevideo.com .youtube.com .ytimg.com delay_pools 2 # 2 delay pools delay_initial_bucket_level 50 # netflix delay_class 1 3 # pool 1 is a class 3 pool delay_parameters 1 none 400000/400000 delay_access 1 allow netflix management_vlan delay_access 1 allow netflix lan_vlan delay_access 1 allow netflix wifi_vlan delay_access 1 deny all
  35. Squid # youtube delay_class 2 3 # pool 2 is

    a class 3 pool delay_parameters 2 none 400000/400000 delay_access 2 allow youtube management_vlan delay_access 2 allow youtube lan_vlan delay_access 2 allow youtube wifi_vlan delay_access 2 deny all
  36. Metrics • Collectd – Plugin based metric gathering daemon –

    100+ plugins: Apache, CPU, Load, Network, Ping, …. • Graphite – Store numeric time-series data – Render graphs of this data on demand – echo "foo.bar 1 `date +%s`" | nc localhost 2003 • Grafana – Query metrics from Graphite (and more!) and generate dashboards
  37. Metrics CollectD FQDNLookup true Interval 120 LoadPlugin df LoadPlugin network

    LoadPlugin load LoadPlugin memory LoadPlugin write_graphite <Plugin write_graphite> <Carbon> Host "localhost" Port "2003" Prefix "collectd." StoreRates true AlwaysAppendDS false EscapeCharacter "_" </Carbon> </Plugin>
  38. Caching Games • DNS overriding • Nginx • https://github.com/multiplay/lancache –

    Blizzard – Steam – Origin – Riot – Sony – Windows Updates – …..
  39. Caching Games Nginx • Requests are sent through Nginx •

    Will check on disk if it already has the same requested data • If not, acts as a caching forwarding proxy (like Squid) • Saves terabytes of data passing through your uplink
  40. Multiple Uplinks • DHCP client by default requests on all

    connected interfaces – Will override the default gateway – Will override your configured nameserver in /etc/resolv.conf – Last connected uplink becomes the primary/default • We only want enough information for routing Write custom dhcp hooks / Use correct flags for your interface
  41. Multiple Uplinks /etc/dhcp/dhclient.conf # Global request subnet-mask, broadcast-address, time-offset, interface-

    mtu, rfc3442-classless-static-routes; # Primary interface "eth0" { request routers; }
  42. Multiple Uplinks mask2cdr () { # Assumes there's no "255."

    after a non-255 byte in the mask local x=${1##*255.} set -- 0^^^128^192^224^240^248^252^254^ $(( ($ {#1} - ${#x})*2 )) ${x%%.*} x=${1%%$3*} echo $(( $2 + (${#x}/4) )) } logger -t routing "Received DHCP Trigger for <%= @value %>" printenv > /tmp/dhcp-<%= @value %> if [ "$interface" = "<%= @value %>" ]; then if [ "$reason" = "BOUND" ]; then logger -t routing "Received DHCP Bound Info for <%= @value %>" WAN_IP=$new_ip_address WAN_GW=$new_routers WAN_NET=$new_network_number WAN_CDIR=$(mask2cdr $new_subnet_mask) logger -t routing "Using the following: ${WAN_IP}/${WAN_CDIR} and GW ${WAN_GW}" logger -t routing "Flushing <%= @table %> routing rules" ip route flush table <%= @table %> ip route add ${WAN_NET}/${WAN_CDIR} dev <%= @value %> src $ {WAN_IP} table <%= @table %> ip route add default via ${WAN_GW} dev <%= @value %> table < %= @table %> logger -t routing "Updated IP Routing rules for <%= @value %>" fi fi
  43. Multiple Uplinks Don’t forget your other local networks! ip route

    add 192.168.10.0/24 dev management_vlan table isp1 ip route add 192.168.20.0/24 dev lan_vlan table isp1 ip route add 192.168.30.0/24 dev wifi_vlan table isp1 ip route add 192.168.10.0/24 dev management_vlan table isp2 ip route add 192.168.20.0/24 dev lan_vlan table isp2 ip route add 192.168.30.0/24 dev wifi_vlan table isp2
  44. Multiple Uplinks Only have the Cache server use the second

    wan ip rule add from 192.168.10.7/32 table isp2 ip rule add to 192.168.10.7/32 table isp2
  45. Traffic Shaping TC – Traffic Control Avoid Bufferbloat by artificially

    lowering your Download and Upload Dropping packets is GOOD (tcp self regulation)
  46. Recomendations • Do a speedtest at the beginning to set

    a baseline • Keep an eye on ping latency • Structure iptables rules for legibility • Place Game server VMs in the same vlan (if broadcast auto discovery) • Have fun
  47. Q&A