Open Source Lan Party Gateway - LOADays 2018

Open Source Lan Party Gateway - LOADays 2018

Presentation on how to setup a Lan Party gateway server.

Used components:

Ubuntu/Centos
dnsmasq
bind9
isc-dhcp-server
LXC
openvSwitch
tc
iptables
squid
collectd
graphite
grafana

Cd0a2e3d500c66c02d6ae3fa18389e89?s=128

Christophe Vanlancker

April 21, 2018
Tweet

Transcript

  1. Open­source LAN Party Gateway Christophe Vanlancker ( @Carroarmato0 )

  2. Organizations ICT-Blue Lan: 2011, 2012, 2013 ~40 gamers Zanzilan: 15w

    ~80 gamers, 15z ~130 gamers
  3. Some History • 80’s: “PCs invade the home” • 90’s:

    – Cheaper networking equipment – Cheaper PCs – Rise of popular games: Quake, Age of Empires, Warcraft, Unreal Tournament, … • 20xx: – Online gaming: League of Legends, World of Warcraft, PUBG, Fortnite, Runscape,...
  4. The Basics • An Internet Connection • Router • Switch

    • DNS • DHCP
  5. Advanced • Services – Virtualization/Containerization – Monitoring/Metrics – Game servers

    • HA • Caching • DNS overriding • Link Aggregation • Multiple UPLinks • QoS/Throttling • Network segregation (VLANs)
  6. Used Technology • OS: Centos/Ubuntu • DNS: dnsmasq / Bind

    • DHCP: dnsmasq / ISC DHCP • Containerization: LXC • Firewall: iptables • Traffic Shaping: tc • Forward Proxy: Squid • Metrics: Graphite, Collectd, Grafana • Networking: Open vSwitch
  7. Router Enable routing functionality • On the fly sysctl -w

    net.ipv4.ip_forward=1 or echo 1 > /proc/sys/net/ipv4/ip_forward • Permanent: /etc/sysctl.conf: net.ipv4.ip_forward = 1
  8. Router NAT Makes packets leaving through your uplink use the

    public IP of the router so that replies can return to it. – iptables -t nat -A POSTROUTING -j MASQUERADE -o <wan interface> or – iptables -t nat -A POSTROUTING -j SNAT -o <wan interface> --to <wan address>
  9. Router NAT SNAT – Requires you to pass the IP

    of your public interface – Keeps track of connections when interface is brought down and back up MASQUERADE – You just need to pass the name of the public interface – Handy when using DHCP and you don't know your public IP beforehand or if it might change over time – Overhead with respect to SNAT as the IP of the device needs to be looked up every packet
  10. Open vSwitch • Multilayer virtual switch • Network automation •

    Standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag) • Supports distribution across multiple physical servers
  11. Open vSwitch • ovs-vswitchd (daemon) • ovsdb-server (daemon) • ovs-vsctl

    • ovs-appctl
  12. Open vSwitch Create a bridge ovs-vsctl add-br internal_br Create fake

    bridges with vlan tags ovs-vsctl add-br management_br internal_br 10 ovs-vsctl add-br lan_br internal_br 20 ovs-vsctl add-br wifi_br internal_br 30 Create a bond device with physical interfaces and attach it to the bridge ovs-vsctl add-bond internal_br bond0 enp0s9 enp0s10 lacp=active Add interface to the bridge ovs-vsctl add-port interal_br enp0s3
  13. Open vSwitch ovs-vsctl show

  14. Open vSwitch ip link show

  15. Open vSwitch Bonding – combining multiple nics • LACP –

    Requires LACP protocol on both ends – OVS falls back to active-backup • Active Backup – Only 1 NIC is active, fallback to secondary • SLB – Allows some limited form of loadbalancing – Use this with dumb switches • balance-tcp
  16. Open vSwitch ovs-appctl bond/show bond0 ovs-appctl lacp/show bond0

  17. DHCP dnsmasq Pro: – Small and simple – DNS included

    Con: – More geard towards desktop (dns caching) and small networks isc-dhcp-server Pro: – The go-to industry standard – Highly configurable Con (?): – Dynamic DNS decoupled
  18. DHCP dnsmasq # Set domain domain=zanzi.lan # Upstream DNS servers

    server=8.8.8.8 server=8.8.4.4 # Prevent non-routable private addresses from being forwarded bogus-priv # Prepend domain to all hosts expand-hosts # Do not read /etc/hosts no-hosts # Read hosts.dnsmasq for hosts entries addn-hosts=/etc/hosts.dnsmasq # Read /etc/ethers for static mac to ip entries read-ethers # Only bind to interfaces that it's listening on bind-interfaces # Be authoritative and barge in when a machine wakes up and broadcasts's a dhcp request dhcp-authoritative
  19. DHCP dnsmasq # Set listening interface, dhcp range and lease

    time dhcp-range=lan_vlan,192.168.20.6,192.168.20.254,8h # Pass default gateway dhcp-option=lan_vlan,3,192.168.20.1 # Pass DNS server dhcp-option=lan_vlan,6,192.168.20.1,192.168.20.2 # Pass search domain dhcp-option=lan_vlan,119,zanzi.lan # Pass NTP server dhcp-option=lan_vlan,42,192.168.20.1,192.168.20.2
  20. DHCP dnsmasq # /etc/ethers 00:07:ef:00:00:03 192.168.10.3 00:07:ef:00:00:04 192.168.10.4 00:07:ef:00:00:07 192.168.10.7

    or # /etc/dnsmasq.conf dhcp-host=00:07:ef:00:00:03,192.168.10.3 dhcp-host=00:07:ef:00:00:04,192.168.10.4 dhcp-host=00:07:ef:00:00:07,192.168.10.7
  21. DHCP isc-dhcp-server # ---------- # dhcpd.conf # ---------- authoritative; #omapi-port

    7911; default-lease-time 43200; max-lease-time 86400; log-facility daemon; option domain-name "zanzi.lan"; option domain-name-servers 192.169.10.1, 192.169.20.1; option fqdn.no-client-update on; # set the "O" and "S" flag bits option fqdn.rcode2 255; option ntp-servers none; ddns-update-style none; include "/etc/dhcp/dhcpd.pools"; include "/etc/dhcp/dhcpd.hosts"; include "/etc/dhcp/dhcpd.ignoredsubnets";
  22. DHCP isc-dhcp-server ################################# # lan pool 192.169.20.0 255.255.255.0 ################################# subnet

    192.169.20.0 netmask 255.255.255.0 { pool { range 192.169.20.5 192.169.20.254; } option domain-name "zanzi.lan"; option subnet-mask 255.255.255.0; option routers 192.169.20.1; option domain-name-servers 192.169.20.1; }
  23. DHCP isc-dhcp-server subnet 192.168.10.0 netmask 255.255.255.0 { … host zanzi-web

    { hardware ethernet 00:07:ef:00:00:03; fixed-address 192.168.10.3; } host zanzi-share { hardware ethernet 00:07:ef:00:00:04; fixed-address 192.168.10.4; } host zanzi-cache { hardware ethernet 00:07:ef:00:00:07; fixed-address 192.168.10.7; } }
  24. DNS dnsmasq Pro: – Small and simple – DNS included

    with Dynamic Hosts Con: – More geard towards desktop (dns caching) and small networks bind9 Pro: – The go-to industry standard – Highly configurable Con (?): – Dynamic DNS decoupled – Harder to configure
  25. DNS dnsmasq # Set domain domain=zanzi.lan # Upstream DNS servers

    server=8.8.8.8 server=8.8.4.4 # Prevent non-routable private addresses from being forwarded bogus-priv # Prepend domain to all hosts expand-hosts # Do not read /etc/hosts no-hosts # Read hosts.dnsmasq for hosts entries addn-hosts=/etc/hosts.dnsmasq # Read /etc/ethers for static mac to ip entries read-ethers # Only bind to interfaces that it's listening on bind-interfaces # Be authoritative and barge in when a machine wakes up and broadcasts's a dhcp request dhcp-authoritative
  26. DNS dnsmasq # Steam address=/.cs.steampowered.com/192.168.10.3 address=/.steamcontent.com/192.168.10.3 address=/content1.steampowered.com/192.168.10.3 address=/content2.steampowered.com/192.168.10.3 address=/content3.steampowered.com/192.168.10.3 address=/content4.steampowered.com/192.168.10.3

    address=/content5.steampowered.com/192.168.10.3 address=/content6.steampowered.com/192.168.10.3 address=/content7.steampowered.com/192.168.10.3 address=/content8.steampowered.com/192.168.10.3 address=/clientconfig.akamai.steamstatic.com/192.168.10.3 address=/hsar.steampowered.com.edgesuite.net/192.168.10.3
  27. DNS bind9 options { listen-on port 53 { any; };

    forwarders { 1.1.1.1; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; allow-query { 192.169.10.0/24; 192.169.20.0/24; }; recursion no; }; zone "169.192.in-addr.arpa" IN { type master; file "169.192.in-addr.arpa"; allow-update { key DHCP_UPDATER }; }; zone "zanzi.lan" IN { type master; file "zanzi.lan"; allow-update { key DHCP_UPDATER }; };
  28. LXC • Full OS Containers • 10x greater density than

    KVM • 57% less latency than KVM • 94% faster instance launching than KVM
  29. LXC

  30. LXC • lxc profile list • lxc profile show default

    • lxc profile edit default • lxc profile apply <container> <profile1>,… • lxc init/launch ubuntu: webserver -t c<CPU>-m<RAM in GB> • lxc start/stop webserver • lxc exec webserver bash config: user.network_mode: dhcp description: Default LXD profile devices: eth0: name: eth0 nictype: bridged parent: management_vlan type: nic name: default used_by: []
  31. LXC Preconfigured Network • lxc init ubuntu: zanzi-web • lxc

    config device add zanzi-web eth0 nic nictype=bridged parent=management_vlan name=eth0 hwaddr=00:07:ef:00:00:03 • lxc start zanzi-web • lxc list
  32. Squid • Caching proxy • HTTP, HTTPS, FTP, … •

    ACLs • Domain level blocking
  33. Squid Allow access acl management_vlan src 192.168.10.0/24 acl lan_vlan src

    192.168.20.0/24 acl wifi_vlan src 192.168.30.0/24 http_access allow management_vlan http_access allow lan_vlan http_access allow wifi_vlan
  34. Squid Ports and packet redirection http_port 8080 transparent https_port 8443

    intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=16MB cert=/etc/squid/ssl_cert/server.pem
  35. Squid Chain PREROUTING (policy ACCEPT 1485 packets, 138K bytes) pkts

    bytes target prot opt in out source destination 0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443 0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443 0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443
  36. Squid HTTP caching cache_mem 4 GB maximum_object_size_in_memory 2048 KB cache_dir

    aufs /var/cache/squid 30000 16 256 maximum_object_size 10 MB minimum_object_size 4 KB
  37. Squid HTTPS caching Cannot be done unless you perform a

    MITM # Become a TCP tunnel without decrypting proxied traffic ssl_bump splice
  38. Squid HTTPS caching – but if you really want to….

    # Connect to server first, generate mimicked cert ssl_bump server-first all # sslcrtd sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db/ -M 4MB -b 4 KB sslcrtd_children 8 startup=1 idle=1 sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER
  39. Squid People will need to accept your CA Certificate so

    that the on- the-fly signed certs produced by Squid are accepted by the OS and browser. Captive Portal?
  40. Squid Blocking domains acl blockeddomain dstdomain "/etc/squid/blocked.domains.acl" speedtest.net fast.com speedtest.telenet.be

    pxs.speedtestcustom.com http_access deny blockeddomain
  41. Squid Blocking domains – SquidGuard dest porn { domainlist porn/domains

    urllist porn/urls } dest warez { domainlist warez/domains urllist warez/urls }
  42. Squid acl { default { pass !porn !warez all redirect

    http://<webserver>/block.html } } url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf
  43. Squid Rate Limiting acl netflix dstdomain .nflxvideo.net .netflix.com .nflximg.net .nflxext.com

    .llnwd.net .fast.com acl youtube dstdomain .googlevideo.com .youtube.com .ytimg.com delay_pools 2 # 2 delay pools delay_initial_bucket_level 50 # netflix delay_class 1 3 # pool 1 is a class 3 pool delay_parameters 1 none 400000/400000 delay_access 1 allow netflix management_vlan delay_access 1 allow netflix lan_vlan delay_access 1 allow netflix wifi_vlan delay_access 1 deny all
  44. Squid # youtube delay_class 2 3 # pool 2 is

    a class 3 pool delay_parameters 2 none 400000/400000 delay_access 2 allow youtube management_vlan delay_access 2 allow youtube lan_vlan delay_access 2 allow youtube wifi_vlan delay_access 2 deny all
  45. Metrics • Collectd – Plugin based metric gathering daemon –

    100+ plugins: Apache, CPU, Load, Network, Ping, …. • Graphite – Store numeric time-series data – Render graphs of this data on demand – echo "foo.bar 1 `date +%s`" | nc localhost 2003 • Grafana – Query metrics from Graphite (and more!) and generate dashboards
  46. Metrics CollectD FQDNLookup true Interval 120 LoadPlugin df LoadPlugin network

    LoadPlugin load LoadPlugin memory LoadPlugin write_graphite <Plugin write_graphite> <Carbon> Host "localhost" Port "2003" Prefix "collectd." StoreRates true AlwaysAppendDS false EscapeCharacter "_" </Carbon> </Plugin>
  47. Metrics Graphite – carbon-cache – whisper – graphite-web

  48. Metrics

  49. Caching Games • DNS overriding • Nginx • https://github.com/multiplay/lancache –

    Blizzard – Steam – Origin – Riot – Sony – Windows Updates – …..
  50. Caching Games DNS # Steam address=/.cs.steampowered.com/192.168.10.3 address=/.steamcontent.com/192.168.10.3 address=/content1.steampowered.com/192.168.10.3 ….. zone

    "steampowered.com" IN { type master; file "steam"; };
  51. Caching Games Nginx • Requests are sent through Nginx •

    Will check on disk if it already has the same requested data • If not, acts as a caching forwarding proxy (like Squid) • Saves terabytes of data passing through your uplink
  52. Multiple Uplinks • DHCP client by default requests on all

    connected interfaces – Will override the default gateway – Will override your configured nameserver in /etc/resolv.conf – Last connected uplink becomes the primary/default • We only want enough information for routing Write custom dhcp hooks / Use correct flags for your interface
  53. Multiple Uplinks /etc/dhcp/dhclient.conf # Global request subnet-mask, broadcast-address, time-offset, interface-

    mtu, rfc3442-classless-static-routes; # Primary interface "eth0" { request routers; }
  54. Multiple Uplinks Routing tables # /etc/iproute2/rt_tables 1 isp1 2 isp2

  55. Multiple Uplinks mask2cdr () { # Assumes there's no "255."

    after a non-255 byte in the mask local x=${1##*255.} set -- 0^^^128^192^224^240^248^252^254^ $(( ($ {#1} - ${#x})*2 )) ${x%%.*} x=${1%%$3*} echo $(( $2 + (${#x}/4) )) } logger -t routing "Received DHCP Trigger for <%= @value %>" printenv > /tmp/dhcp-<%= @value %> if [ "$interface" = "<%= @value %>" ]; then if [ "$reason" = "BOUND" ]; then logger -t routing "Received DHCP Bound Info for <%= @value %>" WAN_IP=$new_ip_address WAN_GW=$new_routers WAN_NET=$new_network_number WAN_CDIR=$(mask2cdr $new_subnet_mask) logger -t routing "Using the following: ${WAN_IP}/${WAN_CDIR} and GW ${WAN_GW}" logger -t routing "Flushing <%= @table %> routing rules" ip route flush table <%= @table %> ip route add ${WAN_NET}/${WAN_CDIR} dev <%= @value %> src $ {WAN_IP} table <%= @table %> ip route add default via ${WAN_GW} dev <%= @value %> table < %= @table %> logger -t routing "Updated IP Routing rules for <%= @value %>" fi fi
  56. Multiple Uplinks Don’t forget your other local networks! ip route

    add 192.168.10.0/24 dev management_vlan table isp1 ip route add 192.168.20.0/24 dev lan_vlan table isp1 ip route add 192.168.30.0/24 dev wifi_vlan table isp1 ip route add 192.168.10.0/24 dev management_vlan table isp2 ip route add 192.168.20.0/24 dev lan_vlan table isp2 ip route add 192.168.30.0/24 dev wifi_vlan table isp2
  57. Multiple Uplinks Only have the Cache server use the second

    wan ip rule add from 192.168.10.7/32 table isp2 ip rule add to 192.168.10.7/32 table isp2
  58. Traffic Shaping TC – Traffic Control Avoid Bufferbloat by artificially

    lowering your Download and Upload Dropping packets is GOOD (tcp self regulation)
  59. iftop

  60. Recomendations • Do a speedtest at the beginning to set

    a baseline • Keep an eye on ping latency • Structure iptables rules for legibility • Place Game server VMs in the same vlan (if broadcast auto discovery) • Have fun
  61. Q&A