Open Source Lan Party Gateway - LOADays 2018

Presentation on how to setup a Lan Party gateway server.

Used components:

Ubuntu/Centos
dnsmasq
bind9
isc-dhcp-server
LXC
openvSwitch
tc
iptables
squid
collectd
graphite
grafana

Transcript

1. Open­source LAN Party Gateway Christophe Vanlancker ( @Carroarmato0 )
2. Organizations ICT-Blue Lan: 2011, 2012, 2013 ~40 gamers Zanzilan: 15w ~80 gamers, 15z ~130 gamers
3. Some History • 80’s: “PCs invade the home” • 90’s: – Cheaper networking equipment – Cheaper PCs – Rise of popular games: Quake, Age of Empires, Warcraft, Unreal Tournament, … • 20xx: – Online gaming: League of Legends, World of Warcraft, PUBG, Fortnite, Runscape,...
4. The Basics • An Internet Connection • Router • Switch • DNS • DHCP
5. Advanced • Services – Virtualization/Containerization – Monitoring/Metrics – Game servers • HA • Caching • DNS overriding • Link Aggregation • Multiple UPLinks • QoS/Throttling • Network segregation (VLANs)
6. Used Technology • OS: Centos/Ubuntu • DNS: dnsmasq / Bind • DHCP: dnsmasq / ISC DHCP • Containerization: LXC • Firewall: iptables • Traffic Shaping: tc • Forward Proxy: Squid • Metrics: Graphite, Collectd, Grafana • Networking: Open vSwitch
7. Router Enable routing functionality • On the fly sysctl -w net.ipv4.ip_forward=1 or echo 1 > /proc/sys/net/ipv4/ip_forward • Permanent: /etc/sysctl.conf: net.ipv4.ip_forward = 1
8. Router NAT Makes packets leaving through your uplink use the public IP of the router so that replies can return to it. – iptables -t nat -A POSTROUTING -j MASQUERADE -o <wan interface> or – iptables -t nat -A POSTROUTING -j SNAT -o <wan interface> --to <wan address>
9. Router NAT SNAT – Requires you to pass the IP of your public interface – Keeps track of connections when interface is brought down and back up MASQUERADE – You just need to pass the name of the public interface – Handy when using DHCP and you don't know your public IP beforehand or if it might change over time – Overhead with respect to SNAT as the IP of the device needs to be looked up every packet
10. Open vSwitch • Multilayer virtual switch • Network automation • Standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag) • Supports distribution across multiple physical servers
11. Open vSwitch • ovs-vswitchd (daemon) • ovsdb-server (daemon) • ovs-vsctl • ovs-appctl
12. Open vSwitch Create a bridge ovs-vsctl add-br internal_br Create fake bridges with vlan tags ovs-vsctl add-br management_br internal_br 10 ovs-vsctl add-br lan_br internal_br 20 ovs-vsctl add-br wifi_br internal_br 30 Create a bond device with physical interfaces and attach it to the bridge ovs-vsctl add-bond internal_br bond0 enp0s9 enp0s10 lacp=active Add interface to the bridge ovs-vsctl add-port interal_br enp0s3
13. Open vSwitch ovs-vsctl show
14. Open vSwitch ip link show
15. Open vSwitch Bonding – combining multiple nics • LACP – Requires LACP protocol on both ends – OVS falls back to active-backup • Active Backup – Only 1 NIC is active, fallback to secondary • SLB – Allows some limited form of loadbalancing – Use this with dumb switches • balance-tcp
16. Open vSwitch ovs-appctl bond/show bond0 ovs-appctl lacp/show bond0
17. DHCP dnsmasq Pro: – Small and simple – DNS included Con: – More geard towards desktop (dns caching) and small networks isc-dhcp-server Pro: – The go-to industry standard – Highly configurable Con (?): – Dynamic DNS decoupled
18. DHCP dnsmasq # Set domain domain=zanzi.lan # Upstream DNS servers server=8.8.8.8 server=8.8.4.4 # Prevent non-routable private addresses from being forwarded bogus-priv # Prepend domain to all hosts expand-hosts # Do not read /etc/hosts no-hosts # Read hosts.dnsmasq for hosts entries addn-hosts=/etc/hosts.dnsmasq # Read /etc/ethers for static mac to ip entries read-ethers # Only bind to interfaces that it's listening on bind-interfaces # Be authoritative and barge in when a machine wakes up and broadcasts's a dhcp request dhcp-authoritative
19. DHCP dnsmasq # Set listening interface, dhcp range and lease time dhcp-range=lan_vlan,192.168.20.6,192.168.20.254,8h # Pass default gateway dhcp-option=lan_vlan,3,192.168.20.1 # Pass DNS server dhcp-option=lan_vlan,6,192.168.20.1,192.168.20.2 # Pass search domain dhcp-option=lan_vlan,119,zanzi.lan # Pass NTP server dhcp-option=lan_vlan,42,192.168.20.1,192.168.20.2
20. DHCP dnsmasq # /etc/ethers 00:07:ef:00:00:03 192.168.10.3 00:07:ef:00:00:04 192.168.10.4 00:07:ef:00:00:07 192.168.10.7 or # /etc/dnsmasq.conf dhcp-host=00:07:ef:00:00:03,192.168.10.3 dhcp-host=00:07:ef:00:00:04,192.168.10.4 dhcp-host=00:07:ef:00:00:07,192.168.10.7
21. DHCP isc-dhcp-server # ---------- # dhcpd.conf # ---------- authoritative; #omapi-port 7911; default-lease-time 43200; max-lease-time 86400; log-facility daemon; option domain-name "zanzi.lan"; option domain-name-servers 192.169.10.1, 192.169.20.1; option fqdn.no-client-update on; # set the "O" and "S" flag bits option fqdn.rcode2 255; option ntp-servers none; ddns-update-style none; include "/etc/dhcp/dhcpd.pools"; include "/etc/dhcp/dhcpd.hosts"; include "/etc/dhcp/dhcpd.ignoredsubnets";
22. DHCP isc-dhcp-server ################################# # lan pool 192.169.20.0 255.255.255.0 ################################# subnet 192.169.20.0 netmask 255.255.255.0 { pool { range 192.169.20.5 192.169.20.254; } option domain-name "zanzi.lan"; option subnet-mask 255.255.255.0; option routers 192.169.20.1; option domain-name-servers 192.169.20.1; }
23. DHCP isc-dhcp-server subnet 192.168.10.0 netmask 255.255.255.0 { … host zanzi-web { hardware ethernet 00:07:ef:00:00:03; fixed-address 192.168.10.3; } host zanzi-share { hardware ethernet 00:07:ef:00:00:04; fixed-address 192.168.10.4; } host zanzi-cache { hardware ethernet 00:07:ef:00:00:07; fixed-address 192.168.10.7; } }
24. DNS dnsmasq Pro: – Small and simple – DNS included with Dynamic Hosts Con: – More geard towards desktop (dns caching) and small networks bind9 Pro: – The go-to industry standard – Highly configurable Con (?): – Dynamic DNS decoupled – Harder to configure
25. DNS dnsmasq # Set domain domain=zanzi.lan # Upstream DNS servers server=8.8.8.8 server=8.8.4.4 # Prevent non-routable private addresses from being forwarded bogus-priv # Prepend domain to all hosts expand-hosts # Do not read /etc/hosts no-hosts # Read hosts.dnsmasq for hosts entries addn-hosts=/etc/hosts.dnsmasq # Read /etc/ethers for static mac to ip entries read-ethers # Only bind to interfaces that it's listening on bind-interfaces # Be authoritative and barge in when a machine wakes up and broadcasts's a dhcp request dhcp-authoritative
26. DNS dnsmasq # Steam address=/.cs.steampowered.com/192.168.10.3 address=/.steamcontent.com/192.168.10.3 address=/content1.steampowered.com/192.168.10.3 address=/content2.steampowered.com/192.168.10.3 address=/content3.steampowered.com/192.168.10.3 address=/content4.steampowered.com/192.168.10.3 address=/content5.steampowered.com/192.168.10.3 address=/content6.steampowered.com/192.168.10.3 address=/content7.steampowered.com/192.168.10.3 address=/content8.steampowered.com/192.168.10.3 address=/clientconfig.akamai.steamstatic.com/192.168.10.3 address=/hsar.steampowered.com.edgesuite.net/192.168.10.3
27. DNS bind9 options { listen-on port 53 { any; }; forwarders { 1.1.1.1; }; directory "/var/named"; dump-file "/var/named/data/cache_dump.db"; statistics-file "/var/named/data/named_stats.txt"; memstatistics-file "/var/named/data/named_mem_stats.txt"; allow-query { 192.169.10.0/24; 192.169.20.0/24; }; recursion no; }; zone "169.192.in-addr.arpa" IN { type master; file "169.192.in-addr.arpa"; allow-update { key DHCP_UPDATER }; }; zone "zanzi.lan" IN { type master; file "zanzi.lan"; allow-update { key DHCP_UPDATER }; };
28. LXC • Full OS Containers • 10x greater density than KVM • 57% less latency than KVM • 94% faster instance launching than KVM
29. LXC
30. LXC • lxc profile list • lxc profile show default • lxc profile edit default • lxc profile apply <container> <profile1>,… • lxc init/launch ubuntu: webserver -t c<CPU>-m<RAM in GB> • lxc start/stop webserver • lxc exec webserver bash config: user.network_mode: dhcp description: Default LXD profile devices: eth0: name: eth0 nictype: bridged parent: management_vlan type: nic name: default used_by: []
31. LXC Preconfigured Network • lxc init ubuntu: zanzi-web • lxc config device add zanzi-web eth0 nic nictype=bridged parent=management_vlan name=eth0 hwaddr=00:07:ef:00:00:03 • lxc start zanzi-web • lxc list
32. Squid • Caching proxy • HTTP, HTTPS, FTP, … • ACLs • Domain level blocking
33. Squid Allow access acl management_vlan src 192.168.10.0/24 acl lan_vlan src 192.168.20.0/24 acl wifi_vlan src 192.168.30.0/24 http_access allow management_vlan http_access allow lan_vlan http_access allow wifi_vlan
34. Squid Ports and packet redirection http_port 8080 transparent https_port 8443 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=16MB cert=/etc/squid/ssl_cert/server.pem
35. Squid Chain PREROUTING (policy ACCEPT 1485 packets, 138K bytes) pkts bytes target prot opt in out source destination 0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080 0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443 0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443 0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443
36. Squid HTTP caching cache_mem 4 GB maximum_object_size_in_memory 2048 KB cache_dir aufs /var/cache/squid 30000 16 256 maximum_object_size 10 MB minimum_object_size 4 KB
37. Squid HTTPS caching Cannot be done unless you perform a MITM # Become a TCP tunnel without decrypting proxied traffic ssl_bump splice
38. Squid HTTPS caching – but if you really want to…. # Connect to server first, generate mimicked cert ssl_bump server-first all # sslcrtd sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db/ -M 4MB -b 4 KB sslcrtd_children 8 startup=1 idle=1 sslproxy_cert_error allow all sslproxy_flags DONT_VERIFY_PEER
39. Squid People will need to accept your CA Certificate so that the on- the-fly signed certs produced by Squid are accepted by the OS and browser. Captive Portal?
40. Squid Blocking domains acl blockeddomain dstdomain "/etc/squid/blocked.domains.acl" speedtest.net fast.com speedtest.telenet.be pxs.speedtestcustom.com http_access deny blockeddomain
41. Squid Blocking domains – SquidGuard dest porn { domainlist porn/domains urllist porn/urls } dest warez { domainlist warez/domains urllist warez/urls }
42. Squid acl { default { pass !porn !warez all redirect http://<webserver>/block.html } } url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf
43. Squid Rate Limiting acl netflix dstdomain .nflxvideo.net .netflix.com .nflximg.net .nflxext.com .llnwd.net .fast.com acl youtube dstdomain .googlevideo.com .youtube.com .ytimg.com delay_pools 2 # 2 delay pools delay_initial_bucket_level 50 # netflix delay_class 1 3 # pool 1 is a class 3 pool delay_parameters 1 none 400000/400000 delay_access 1 allow netflix management_vlan delay_access 1 allow netflix lan_vlan delay_access 1 allow netflix wifi_vlan delay_access 1 deny all
44. Squid # youtube delay_class 2 3 # pool 2 is a class 3 pool delay_parameters 2 none 400000/400000 delay_access 2 allow youtube management_vlan delay_access 2 allow youtube lan_vlan delay_access 2 allow youtube wifi_vlan delay_access 2 deny all
45. Metrics • Collectd – Plugin based metric gathering daemon – 100+ plugins: Apache, CPU, Load, Network, Ping, …. • Graphite – Store numeric time-series data – Render graphs of this data on demand – echo "foo.bar 1 `date +%s`" | nc localhost 2003 • Grafana – Query metrics from Graphite (and more!) and generate dashboards
46. Metrics CollectD FQDNLookup true Interval 120 LoadPlugin df LoadPlugin network LoadPlugin load LoadPlugin memory LoadPlugin write_graphite <Plugin write_graphite> <Carbon> Host "localhost" Port "2003" Prefix "collectd." StoreRates true AlwaysAppendDS false EscapeCharacter "_" </Carbon> </Plugin>
47. Metrics Graphite – carbon-cache – whisper – graphite-web
48. Metrics
49. Caching Games • DNS overriding • Nginx • https://github.com/multiplay/lancache – Blizzard – Steam – Origin – Riot – Sony – Windows Updates – …..
50. Caching Games DNS # Steam address=/.cs.steampowered.com/192.168.10.3 address=/.steamcontent.com/192.168.10.3 address=/content1.steampowered.com/192.168.10.3 ….. zone "steampowered.com" IN { type master; file "steam"; };
51. Caching Games Nginx • Requests are sent through Nginx • Will check on disk if it already has the same requested data • If not, acts as a caching forwarding proxy (like Squid) • Saves terabytes of data passing through your uplink
52. Multiple Uplinks • DHCP client by default requests on all connected interfaces – Will override the default gateway – Will override your configured nameserver in /etc/resolv.conf – Last connected uplink becomes the primary/default • We only want enough information for routing Write custom dhcp hooks / Use correct flags for your interface
53. Multiple Uplinks /etc/dhcp/dhclient.conf # Global request subnet-mask, broadcast-address, time-offset, interface- mtu, rfc3442-classless-static-routes; # Primary interface "eth0" { request routers; }
54. Multiple Uplinks Routing tables # /etc/iproute2/rt_tables 1 isp1 2 isp2
55. Multiple Uplinks mask2cdr () { # Assumes there's no "255." after a non-255 byte in the mask local x=${1##*255.} set -- 0^^^128^192^224^240^248^252^254^ $(( ($ {#1} - ${#x})*2 )) ${x%%.*} x=${1%%$3*} echo $(( $2 + (${#x}/4) )) } logger -t routing "Received DHCP Trigger for <%= @value %>" printenv > /tmp/dhcp-<%= @value %> if [ "$interface" = "<%= @value %>" ]; then if [ "$reason" = "BOUND" ]; then logger -t routing "Received DHCP Bound Info for <%= @value %>" WAN_IP=$new_ip_address WAN_GW=$new_routers WAN_NET=$new_network_number WAN_CDIR=$(mask2cdr $new_subnet_mask) logger -t routing "Using the following: ${WAN_IP}/${WAN_CDIR} and GW ${WAN_GW}" logger -t routing "Flushing <%= @table %> routing rules" ip route flush table <%= @table %> ip route add ${WAN_NET}/${WAN_CDIR} dev <%= @value %> src $ {WAN_IP} table <%= @table %> ip route add default via ${WAN_GW} dev <%= @value %> table < %= @table %> logger -t routing "Updated IP Routing rules for <%= @value %>" fi fi
56. Multiple Uplinks Don’t forget your other local networks! ip route add 192.168.10.0/24 dev management_vlan table isp1 ip route add 192.168.20.0/24 dev lan_vlan table isp1 ip route add 192.168.30.0/24 dev wifi_vlan table isp1 ip route add 192.168.10.0/24 dev management_vlan table isp2 ip route add 192.168.20.0/24 dev lan_vlan table isp2 ip route add 192.168.30.0/24 dev wifi_vlan table isp2
57. Multiple Uplinks Only have the Cache server use the second wan ip rule add from 192.168.10.7/32 table isp2 ip rule add to 192.168.10.7/32 table isp2
58. Traffic Shaping TC – Traffic Control Avoid Bufferbloat by artificially lowering your Download and Upload Dropping packets is GOOD (tcp self regulation)
59. iftop
60. Recomendations • Do a speedtest at the beginning to set a baseline • Keep an eye on ping latency • Structure iptables rules for legibility • Place Game server VMs in the same vlan (if broadcast auto discovery) • Have fun
61. Q&A