Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Open Source Lan Party Gateway - LOADays 2018

Open Source Lan Party Gateway - LOADays 2018

Presentation on how to setup a Lan Party gateway server.

Used components:

Ubuntu/Centos
dnsmasq
bind9
isc-dhcp-server
LXC
openvSwitch
tc
iptables
squid
collectd
graphite
grafana

Christophe Vanlancker

April 21, 2018
Tweet

More Decks by Christophe Vanlancker

Other Decks in Technology

Transcript

  1. Open­source
    LAN Party
    Gateway
    Christophe Vanlancker ( @Carroarmato0 )

    View Slide

  2. Organizations
    ICT-Blue Lan: 2011, 2012, 2013 ~40 gamers
    Zanzilan: 15w ~80 gamers, 15z ~130 gamers

    View Slide

  3. Some History

    80’s: “PCs invade the home”

    90’s:
    – Cheaper networking equipment
    – Cheaper PCs
    – Rise of popular games: Quake, Age of Empires, Warcraft, Unreal
    Tournament, …

    20xx:
    – Online gaming: League of Legends, World of Warcraft, PUBG, Fortnite,
    Runscape,...

    View Slide

  4. The Basics

    An Internet Connection

    Router

    Switch

    DNS

    DHCP

    View Slide

  5. Advanced

    Services
    – Virtualization/Containerization
    – Monitoring/Metrics
    – Game servers

    HA

    Caching

    DNS overriding

    Link Aggregation

    Multiple UPLinks

    QoS/Throttling

    Network segregation (VLANs)

    View Slide

  6. Used Technology

    OS: Centos/Ubuntu

    DNS: dnsmasq / Bind

    DHCP: dnsmasq / ISC DHCP

    Containerization: LXC

    Firewall: iptables

    Traffic Shaping: tc

    Forward Proxy: Squid

    Metrics: Graphite, Collectd, Grafana

    Networking: Open vSwitch

    View Slide

  7. Router
    Enable routing functionality

    On the fly
    sysctl -w net.ipv4.ip_forward=1
    or
    echo 1 > /proc/sys/net/ipv4/ip_forward

    Permanent:
    /etc/sysctl.conf:
    net.ipv4.ip_forward = 1

    View Slide

  8. Router
    NAT
    Makes packets leaving through your uplink use the public IP of
    the router so that replies can return to it.
    – iptables -t nat -A POSTROUTING -j MASQUERADE -o
    or
    – iptables -t nat -A POSTROUTING -j SNAT -o --to

    View Slide

  9. Router
    NAT
    SNAT
    – Requires you to pass the IP of your public interface
    – Keeps track of connections when interface is brought down and back up
    MASQUERADE
    – You just need to pass the name of the public interface
    – Handy when using DHCP and you don't know your public IP beforehand or if it might change over
    time
    – Overhead with respect to SNAT as the IP of the device needs to be looked up every packet

    View Slide

  10. Open vSwitch

    Multilayer virtual switch

    Network automation

    Standard management
    interfaces and protocols (e.g.
    NetFlow, sFlow, IPFIX,
    RSPAN, CLI, LACP, 802.1ag)

    Supports distribution across
    multiple physical servers

    View Slide

  11. Open vSwitch

    ovs-vswitchd (daemon)

    ovsdb-server (daemon)

    ovs-vsctl

    ovs-appctl

    View Slide

  12. Open vSwitch
    Create a bridge
    ovs-vsctl add-br internal_br
    Create fake bridges with vlan tags
    ovs-vsctl add-br management_br internal_br 10
    ovs-vsctl add-br lan_br internal_br 20
    ovs-vsctl add-br wifi_br internal_br 30
    Create a bond device with physical interfaces and attach it to the bridge
    ovs-vsctl add-bond internal_br bond0 enp0s9 enp0s10 lacp=active
    Add interface to the bridge
    ovs-vsctl add-port interal_br enp0s3

    View Slide

  13. Open vSwitch
    ovs-vsctl show

    View Slide

  14. Open vSwitch
    ip link show

    View Slide

  15. Open vSwitch
    Bonding – combining multiple nics

    LACP
    – Requires LACP protocol on both ends
    – OVS falls back to active-backup

    Active Backup
    – Only 1 NIC is active, fallback to secondary

    SLB
    – Allows some limited form of loadbalancing
    – Use this with dumb switches

    balance-tcp

    View Slide

  16. Open vSwitch
    ovs-appctl bond/show bond0
    ovs-appctl lacp/show bond0

    View Slide

  17. DHCP
    dnsmasq
    Pro:
    – Small and simple
    – DNS included
    Con:
    – More geard towards desktop (dns caching) and small networks
    isc-dhcp-server
    Pro:
    – The go-to industry standard
    – Highly configurable
    Con (?):
    – Dynamic DNS decoupled

    View Slide

  18. DHCP
    dnsmasq
    # Set domain
    domain=zanzi.lan
    # Upstream DNS servers
    server=8.8.8.8
    server=8.8.4.4
    # Prevent non-routable private addresses from being forwarded
    bogus-priv
    # Prepend domain to all hosts
    expand-hosts
    # Do not read /etc/hosts
    no-hosts
    # Read hosts.dnsmasq for hosts entries
    addn-hosts=/etc/hosts.dnsmasq
    # Read /etc/ethers for static mac to ip entries
    read-ethers
    # Only bind to interfaces that it's listening on
    bind-interfaces
    # Be authoritative and barge in when a machine wakes up and broadcasts's a dhcp request
    dhcp-authoritative

    View Slide

  19. DHCP
    dnsmasq
    # Set listening interface, dhcp range and lease time
    dhcp-range=lan_vlan,192.168.20.6,192.168.20.254,8h
    # Pass default gateway
    dhcp-option=lan_vlan,3,192.168.20.1
    # Pass DNS server
    dhcp-option=lan_vlan,6,192.168.20.1,192.168.20.2
    # Pass search domain
    dhcp-option=lan_vlan,119,zanzi.lan
    # Pass NTP server
    dhcp-option=lan_vlan,42,192.168.20.1,192.168.20.2

    View Slide

  20. DHCP
    dnsmasq
    # /etc/ethers
    00:07:ef:00:00:03 192.168.10.3
    00:07:ef:00:00:04 192.168.10.4
    00:07:ef:00:00:07 192.168.10.7
    or
    # /etc/dnsmasq.conf
    dhcp-host=00:07:ef:00:00:03,192.168.10.3
    dhcp-host=00:07:ef:00:00:04,192.168.10.4
    dhcp-host=00:07:ef:00:00:07,192.168.10.7

    View Slide

  21. DHCP
    isc-dhcp-server
    # ----------
    # dhcpd.conf
    # ----------
    authoritative;
    #omapi-port 7911;
    default-lease-time 43200;
    max-lease-time 86400;
    log-facility daemon;
    option domain-name "zanzi.lan";
    option domain-name-servers 192.169.10.1, 192.169.20.1;
    option fqdn.no-client-update on; # set the "O" and "S" flag bits
    option fqdn.rcode2 255;
    option ntp-servers none;
    ddns-update-style none;
    include "/etc/dhcp/dhcpd.pools";
    include "/etc/dhcp/dhcpd.hosts";
    include "/etc/dhcp/dhcpd.ignoredsubnets";

    View Slide

  22. DHCP
    isc-dhcp-server
    #################################
    # lan pool 192.169.20.0 255.255.255.0
    #################################
    subnet 192.169.20.0 netmask 255.255.255.0 {
    pool
    {
    range 192.169.20.5 192.169.20.254;
    }
    option domain-name "zanzi.lan";
    option subnet-mask 255.255.255.0;
    option routers 192.169.20.1;
    option domain-name-servers 192.169.20.1;
    }

    View Slide

  23. DHCP
    isc-dhcp-server
    subnet 192.168.10.0 netmask 255.255.255.0 {

    host zanzi-web {
    hardware ethernet 00:07:ef:00:00:03;
    fixed-address 192.168.10.3;
    }
    host zanzi-share {
    hardware ethernet 00:07:ef:00:00:04;
    fixed-address 192.168.10.4;
    }
    host zanzi-cache {
    hardware ethernet 00:07:ef:00:00:07;
    fixed-address 192.168.10.7;
    }
    }

    View Slide

  24. DNS
    dnsmasq
    Pro:
    – Small and simple
    – DNS included with Dynamic Hosts
    Con:
    – More geard towards desktop (dns caching) and small networks
    bind9
    Pro:
    – The go-to industry standard
    – Highly configurable
    Con (?):
    – Dynamic DNS decoupled
    – Harder to configure

    View Slide

  25. DNS
    dnsmasq
    # Set domain
    domain=zanzi.lan
    # Upstream DNS servers
    server=8.8.8.8
    server=8.8.4.4
    # Prevent non-routable private addresses from being forwarded
    bogus-priv
    # Prepend domain to all hosts
    expand-hosts
    # Do not read /etc/hosts
    no-hosts
    # Read hosts.dnsmasq for hosts entries
    addn-hosts=/etc/hosts.dnsmasq
    # Read /etc/ethers for static mac to ip entries
    read-ethers
    # Only bind to interfaces that it's listening on
    bind-interfaces
    # Be authoritative and barge in when a machine wakes up and broadcasts's a
    dhcp request
    dhcp-authoritative

    View Slide

  26. DNS
    dnsmasq
    # Steam
    address=/.cs.steampowered.com/192.168.10.3
    address=/.steamcontent.com/192.168.10.3
    address=/content1.steampowered.com/192.168.10.3
    address=/content2.steampowered.com/192.168.10.3
    address=/content3.steampowered.com/192.168.10.3
    address=/content4.steampowered.com/192.168.10.3
    address=/content5.steampowered.com/192.168.10.3
    address=/content6.steampowered.com/192.168.10.3
    address=/content7.steampowered.com/192.168.10.3
    address=/content8.steampowered.com/192.168.10.3
    address=/clientconfig.akamai.steamstatic.com/192.168.10.3
    address=/hsar.steampowered.com.edgesuite.net/192.168.10.3

    View Slide

  27. DNS
    bind9
    options {
    listen-on port 53 { any; };
    forwarders { 1.1.1.1; };
    directory "/var/named";
    dump-file "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    allow-query { 192.169.10.0/24; 192.169.20.0/24; };
    recursion no;
    };
    zone "169.192.in-addr.arpa" IN {
    type master;
    file "169.192.in-addr.arpa";
    allow-update { key DHCP_UPDATER };
    };
    zone "zanzi.lan" IN {
    type master;
    file "zanzi.lan";
    allow-update { key DHCP_UPDATER };
    };

    View Slide

  28. LXC

    Full OS Containers

    10x greater density than
    KVM

    57% less latency than KVM

    94% faster instance
    launching than KVM

    View Slide

  29. LXC

    View Slide

  30. LXC

    lxc profile list

    lxc profile show default

    lxc profile edit default

    lxc profile apply ,…

    lxc init/launch ubuntu: webserver -t
    c-m

    lxc start/stop webserver

    lxc exec webserver bash
    config:
    user.network_mode: dhcp
    description: Default LXD profile
    devices:
    eth0:
    name: eth0
    nictype: bridged
    parent: management_vlan
    type: nic
    name: default
    used_by: []

    View Slide

  31. LXC
    Preconfigured Network

    lxc init ubuntu: zanzi-web

    lxc config device add zanzi-web eth0 nic nictype=bridged parent=management_vlan name=eth0
    hwaddr=00:07:ef:00:00:03

    lxc start zanzi-web

    lxc list

    View Slide

  32. Squid

    Caching proxy

    HTTP, HTTPS, FTP, …

    ACLs

    Domain level blocking

    View Slide

  33. Squid
    Allow access acl management_vlan src 192.168.10.0/24
    acl lan_vlan src 192.168.20.0/24
    acl wifi_vlan src 192.168.30.0/24
    http_access allow management_vlan
    http_access allow lan_vlan
    http_access allow wifi_vlan

    View Slide

  34. Squid
    Ports and packet redirection
    http_port 8080 transparent
    https_port 8443 intercept ssl-bump generate-host-certificates=on
    dynamic_cert_mem_cache_size=16MB cert=/etc/squid/ssl_cert/server.pem

    View Slide

  35. Squid
    Chain PREROUTING (policy ACCEPT 1485 packets, 138K bytes)
    pkts bytes target prot opt in out source destination
    0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080
    0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080
    0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 80 redir ports 8080
    0 0 REDIRECT tcp -- * * 192.168.10.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443
    0 0 REDIRECT tcp -- * * 192.168.20.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443
    0 0 REDIRECT tcp -- * * 192.168.30.0/24 0.0.0.0/0 multiport dports 443 redir ports 8443

    View Slide

  36. Squid
    HTTP caching
    cache_mem 4 GB
    maximum_object_size_in_memory 2048 KB
    cache_dir aufs /var/cache/squid 30000 16 256
    maximum_object_size 10 MB
    minimum_object_size 4 KB

    View Slide

  37. Squid
    HTTPS caching
    Cannot be done unless you perform a MITM
    # Become a TCP tunnel without decrypting proxied traffic
    ssl_bump splice

    View Slide

  38. Squid
    HTTPS caching – but if you really want to….
    # Connect to server first, generate mimicked cert
    ssl_bump server-first all
    # sslcrtd
    sslcrtd_program /usr/lib64/squid/ssl_crtd -s /var/lib/ssl_db/ -M 4MB -b 4 KB
    sslcrtd_children 8 startup=1 idle=1
    sslproxy_cert_error allow all
    sslproxy_flags DONT_VERIFY_PEER

    View Slide

  39. Squid
    People will need to accept your CA Certificate so that the on-
    the-fly signed certs produced by Squid are accepted by the OS
    and browser.
    Captive Portal?

    View Slide

  40. Squid
    Blocking domains
    acl blockeddomain dstdomain "/etc/squid/blocked.domains.acl"
    speedtest.net
    fast.com
    speedtest.telenet.be
    pxs.speedtestcustom.com
    http_access deny blockeddomain

    View Slide

  41. Squid
    Blocking domains – SquidGuard
    dest porn {
    domainlist porn/domains
    urllist porn/urls
    }
    dest warez {
    domainlist warez/domains
    urllist warez/urls
    }

    View Slide

  42. Squid
    acl {
    default {
    pass !porn !warez all
    redirect http:///block.html
    }
    }
    url_rewrite_program /usr/local/bin/squidGuard -c /usr/local/squidGuard/squidGuard.conf

    View Slide

  43. Squid
    Rate Limiting
    acl netflix dstdomain .nflxvideo.net .netflix.com .nflximg.net .nflxext.com .llnwd.net .fast.com
    acl youtube dstdomain .googlevideo.com .youtube.com .ytimg.com
    delay_pools 2 # 2 delay pools
    delay_initial_bucket_level 50
    # netflix
    delay_class 1 3 # pool 1 is a class 3 pool
    delay_parameters 1 none 400000/400000
    delay_access 1 allow netflix management_vlan
    delay_access 1 allow netflix lan_vlan
    delay_access 1 allow netflix wifi_vlan
    delay_access 1 deny all

    View Slide

  44. Squid
    # youtube
    delay_class 2 3 # pool 2 is a class 3 pool
    delay_parameters 2 none 400000/400000
    delay_access 2 allow youtube management_vlan
    delay_access 2 allow youtube lan_vlan
    delay_access 2 allow youtube wifi_vlan
    delay_access 2 deny all

    View Slide

  45. Metrics

    Collectd
    – Plugin based metric gathering daemon
    – 100+ plugins: Apache, CPU, Load, Network, Ping, ….

    Graphite
    – Store numeric time-series data
    – Render graphs of this data on demand
    – echo "foo.bar 1 `date +%s`" | nc localhost 2003

    Grafana
    – Query metrics from Graphite (and more!) and generate dashboards

    View Slide

  46. Metrics
    CollectD
    FQDNLookup true
    Interval 120
    LoadPlugin df
    LoadPlugin network
    LoadPlugin load
    LoadPlugin memory
    LoadPlugin write_graphite


    Host "localhost"
    Port "2003"
    Prefix "collectd."
    StoreRates true
    AlwaysAppendDS false
    EscapeCharacter "_"


    View Slide

  47. Metrics
    Graphite
    – carbon-cache
    – whisper
    – graphite-web

    View Slide

  48. Metrics

    View Slide

  49. Caching Games

    DNS overriding

    Nginx

    https://github.com/multiplay/lancache
    – Blizzard
    – Steam
    – Origin
    – Riot
    – Sony
    – Windows Updates
    – …..

    View Slide

  50. Caching Games
    DNS # Steam
    address=/.cs.steampowered.com/192.168.10.3
    address=/.steamcontent.com/192.168.10.3
    address=/content1.steampowered.com/192.168.10.3
    …..
    zone "steampowered.com" IN {
    type master;
    file "steam";
    };

    View Slide

  51. Caching Games
    Nginx ●
    Requests are sent through Nginx

    Will check on disk if it already has the same
    requested data

    If not, acts as a caching forwarding proxy (like
    Squid)

    Saves terabytes of data passing through your
    uplink

    View Slide

  52. Multiple Uplinks

    DHCP client by default requests on all connected interfaces
    – Will override the default gateway
    – Will override your configured nameserver in /etc/resolv.conf
    – Last connected uplink becomes the primary/default

    We only want enough information for routing
    Write custom dhcp hooks / Use correct flags for your interface

    View Slide

  53. Multiple Uplinks
    /etc/dhcp/dhclient.conf
    # Global
    request subnet-mask, broadcast-address, time-offset, interface-
    mtu, rfc3442-classless-static-routes;
    # Primary
    interface "eth0" { request routers; }

    View Slide

  54. Multiple Uplinks
    Routing tables
    # /etc/iproute2/rt_tables
    1 isp1
    2 isp2

    View Slide

  55. Multiple Uplinks
    mask2cdr ()
    {
    # Assumes there's no "255." after a non-255 byte in
    the mask
    local x=${1##*255.}
    set -- 0^^^128^192^224^240^248^252^254^ $(( ($
    {#1} - ${#x})*2 )) ${x%%.*}
    x=${1%%$3*}
    echo $(( $2 + (${#x}/4) ))
    }
    logger -t routing "Received DHCP Trigger for <%=
    @value %>"
    printenv > /tmp/dhcp-<%= @value %>
    if [ "$interface" = "<%= @value %>" ]; then
    if [ "$reason" = "BOUND" ]; then
    logger -t routing "Received DHCP Bound Info for <%= @value %>"
    WAN_IP=$new_ip_address
    WAN_GW=$new_routers
    WAN_NET=$new_network_number
    WAN_CDIR=$(mask2cdr $new_subnet_mask)
    logger -t routing "Using the following: ${WAN_IP}/${WAN_CDIR} and
    GW ${WAN_GW}"
    logger -t routing "Flushing <%= @table %> routing rules"
    ip route flush table <%= @table %>
    ip route add ${WAN_NET}/${WAN_CDIR} dev <%= @value %> src $
    {WAN_IP} table <%= @table %>
    ip route add default via ${WAN_GW} dev <%= @value %> table <
    %= @table %>
    logger -t routing "Updated IP Routing rules for <%= @value %>"
    fi
    fi

    View Slide

  56. Multiple Uplinks
    Don’t forget your other local networks!
    ip route add 192.168.10.0/24 dev management_vlan table isp1
    ip route add 192.168.20.0/24 dev lan_vlan table isp1
    ip route add 192.168.30.0/24 dev wifi_vlan table isp1
    ip route add 192.168.10.0/24 dev management_vlan table isp2
    ip route add 192.168.20.0/24 dev lan_vlan table isp2
    ip route add 192.168.30.0/24 dev wifi_vlan table isp2

    View Slide

  57. Multiple Uplinks
    Only have the Cache server use the second wan
    ip rule add from 192.168.10.7/32 table isp2
    ip rule add to 192.168.10.7/32 table isp2

    View Slide

  58. Traffic Shaping
    TC – Traffic Control
    Avoid Bufferbloat by
    artificially lowering your
    Download and Upload
    Dropping packets is GOOD
    (tcp self regulation)

    View Slide

  59. iftop

    View Slide

  60. Recomendations

    Do a speedtest at the beginning to set a baseline

    Keep an eye on ping latency

    Structure iptables rules for legibility

    Place Game server VMs in the same vlan (if broadcast auto
    discovery)

    Have fun

    View Slide

  61. Q&A

    View Slide