LDNWebPerf February 2018 - Michael Tremante

Large Scale DDoS Mitigation
A session presented by Michael Tremante

On Tuesday, 6th February 2018

Skill level: Beginner / Intermediate

Going to walkthrough some real world use cases of large DDoS attacks (including what happened to Donald Trump's website during the US presidential election campaign) and how Cloudflare infrastructure is built to handle the load from a technical perspective.

Transcript

1. (D)DoS Attacks
   Michael Tremante
   mst@cloudflare.com
   

   View Slide

2. Agenda
   We talked about DDoS in December. Let’s talk about it again!
   ● Quick networking 101 recap
   ● Coding the basic DoS attack
   ● Examples & pretty charts
   ● Donald Trump
   * disclaimer: I work for Cloudflare
   

   View Slide

3. Before we start
   

   View Slide

4. Bandwidth Kilo 1000
   Mega 1000 Kilo
   Giga 1000 Mega
   Tera 1000 Giga
   Peta 1000 Tera
   When talking about throughput:
   ● Big B: Byte
   ● Small b: Bit (usually we use this)
   x8 difference
   1 Mbps => 1 Mega bit per second
   1 GBps => 1 Giga byte per second
   * After all lazy loading has taken place
   LDNWebPerf Homepage 157KB
   Google Homepage 963KB
   BBC Homepage* 4.1MB
   DVD Movie 1-2GB
   Blue Ray Movie 4-6GB
   My hard disk 500GB
   Keep these
   examples in mind
   

   View Slide

5. Google is your friend
   

   View Slide

6. Network Models
   3
   4
   7 We are going
   to talk a little
   about these
   

   View Slide

7. To DoS or to DDoS...
   DoS: Denial of Service
   DDoS: Distributed Denial of Service
   Aim - disable your application so that your users cannot access it
   How - your choice
   Why - retaliation, extortion, distraction
   Examples of how: overloading CPU, saturating bandwidth, disabling DNS
   

   View Slide

8. Recently we were DDoS-ing Neteller:
   https://twitter.com/neteller/status/583363894665715712
   Yes, our attacks are powerful.
   So, it’s your turn!
   Your site is going under attack unless you pay 40 Bitcoin.
   Pay to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q
   Please note that it will not be easy to mitigate our attack, because our current UDP flood power is
   400-500 Gbps, so don't even bother. At least, don't expect
   cheap services like CloudFlare or Incapsula to help...but you can try. :)
   Right now we are running small demonstrative attack. Don't worry, it will not be that hard (it shouldn't crash your site) and it will stop in 1 hour. It's just
   to prove that we are serious.
   We are aware that you probably don't have 40 BTC at the moment, so we are giving you 24 hours. Current price of 1 BTC is about 230 USD, so we are cheap, at the
   moment. But if you ignore us, price will increase.
   IMPORTANT: You don’t even have to reply. Just pay 40 BTC to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q – we will know it’s you and you will never hear from us again. We
   say it because for big companies it's usually the problem as they don't want that there is proof that they cooperated. If you need to contact us, feel free to
   use some free email service. Or contact us via Bitmessage: BM-NC1jRewNdHxX3jHrufjxDsRWXGdNisY5
   But if you ignore us, and don't pay us within a given time, long term attack will start, price to stop will go to 100 BTC and will keep increasing for every hour
   of attack.
   IMPORTANT: It’s a one-time payment. Pay and you will not hear from us ever again!
   We do bad things, but we keep our word.
   

   View Slide

9. How difficult is it?
   

   View Slide

10. View Slide

11. Types of DDoS - Floods
    ● Does not amplify, and is more
    successful with a botnet
    ● Leverages a weakness of a
    protocol/application to
    overload/consume
    resources/queues etc.
    ● Some examples: SYN Flood, Ping
    Flood, UDP Flood and ever more
    recently HTTP Floods
    CC BY 2.0 image by Isaí Moreno
    

    View Slide

12. SYN Floods
    ● TCP connections starts with a 3 way
    handshake: SYN, SYN-ACK, ACK
    ● Usually implemented with two
    queues on the server:
    ○ SYN Queue
    ○ Accept Queue
    $ sysctl net.core.somaxconn
    net.core.somaxconn = 128
    $ sysctl net.ipv4.tcp_synack_retries
    net.ipv4.tcp_synack_retries = 5
    A SYN Flood aims to fill up this guy
    

    View Slide

13. Let’s try…
    ● Basic Python knowledge
    ● Advanced Google knowledge
    ● SYN Floods are well understood
    ….
    Meet Scapy:
    a powerful interactive packet manipulation program
    

    View Slide

14. Scapy SYN Flood
    ● Create IP packet
    ● Create TCP packet
    ● Set SYN flag
    ● Random SRC port
    ● Destination port 80
    ● Send packets!
    Need to add firewall rule to stop the OS sending RST packets in response to SYN ACKS
    firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -m tcp -s 151.236.47.154
    --tcp-flags RST RST -j DROP
    

    View Slide

15. Let’s make it happen
    ● Let’s check the size of the SYN queue on the target:
    watch -n 0.2 "ss -n state syn-recv sport = :80 | wc -l"
    ● To monitor network traffic:
    tcpdump -i em1 host 46.101.47.105 and tcp port 80
    ● Let’s fill up the SYN queue…
    ● Test site:
    http://ldnwebperf.codelocket.com
    

    View Slide

16. View Slide

17. View Slide

18. SYN Cookies
    Let’s enable SYN Cookies in /etc/sysctl.conf
    net.ipv4.tcp_syncookies = 1
    sysctl -p
    sysctl net.ipv4.tcp_syncookies
    

    View Slide

19. SYN Cookies
    TCP packets have a 32 bit sequence number
    The server normally chooses a random number
    but with SYN cookies...
    It crafts (with magic) a special sequence number that encodes some of
    the parameters sent in the SYN packet concatenated with a
    cryptographic hash - no need for SYN queue!
    client replies with ACK and sequence + 1
    Server removes 1 from sequence number, validates hash, retrieves
    parameters and initiates socket
    SYN received
    SYN ACK sent
    ACK received
    

    View Slide

20. SYN Cookies
    ● Not everything fits in the sequence number
    ○ Some use cases are more affected (e.g. mail relays)
    ● Usually (good to check) enabled by default
    ● Very old concept invented by Daniel J. Bernstein and Eric Schenk in
    September 1996
    ● Had reasonable performance impact since not too long ago (fixed Jan 2016
    with kernel 4.4 release) - kernel now able to handle M of SYN cookies per
    second
    ● Other workarounds exist: SYN_PROXY iptables module
    

    View Slide

21. Big SYN floods?
    ● Graph from November 2017
    ● Peaks at 250M packets per second
    ● Cloudflare sets net.core.somaxconn = 16384 and employs SYN
    cookies and a number of other kernel optimizations but this is too much
    ● Cloudflare uses p0f to fingerprint packets, converts the fingerprints to BPF
    format for consumption in our “custom” firewall/iptables (Gatebot)
    

    View Slide

22. Big SYN floods
    March 2017, SYN flood (60 GBps = 480 Gbps)
    500GB (e.g. my hard disk) of data ever 8.3 seconds in SYN packets
    

    View Slide

23. Types of DDoS - Amplification
    The original amplification attack was known as the SMURF attack
    1. Control a small botnet capable of sending 100Mbps
    2. Send ICMP requests (e.g. ping) to the target network broadcast address (e.g. X.X.X.255)
    3. Spoof your source IP to be the one of your victim
    4. Router does not verify source (due to no handshake in ICMP)
    5. Router forwards request to all devices on the network
    6. All devices reply towards target IP
    7. Amplification factor is given (more or less) by the number of devices
    8. Network goes down!
    SMURF attacks are no longer a thing… easy to filter
    

    View Slide

24. DNS Amplification
    DNS Amplification attacks are still common today
    1. Allows you to spoof source IP (over UDP)
    2. Response can be larger than the question
    DNS is a core, ubiquitous Internet platform that meets these
    criteria and therefore has become the largest source of
    amplification attacks
    

    View Slide

25. DNS Amplification
    ● Sample query: dig ANY isc.org @x.x.x.x
    ● Sample response:
    https://www.codelocket.com/files/large-dns-response.txt
    ● 64 byte query that resulted in a 3,223 byte response
    ● 50x amplification
    ● DNSSEC makes amplification worse
    

    View Slide

26. Scapy DNS Amplification
    1. We need resolvers that reply to ANY queries
    2. We need poorly managed resolvers that don’t do filtering
    3. We need to send packets from a network that does not do src IP filtering
    

    View Slide

27. DNS Resolvers
    

    View Slide

28. DNS Resolver List
    ● Many public lists: https://public-dns.info/
    ● When I checked nearly 30k servers available over IPv4
    ● However:
    ○ Many servers won’t reply to all DNS query types
    ○ Many servers now have rate limiting and other filtering methods deployed
    ○ The list needs “cleaning”
    ○ E.g. Google 8.8.8.8 does not reply to ANY query for isc.org
    ● Team CYMRU are trying to fix the DNS Resolver problem:
    ○ http://www.team-cymru.org/Open-Resolver-Challenge.html
    

    View Slide

29. View Slide

30. Need to create a botnet...
    ● Hardest bit
    ● Let’s take one example… the WireX botnet
    ● About 70k compromised devices on average
    

    View Slide

31. View Slide

32. View Slide

33. Donald Trump
    

    View Slide

34. US Presidential Election
    ● Protected Trump Organization
    campaign website (donaldjtrump.com)
    ● HTTP flood example
    ● For details please visit:
    https://www.cloudflare.com/case-studi
    es/trump/
    

    View Slide

35. Thanks!
    

    View Slide