LDNWebPerf February 2018 - Michael Tremante

LDNWebPerf February 2018 - Michael Tremante

Large Scale DDoS Mitigation
A session presented by Michael Tremante

On Tuesday, 6th February 2018

Skill level: Beginner / Intermediate

Going to walkthrough some real world use cases of large DDoS attacks (including what happened to Donald Trump's website during the US presidential election campaign) and how Cloudflare infrastructure is built to handle the load from a technical perspective.

27458e8c0002cdbf98765a8486de954d?s=128

London Web Performance Group

February 06, 2018
Tweet

Transcript

  1. (D)DoS Attacks Michael Tremante mst@cloudflare.com

  2. Agenda We talked about DDoS in December. Let’s talk about

    it again! • Quick networking 101 recap • Coding the basic DoS attack • Examples & pretty charts • Donald Trump * disclaimer: I work for Cloudflare
  3. Before we start

  4. Bandwidth Kilo 1000 Mega 1000 Kilo Giga 1000 Mega Tera

    1000 Giga Peta 1000 Tera When talking about throughput: • Big B: Byte • Small b: Bit (usually we use this) x8 difference 1 Mbps => 1 Mega bit per second 1 GBps => 1 Giga byte per second * After all lazy loading has taken place LDNWebPerf Homepage 157KB Google Homepage 963KB BBC Homepage* 4.1MB DVD Movie 1-2GB Blue Ray Movie 4-6GB My hard disk 500GB Keep these examples in mind
  5. Google is your friend

  6. Network Models 3 4 7 We are going to talk

    a little about these
  7. To DoS or to DDoS... DoS: Denial of Service DDoS:

    Distributed Denial of Service Aim - disable your application so that your users cannot access it How - your choice Why - retaliation, extortion, distraction Examples of how: overloading CPU, saturating bandwidth, disabling DNS
  8. Recently we were DDoS-ing Neteller: https://twitter.com/neteller/status/583363894665715712 Yes, our attacks are

    powerful. So, it’s your turn! Your site is going under attack unless you pay 40 Bitcoin. Pay to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q Please note that it will not be easy to mitigate our attack, because our current UDP flood power is 400-500 Gbps, so don't even bother. At least, don't expect cheap services like CloudFlare or Incapsula to help...but you can try. :) Right now we are running small demonstrative attack. Don't worry, it will not be that hard (it shouldn't crash your site) and it will stop in 1 hour. It's just to prove that we are serious. We are aware that you probably don't have 40 BTC at the moment, so we are giving you 24 hours. Current price of 1 BTC is about 230 USD, so we are cheap, at the moment. But if you ignore us, price will increase. IMPORTANT: You don’t even have to reply. Just pay 40 BTC to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q – we will know it’s you and you will never hear from us again. We say it because for big companies it's usually the problem as they don't want that there is proof that they cooperated. If you need to contact us, feel free to use some free email service. Or contact us via Bitmessage: BM-NC1jRewNdHxX3jHrufjxDsRWXGdNisY5 But if you ignore us, and don't pay us within a given time, long term attack will start, price to stop will go to 100 BTC and will keep increasing for every hour of attack. IMPORTANT: It’s a one-time payment. Pay and you will not hear from us ever again! We do bad things, but we keep our word.
  9. How difficult is it?

  10. None
  11. Types of DDoS - Floods • Does not amplify, and

    is more successful with a botnet • Leverages a weakness of a protocol/application to overload/consume resources/queues etc. • Some examples: SYN Flood, Ping Flood, UDP Flood and ever more recently HTTP Floods CC BY 2.0 image by Isaí Moreno
  12. SYN Floods • TCP connections starts with a 3 way

    handshake: SYN, SYN-ACK, ACK • Usually implemented with two queues on the server: ◦ SYN Queue ◦ Accept Queue $ sysctl net.core.somaxconn net.core.somaxconn = 128 $ sysctl net.ipv4.tcp_synack_retries net.ipv4.tcp_synack_retries = 5 A SYN Flood aims to fill up this guy
  13. Let’s try… • Basic Python knowledge • Advanced Google knowledge

    • SYN Floods are well understood …. Meet Scapy: a powerful interactive packet manipulation program
  14. Scapy SYN Flood • Create IP packet • Create TCP

    packet • Set SYN flag • Random SRC port • Destination port 80 • Send packets! Need to add firewall rule to stop the OS sending RST packets in response to SYN ACKS firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -m tcp -s 151.236.47.154 --tcp-flags RST RST -j DROP
  15. Let’s make it happen • Let’s check the size of

    the SYN queue on the target: watch -n 0.2 "ss -n state syn-recv sport = :80 | wc -l" • To monitor network traffic: tcpdump -i em1 host 46.101.47.105 and tcp port 80 • Let’s fill up the SYN queue… • Test site: http://ldnwebperf.codelocket.com
  16. None
  17. None
  18. SYN Cookies Let’s enable SYN Cookies in /etc/sysctl.conf net.ipv4.tcp_syncookies =

    1 sysctl -p sysctl net.ipv4.tcp_syncookies
  19. SYN Cookies TCP packets have a 32 bit sequence number

    The server normally chooses a random number but with SYN cookies... It crafts (with magic) a special sequence number that encodes some of the parameters sent in the SYN packet concatenated with a cryptographic hash - no need for SYN queue! client replies with ACK and sequence + 1 Server removes 1 from sequence number, validates hash, retrieves parameters and initiates socket SYN received SYN ACK sent ACK received
  20. SYN Cookies • Not everything fits in the sequence number

    ◦ Some use cases are more affected (e.g. mail relays) • Usually (good to check) enabled by default • Very old concept invented by Daniel J. Bernstein and Eric Schenk in September 1996 • Had reasonable performance impact since not too long ago (fixed Jan 2016 with kernel 4.4 release) - kernel now able to handle M of SYN cookies per second • Other workarounds exist: SYN_PROXY iptables module
  21. Big SYN floods? • Graph from November 2017 • Peaks

    at 250M packets per second • Cloudflare sets net.core.somaxconn = 16384 and employs SYN cookies and a number of other kernel optimizations but this is too much • Cloudflare uses p0f to fingerprint packets, converts the fingerprints to BPF format for consumption in our “custom” firewall/iptables (Gatebot)
  22. Big SYN floods March 2017, SYN flood (60 GBps =

    480 Gbps) 500GB (e.g. my hard disk) of data ever 8.3 seconds in SYN packets
  23. Types of DDoS - Amplification The original amplification attack was

    known as the SMURF attack 1. Control a small botnet capable of sending 100Mbps 2. Send ICMP requests (e.g. ping) to the target network broadcast address (e.g. X.X.X.255) 3. Spoof your source IP to be the one of your victim 4. Router does not verify source (due to no handshake in ICMP) 5. Router forwards request to all devices on the network 6. All devices reply towards target IP 7. Amplification factor is given (more or less) by the number of devices 8. Network goes down! SMURF attacks are no longer a thing… easy to filter
  24. DNS Amplification DNS Amplification attacks are still common today 1.

    Allows you to spoof source IP (over UDP) 2. Response can be larger than the question DNS is a core, ubiquitous Internet platform that meets these criteria and therefore has become the largest source of amplification attacks
  25. DNS Amplification • Sample query: dig ANY isc.org @x.x.x.x •

    Sample response: https://www.codelocket.com/files/large-dns-response.txt • 64 byte query that resulted in a 3,223 byte response • 50x amplification • DNSSEC makes amplification worse
  26. Scapy DNS Amplification 1. We need resolvers that reply to

    ANY queries 2. We need poorly managed resolvers that don’t do filtering 3. We need to send packets from a network that does not do src IP filtering
  27. DNS Resolvers

  28. DNS Resolver List • Many public lists: https://public-dns.info/ • When

    I checked nearly 30k servers available over IPv4 • However: ◦ Many servers won’t reply to all DNS query types ◦ Many servers now have rate limiting and other filtering methods deployed ◦ The list needs “cleaning” ◦ E.g. Google 8.8.8.8 does not reply to ANY query for isc.org • Team CYMRU are trying to fix the DNS Resolver problem: ◦ http://www.team-cymru.org/Open-Resolver-Challenge.html
  29. None
  30. Need to create a botnet... • Hardest bit • Let’s

    take one example… the WireX botnet • About 70k compromised devices on average
  31. None
  32. None
  33. Donald Trump

  34. US Presidential Election • Protected Trump Organization campaign website (donaldjtrump.com)

    • HTTP flood example • For details please visit: https://www.cloudflare.com/case-studi es/trump/
  35. Thanks!