LDNWebPerf February 2018 - Michael Tremante

Large Scale DDoS Mitigation
A session presented by Michael Tremante

On Tuesday, 6th February 2018

Skill level: Beginner / Intermediate

Going to walkthrough some real world use cases of large DDoS attacks (including what happened to Donald Trump's website during the US presidential election campaign) and how Cloudflare infrastructure is built to handle the load from a technical perspective.

Transcript

1. (D)DoS Attacks Michael Tremante mst@cloudflare.com
2. Agenda We talked about DDoS in December. Let’s talk about it again! • Quick networking 101 recap • Coding the basic DoS attack • Examples & pretty charts • Donald Trump * disclaimer: I work for Cloudflare
3. Before we start
4. Bandwidth Kilo 1000 Mega 1000 Kilo Giga 1000 Mega Tera 1000 Giga Peta 1000 Tera When talking about throughput: • Big B: Byte • Small b: Bit (usually we use this) x8 difference 1 Mbps => 1 Mega bit per second 1 GBps => 1 Giga byte per second * After all lazy loading has taken place LDNWebPerf Homepage 157KB Google Homepage 963KB BBC Homepage* 4.1MB DVD Movie 1-2GB Blue Ray Movie 4-6GB My hard disk 500GB Keep these examples in mind
5. Google is your friend
6. Network Models 3 4 7 We are going to talk a little about these
7. To DoS or to DDoS... DoS: Denial of Service DDoS: Distributed Denial of Service Aim - disable your application so that your users cannot access it How - your choice Why - retaliation, extortion, distraction Examples of how: overloading CPU, saturating bandwidth, disabling DNS
8. Recently we were DDoS-ing Neteller: https://twitter.com/neteller/status/583363894665715712 Yes, our attacks are powerful. So, it’s your turn! Your site is going under attack unless you pay 40 Bitcoin. Pay to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q Please note that it will not be easy to mitigate our attack, because our current UDP flood power is 400-500 Gbps, so don't even bother. At least, don't expect cheap services like CloudFlare or Incapsula to help...but you can try. :) Right now we are running small demonstrative attack. Don't worry, it will not be that hard (it shouldn't crash your site) and it will stop in 1 hour. It's just to prove that we are serious. We are aware that you probably don't have 40 BTC at the moment, so we are giving you 24 hours. Current price of 1 BTC is about 230 USD, so we are cheap, at the moment. But if you ignore us, price will increase. IMPORTANT: You don’t even have to reply. Just pay 40 BTC to 17PxcnK84x98B9TdEbUvuCeivM7yaH1x4Q – we will know it’s you and you will never hear from us again. We say it because for big companies it's usually the problem as they don't want that there is proof that they cooperated. If you need to contact us, feel free to use some free email service. Or contact us via Bitmessage: BM-NC1jRewNdHxX3jHrufjxDsRWXGdNisY5 But if you ignore us, and don't pay us within a given time, long term attack will start, price to stop will go to 100 BTC and will keep increasing for every hour of attack. IMPORTANT: It’s a one-time payment. Pay and you will not hear from us ever again! We do bad things, but we keep our word.
9. How difficult is it?
11. Types of DDoS - Floods • Does not amplify, and is more successful with a botnet • Leverages a weakness of a protocol/application to overload/consume resources/queues etc. • Some examples: SYN Flood, Ping Flood, UDP Flood and ever more recently HTTP Floods CC BY 2.0 image by Isaí Moreno
12. SYN Floods • TCP connections starts with a 3 way handshake: SYN, SYN-ACK, ACK • Usually implemented with two queues on the server: ◦ SYN Queue ◦ Accept Queue $ sysctl net.core.somaxconn net.core.somaxconn = 128 $ sysctl net.ipv4.tcp_synack_retries net.ipv4.tcp_synack_retries = 5 A SYN Flood aims to fill up this guy
13. Let’s try… • Basic Python knowledge • Advanced Google knowledge • SYN Floods are well understood …. Meet Scapy: a powerful interactive packet manipulation program
14. Scapy SYN Flood • Create IP packet • Create TCP packet • Set SYN flag • Random SRC port • Destination port 80 • Send packets! Need to add firewall rule to stop the OS sending RST packets in response to SYN ACKS firewall-cmd --permanent --direct --add-rule ipv4 filter OUTPUT 0 -p tcp -m tcp -s 151.236.47.154 --tcp-flags RST RST -j DROP
15. Let’s make it happen • Let’s check the size of the SYN queue on the target: watch -n 0.2 "ss -n state syn-recv sport = :80 | wc -l" • To monitor network traffic: tcpdump -i em1 host 46.101.47.105 and tcp port 80 • Let’s fill up the SYN queue… • Test site: http://ldnwebperf.codelocket.com
18. SYN Cookies Let’s enable SYN Cookies in /etc/sysctl.conf net.ipv4.tcp_syncookies = 1 sysctl -p sysctl net.ipv4.tcp_syncookies
19. SYN Cookies TCP packets have a 32 bit sequence number The server normally chooses a random number but with SYN cookies... It crafts (with magic) a special sequence number that encodes some of the parameters sent in the SYN packet concatenated with a cryptographic hash - no need for SYN queue! client replies with ACK and sequence + 1 Server removes 1 from sequence number, validates hash, retrieves parameters and initiates socket SYN received SYN ACK sent ACK received
20. SYN Cookies • Not everything fits in the sequence number ◦ Some use cases are more affected (e.g. mail relays) • Usually (good to check) enabled by default • Very old concept invented by Daniel J. Bernstein and Eric Schenk in September 1996 • Had reasonable performance impact since not too long ago (fixed Jan 2016 with kernel 4.4 release) - kernel now able to handle M of SYN cookies per second • Other workarounds exist: SYN_PROXY iptables module
21. Big SYN floods? • Graph from November 2017 • Peaks at 250M packets per second • Cloudflare sets net.core.somaxconn = 16384 and employs SYN cookies and a number of other kernel optimizations but this is too much • Cloudflare uses p0f to fingerprint packets, converts the fingerprints to BPF format for consumption in our “custom” firewall/iptables (Gatebot)
22. Big SYN floods March 2017, SYN flood (60 GBps = 480 Gbps) 500GB (e.g. my hard disk) of data ever 8.3 seconds in SYN packets
23. Types of DDoS - Amplification The original amplification attack was known as the SMURF attack 1. Control a small botnet capable of sending 100Mbps 2. Send ICMP requests (e.g. ping) to the target network broadcast address (e.g. X.X.X.255) 3. Spoof your source IP to be the one of your victim 4. Router does not verify source (due to no handshake in ICMP) 5. Router forwards request to all devices on the network 6. All devices reply towards target IP 7. Amplification factor is given (more or less) by the number of devices 8. Network goes down! SMURF attacks are no longer a thing… easy to filter
24. DNS Amplification DNS Amplification attacks are still common today 1. Allows you to spoof source IP (over UDP) 2. Response can be larger than the question DNS is a core, ubiquitous Internet platform that meets these criteria and therefore has become the largest source of amplification attacks
25. DNS Amplification • Sample query: dig ANY isc.org @x.x.x.x • Sample response: https://www.codelocket.com/files/large-dns-response.txt • 64 byte query that resulted in a 3,223 byte response • 50x amplification • DNSSEC makes amplification worse
26. Scapy DNS Amplification 1. We need resolvers that reply to ANY queries 2. We need poorly managed resolvers that don’t do filtering 3. We need to send packets from a network that does not do src IP filtering
27. DNS Resolvers
28. DNS Resolver List • Many public lists: https://public-dns.info/ • When I checked nearly 30k servers available over IPv4 • However: ◦ Many servers won’t reply to all DNS query types ◦ Many servers now have rate limiting and other filtering methods deployed ◦ The list needs “cleaning” ◦ E.g. Google 8.8.8.8 does not reply to ANY query for isc.org • Team CYMRU are trying to fix the DNS Resolver problem: ◦ http://www.team-cymru.org/Open-Resolver-Challenge.html
30. Need to create a botnet... • Hardest bit • Let’s take one example… the WireX botnet • About 70k compromised devices on average
33. Donald Trump
34. US Presidential Election • Protected Trump Organization campaign website (donaldjtrump.com) • HTTP flood example • For details please visit: https://www.cloudflare.com/case-studi es/trump/
35. Thanks!