Tweet
Tweet

Slide 1

Slide 1 text

How Cloudflare deals with largest DDoS attacks Marek Majkowski @majek04

Slide 2

Slide 2 text

2

Slide 3

Slide 3 text

Reverse proxy 3 Eyeball Reverse proxy Origin server • Optimizations • Caching • DDoS protection • Security

Slide 4

Slide 4 text

4

Slide 5

Slide 5 text

5

Slide 6

Slide 6 text

How to perform an attack? 6

Slide 7

Slide 7 text

7 X Attacker Visitor Target Bots

Slide 8

Slide 8 text

1. Get a cool costume 8

Slide 9

Slide 9 text

9 2. Fix your keyboard

Slide 10

Slide 10 text

3. Get a cool name 10 • Anonymous • Lizard Squad • WireX • Syrian Electronic Army • vDOS

Slide 11

Slide 11 text

4. Choose attack type • L3 Amplification (requires IP Spoofing) • L3 Direct • L7 - repetitive • L7 - smart 11

Slide 12

Slide 12 text

Attack types • Amplification • SYN • DNS • HTTP • HTTPS 12 ⟯ ⟯ OSI L3 (source IP irrelevant) OSI L7 (established TCP/IP)

Slide 13

Slide 13 text

Attack types 13 IP Spoofing Large Easy to block Easy to implement L3 Amplification required YES yes yes L3 Direct desired no depends depends L7 repetitive no yes yes L7 smart no no no

Slide 14

Slide 14 text

14 more bots == better

Slide 15

Slide 15 text

5. Choose infection vector • Ask users - hacktivism • Windows XP zombies • VPS Servers - Shellshock • CPE routers - Mirai • Android phones - WireX 15

Slide 16

Slide 16 text

Infection vectors 16 IP Spoofing Plenty? Infection software Windows XP maybe, hard to maintain acquire? Zeus VPS Servers sometimes internet scan ? CPE Routers sometimes new vulnerability custom Android phones yes phishing custom

Slide 17

Slide 17 text

17 https://www.fbi.gov/wanted/cyber

Slide 18

Slide 18 text

18

Slide 19

Slide 19 text

Five case studies 19

Slide 20

Slide 20 text

1. Asia direct 20

Slide 21

Slide 21 text

21

Slide 22

Slide 22 text

22 Target Server Attacker 510 Gbps Direct SYN flood

Slide 23

Slide 23 text

23

Slide 24

Slide 24 text

24 1 in 10000 packets

Slide 25

Slide 25 text

Internet economics 25 Tier 1 provider Attacking network Target network $ $

Slide 26

Slide 26 text

26 Spoofed? (source: DaPuglet)

Slide 27

Slide 27 text

27 5.6.7.8 8.8.8.8 IP Spoofing

Slide 28

Slide 28 text

28 Enables impersonation Real 8.8.8.8 Destination 5.6.7.8 Spoofed 8.8.8.8

Slide 29

Slide 29 text

29

Slide 30

Slide 30 text

30

Slide 31

Slide 31 text

31

Slide 32

Slide 32 text

32

Slide 33

Slide 33 text

33

Slide 34

Slide 34 text

34

Slide 35

Slide 35 text

35

Slide 36

Slide 36 text

36

Slide 37

Slide 37 text

Asia direct: Profile • Direct SYN floods • VERY big - 510Gbps, 300Mpps • Hitting small number of pops • Capable of IP Spoofing • Attacker: ???? • Infection: Servers 37

Slide 38

Slide 38 text

Is it a day job? 38

Slide 39

Slide 39 text

Asia: mitigation 39 iptables -A INPUT \ --dst 1.2.3.4 \ -p udp --dport 53 \ -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0" \ -j DROP

Slide 40

Slide 40 text

BPF Bytecode 40 ldx 4*([14]&0xf) ld #34 add x tax lb_0: ldb [x + 0] add x add #1 tax ld [x + 0] jneq #0x07657861, lb_1 ld [x + 4] jneq #0x6d706c65, lb_1 ld [x + 8] jneq #0x03636f6d, lb_1 ldb [x + 12] jneq #0x00, lb_1 ret #1 lb_1: ret #0

Slide 41

Slide 41 text

41

Slide 42

Slide 42 text

42 Internet Router NIC Kernel App XDP Iptables

Slide 43

Slide 43 text

Application integration 43 1.2.3.4 1.2.3.5 1.2.3.6 dig A example.com 1.2.3.7 X

Slide 44

Slide 44 text

2. Asia amplification 44

Slide 45

Slide 45 text

45

Slide 46

Slide 46 text

46 UDP Server UDP Client request response

Slide 47

Slide 47 text

47 Attacker Target UDP Server request response

Slide 48

Slide 48 text

48 Attacker Target UDP Server request response 10 bytes 100 bytes

Slide 49

Slide 49 text

49 Attacker Target UDP Servers requests responses

Slide 50

Slide 50 text

50

Slide 51

Slide 51 text

51 192.0.2.0/24 Internet Los Angeles 192.0.2.0/24 London 192.0.2.0/24 Amsterdam 192.0.2.0/24 Moscow 192.0.2.0/24 San Jose 192.0.2.0/24 New York

Slide 52

Slide 52 text

52

Slide 53

Slide 53 text

March 2013: Spamhaus 53 300 Gbps of traffic 27 Gbps of spoofing Exposed DNS Resolvers

Slide 54

Slide 54 text

54

Slide 55

Slide 55 text

55 Mbps - AMP sport=1900 min:406 avg:22608 med=4663 max:186124 dev:35565 count:1319 Mbps - AMP sport=1900: value |-------------------------------------------------- count 134 | 0 268 | *************************************** 145 536 |************************************************** 182 1073 | ******************************************** 163 2147 | ******************************************* 157 4294 | *********************************** 131 8589 | ************************ 88 17179 | ************************************************ 176 34359 | ************************************ 133 68719 | ******************************** 117 137438 | ******* 27

Slide 56

Slide 56 text

Asia SSDP: Profile • Amplification SSDP attakcs • Pretty large - 186Gbps • Hitting LARGE number of pops • (we know) source is in Asia, Interesting targets • Requires IP Spoofing • Attacker: ???? • Infection: Servers 56

Slide 57

Slide 57 text

57

Slide 58

Slide 58 text

58

Slide 59

Slide 59 text

59 https://badupnp.benjojo.co.uk/

Slide 60

Slide 60 text

3. IoT - connected cameras 60

Slide 61

Slide 61 text

61

Slide 62

Slide 62 text

62

Slide 63

Slide 63 text

63

Slide 64

Slide 64 text

64 IP reputation (source: the internet)

Slide 65

Slide 65 text

65

Slide 66

Slide 66 text

66

Slide 67

Slide 67 text

67 GET /en HTTP/1.1 User-Agent: Cookie: Host: example.com Connection: close Content-Length: 800000 a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=&a[]=&b[]=...

Slide 68

Slide 68 text

IoT: Profile • Repetitive L7 attacks • Pretty large - 700k rps, 400Gbps, • 128k source IP's • Ukraine, Vietnam • Attacker: ???? • Infection: CCTV Botnet 68

Slide 69

Slide 69 text

69 Internet Router NIC Kernel App conntrack iptables fingerprints

Slide 70

Slide 70 text

4. Porcupine 70

Slide 71

Slide 71 text

71

Slide 72

Slide 72 text

72

Slide 73

Slide 73 text

73

Slide 74

Slide 74 text

74

Slide 75

Slide 75 text

75

Slide 76

Slide 76 text

Porcupine: Profile • Junk payload L7 attacks • Pretty large - 1M rps, 200k IP's/h • Brasil, Algeria, Tunisia, Ukraine • Attacker: . • Infection: . 76

Slide 77

Slide 77 text

77

Slide 78

Slide 78 text

5. WireX 78

Slide 79

Slide 79 text

79

Slide 80

Slide 80 text

80 User-Agent: jigpuzbcomkenhvladtwysqfxr User-Agent: yudjmikcvzoqwsbflghtxpanre User-Agent: mckvhaflwzbderiysoguxnqtpj User-Agent: deogjvtynmcxzwfsbahirukqpl User-Agent: fdmjczoeyarnuqkbgtlivsxhwp User-Agent: yczfxlrenuqtwmavhojpigkdsb User-Agent: dnlseufokcgvmajqzpbtrwyxih

Slide 81

Slide 81 text

81

Slide 82

Slide 82 text

82

Slide 83

Slide 83 text

83

Slide 84

Slide 84 text

84 function attack(String target, String userAgent, String referer) { HashMap WebViewHeaders = new HashMap(); WebViewHeaders->put(“Referer”,referer); WebViewHeaders->put(“X-Requested-With”,””); WebView[] AttackerViews = new WebView[100]; for (int i=0; iclearHistory(); AttackerViews[i]->clearFormData(); AttackerViews[i]->clearCache(true); WebViewSettings AWVS = AttackerViews[i]->getSettings() AttackWebViewSettings->setJavaScriptEnabled(true); AttackWebViewSettings->setUserAgentString(userAgent); AttackWebViewSettings->setCacheMode(LOAD_NO_CACHE); this->deleteDatabase(“webview.db”); this->deleteDatabase(“webviewCache.db”); AttackerViews[i]->loadUrl(target,WebViewHeaders); } } }

Slide 85

Slide 85 text

WireX: Profile • Real-looking L7 requests • Mid-size - 80k rps, 120k IP's/h • Location: Everywhere • Targets: interesting • Attacker: . • Infection: 300+ Android apps 85

Slide 86

Slide 86 text

6. Elections 86

Slide 87

Slide 87 text

87

Slide 88

Slide 88 text

Attack data • trump.com: ~54,000 attacks per day • donaldjtrump.com: ~501,000 attacks per day • trump.com: Attacked on 100% of days • donaldjtrump.com: Attacked on 94.4% of days 88

Slide 89

Slide 89 text

Attack data • trump.com: • 25 Apr 2016 ~3,4M requests (39 rps) • donaldjtrump.com: • 10 Dec 2015 ~15M requests (173 rps) 89

Slide 90

Slide 90 text

90

Slide 91

Slide 91 text

91

Slide 92

Slide 92 text

7. The great cannon 92

Slide 93

Slide 93 text

93 Eyeball Target Infected website

Slide 94

Slide 94 text

94

Slide 95

Slide 95 text

Infected website? • Compromised JS • Competition case • China great firewall • L3 injection to scripts • Rogue AD 95

Slide 96

Slide 96 text

Cloudflare 96

Slide 97

Slide 97 text

97 1. Architecture

Slide 98

Slide 98 text

98 2. Mitigation in software

Slide 99

Slide 99 text

99 Attack Detection Mitigation Automation

Slide 100

Slide 100 text

Thanks! • Architected for DDoS • Iptables + BPF great for L3 • Kernel bypass • XDP • Iptables + Conntrack great for L7 • Also: connlimit, hashlimits, ipset 100 marek@cloudflare.com @majek04