Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building an automated DDoS mitigation pipeline
Search
majek04
January 23, 2016
Programming
5
1.3k
Building an automated DDoS mitigation pipeline
majek04
January 23, 2016
Tweet
Share
More Decks by majek04
See All by majek04
BPF programmable socket lookup
majek04
0
540
Linux at Cloudflare
majek04
3
6.8k
DDoS Landscape
majek04
0
350
Inside Cloudbleed
majek04
3
2.4k
Golang sucks
majek04
21
51k
Gatelogic - Somewhat functional reactive framework in Python
majek04
1
4.5k
How Cloudflare deals with largest DDoS attacks?
majek04
2
2.9k
Why we chose Service Worker API
majek04
0
2.3k
IP Spoofing - DEFCON
majek04
1
770
Other Decks in Programming
See All in Programming
しくじり先生 Image Matching Challenge 2024 編
goosehaaan
0
810
Composing an API the *right* way (Droidcon Berlin 2024)
zsmb
1
450
はしめてのプログラミングとロボット制御
watawatavoltage
0
290
Play Billing Library 7.0.0 変更点まとめ@potatotips#88
kako351
0
160
CSC307 Lecture 12
javiergs
PRO
0
220
開発部に不満を持っていたCSがエンジニアにジョブチェンしてわかった「勝手に諦めない」ことの大切さ
sakuraikotone
28
16k
Harnessing Large Language Models for Training-free Video Anomaly Detection
tereka114
1
1.3k
日付と正規化
megmogmog1965
0
140
ドメイン駆動設計の実践
masuda220
PRO
19
5.3k
I/O Extended Android in Korea 2024 ~ Whats new in Android development tools
pluu
0
250
わかりやすい正解を捨てて、コトに向き合う - スクラムフェス金沢2024 スポンサーセッション
yusukekokubo
0
180
Android開発者のための Kotlin Multiplatform入門
ntaro
0
190
Featured
See All Featured
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
105
6.8k
Making the Leap to Tech Lead
cromwellryan
127
8.7k
It's Worth the Effort
3n
181
27k
The Invisible Side of Design
smashingmag
294
50k
Faster Mobile Websites
deanohume
303
30k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
Imperfection Machines: The Place of Print at Facebook
scottboms
262
13k
Stop Working from a Prison Cell
hatefulcrawdad
266
20k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
17
8.7k
Debugging Ruby Performance
tmm1
71
11k
Code Reviewing Like a Champion
maltzj
517
39k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
248
20k
Transcript
Building an automated DDoS Mitigation Pipeline Marek Majkowski
2 "Help Build a Better Internet"
Content neutral 3
DDoS is a threat 4
5 Malicious Attacker Internet Provider Origin Server CloudFlare Server trust
& safety team w orking w ith operators public outreach Big effort im proving our infrastructure
6 Automated DDoS Mitigations Malicious Attacker Internet Provider Origin Server
CloudFlare Server autom ating m itigations
7 attack volume CloudFlare network capacity >
BGP Nullroute and move on 8 ! route 1.2.3.4/32 {!
discard;! community [ 13335:666 13335:668 13335:36006 ];! }!
attack volume CloudFlare network capacity < 9
10 BGP Nullrouting Router firewall Server firewall Application Less damage
Reducing damage
11 BGP Nullrouting IP Router firewall IP, port, packet length
Server firewall all above + stateless DPI parameters Application all above + application logic More precision Reducing damage
12 Operator Precision Speed
13
14 Automation Precision Speed
15 Gatebot Precision Speed Automatic attack handling
Attack Detection Automatic attack handling 16 Mitigation Reactive Automation
The attack 17
High volume packet floods 18 Packets per second
DNS packet flood 19 ! $ tcpdump -ni eth2 inbound
and port 53 -c 100! ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476% [1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2333 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
1 in 10k packets is "real" 20
Finding attack parameters 21 ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476%
[1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2336 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
Mitigation 22 Mitigation Operator
Where to DROP? 23 Application iptables Router
Traffic matching with BPF 24 ! iptables -A INPUT \!
--dst 1.2.3.4 \! -p udp --dport 53 \! -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0" \! -j DROP!
25 ! ldx 4*([14]&0xf)! ld #34! add x! tax! lb_0:!
ldb [x + 0]! add x! add #1! tax! ld [x + 0]! jneq #0x07657861, lb_1! ld [x + 4]! jneq #0x6d706c65, lb_1! ld [x + 8]! jneq #0x03636f6d, lb_1! ldb [x + 12]! jneq #0x00, lb_1! ret #1! lb_1:! ret #0! BPF bytecode
26
Deployment 27 iptables Mitigation Database
Mitigation database 28 ! $ gatekeeper dnsbpf list! --ip=1.2.3.4 *.example.com!
--ip=4.3.2.1 www.test.de *.www.test.de! --ip=4.3.4.4 *.cloudflare.com --except=www.** --except=ns1.**! --ip=2.3.1.4 www.onedomain.com,wwww.seconddomain.com! --ip=1.2.3.0/24 test.com! ! $ gatekeeper dnsbpf add -- --ip=4.3.2.1 *.newattack.com!
Detection 29 Attack Detection
Sflow 30 Sflow Central Aggregation
What is an "attack"? 31
"Attack" is large 32 Large attacks Small attacks Packets per
second
33 Attacks Mitigation "Attack" can be mitigated Attack Detection Mitigation
Database Attack Description = Mitigation 33 iptables Sflow
34 ! Mpps Descr! 3.878 --ip=141.245.59.191/32! 2.878 --ip=141.245.59.192/32! 1.878 --ip=141.245.59.193/32!
1.878 --ip=141.245.59.194/32! 1.878 --ip=141.245.59.195/32! 1.878 --ip=141.245.59.196/32! 1.878 --ip=141.245.59.197/32! 1.878 --ip=141.245.59.198/32! 1.878 --ip=141.245.59.199/32! ...! ! Mpps Descr! 35.878 --ip=141.245.59.0/24! vs "Attacks" shall be aggregated
35 An attack-finding algorithm
Top N / Heavy hitters • Fixed memory size; Algorithm:
Space Saving • https://github.com/cloudflare/golibs 36 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1
Multiple dimensions 37 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24
Multiple dimensions 38 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Multiple dimensions 39 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 reporting threshold: 1M
Attack report 40 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80! 12.2 --ip=1.2.3.4! 2.4 --ip=42.1.2.4! 12.2 --ip=1.2.3.0/24! 2.4 --ip=42.1.2.0/24!
Multiple dimensions 41 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 0.1M 1.2.3.4 0M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 0.1M 1.2.3.0/24 0M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Attack report 42 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80!
Scales well 43
Reactive automation 44 Reactive Automation
Connecting the pieces 45 sflow iptables Attack Detection Mitigation Database
?
46 ! --ip=1.2.3.4 example.com! ! --ip=1.2.3.4 example.com --qps=100! Reactive Rule
47 ! --ip=1.2.3.4 example.com --qps=500! ! example.com = FREE |
PAID! Reactive Rule ! --ip=1.2.3.4 example.com!
48 ! --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500! Reactive Rule !
example.com subdomains:! (www, ns1, ns2)! ! --ip=1.2.3.4 example.com! ! example.com = FREE | PAID!
49 Input Steam extra stream extra stream Output Stream Reactive
Rule
Chain of transformations 50 ! def dns_mitigation(attack, plan, subdomains):! domain
= attack['domain']! ! qps = 100! if plan[domain] == 'business':! qps = 500! ! mitigation =! attack['description'] + \! ' --qps=%s' % qps + \! ' --except=%s'.join(subdomains[domain])! ! return mitigation!
Fully composable 51
Putting it all together 52
Putting it all together 53 Mitigation Database sflow iptables Attack
Detection Reactive Automation 53
Gatebot: frequency 54 Gatebot actions per day 3 months
Gatebot: volume 55 1 week
Summary 56
The fight goes on 57 Malicious Attacker Internet Provider Origin
Server CloudFlare Server trust & safety team w orking w ith operators public outreach im proving our infrastructure
! ! • https://blog.cloudflare.com • https://github.com/cloudflare 58 marek@cloudflare.com @majek04 Thanks!
and good luck! @cfgatebot