Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building an automated DDoS mitigation pipeline
Search
majek04
January 23, 2016
Programming
5
1.3k
Building an automated DDoS mitigation pipeline
majek04
January 23, 2016
Tweet
Share
More Decks by majek04
See All by majek04
BPF programmable socket lookup
majek04
0
560
Linux at Cloudflare
majek04
3
7k
DDoS Landscape
majek04
0
360
Inside Cloudbleed
majek04
3
2.5k
Golang sucks
majek04
21
52k
Gatelogic - Somewhat functional reactive framework in Python
majek04
1
4.6k
How Cloudflare deals with largest DDoS attacks?
majek04
2
3k
Why we chose Service Worker API
majek04
0
2.3k
IP Spoofing - DEFCON
majek04
1
790
Other Decks in Programming
See All in Programming
Debugging: A Few Things You Should Know
jmatsu
1
130
CSC509 Lecture 06
javiergs
PRO
0
130
Serverless renderování Webových komponent
rarous
PRO
0
140
Mercari AI/LLM Hackathon TeamBの発表資料
imaikosuke
0
190
もう実家に手頃な情シス娘は不要!Bedrockでもう一人の娘を作る
komakichi
2
150
文化が生産性を作る
jimpei
3
630
CSC305 Lecture 09
javiergs
PRO
0
120
のびしろを広げる巻き込まれ力:偶然を活かすキャリアの作り方/oso2024
takahashiikki
1
250
2024-10-01 dev2next - Observability for Modern JVM Applications
jonatan_ivanov
1
140
4年間変わらなかった YOUTRUSTのアーキテクチャ
daiki1003
2
680
/←このスケジュール表に立ち向かう フロントエンド開発戦略 / A front-end development strategy to tackle a single-slash schedule.
nrslib
1
430
Honoの来た道とこれから
yusukebe
14
2.2k
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
296
20k
Speed Design
sergeychernyshev
23
550
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Building Your Own Lightsaber
phodgson
102
6k
A designer walks into a library…
pauljervisheath
202
24k
It's Worth the Effort
3n
183
27k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
43
6.6k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
228
52k
Optimising Largest Contentful Paint
csswizardry
31
2.9k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
260
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.8k
Transcript
Building an automated DDoS Mitigation Pipeline Marek Majkowski
2 "Help Build a Better Internet"
Content neutral 3
DDoS is a threat 4
5 Malicious Attacker Internet Provider Origin Server CloudFlare Server trust
& safety team w orking w ith operators public outreach Big effort im proving our infrastructure
6 Automated DDoS Mitigations Malicious Attacker Internet Provider Origin Server
CloudFlare Server autom ating m itigations
7 attack volume CloudFlare network capacity >
BGP Nullroute and move on 8 ! route 1.2.3.4/32 {!
discard;! community [ 13335:666 13335:668 13335:36006 ];! }!
attack volume CloudFlare network capacity < 9
10 BGP Nullrouting Router firewall Server firewall Application Less damage
Reducing damage
11 BGP Nullrouting IP Router firewall IP, port, packet length
Server firewall all above + stateless DPI parameters Application all above + application logic More precision Reducing damage
12 Operator Precision Speed
13
14 Automation Precision Speed
15 Gatebot Precision Speed Automatic attack handling
Attack Detection Automatic attack handling 16 Mitigation Reactive Automation
The attack 17
High volume packet floods 18 Packets per second
DNS packet flood 19 ! $ tcpdump -ni eth2 inbound
and port 53 -c 100! ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476% [1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2333 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
1 in 10k packets is "real" 20
Finding attack parameters 21 ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476%
[1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2336 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
Mitigation 22 Mitigation Operator
Where to DROP? 23 Application iptables Router
Traffic matching with BPF 24 ! iptables -A INPUT \!
--dst 1.2.3.4 \! -p udp --dport 53 \! -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0" \! -j DROP!
25 ! ldx 4*([14]&0xf)! ld #34! add x! tax! lb_0:!
ldb [x + 0]! add x! add #1! tax! ld [x + 0]! jneq #0x07657861, lb_1! ld [x + 4]! jneq #0x6d706c65, lb_1! ld [x + 8]! jneq #0x03636f6d, lb_1! ldb [x + 12]! jneq #0x00, lb_1! ret #1! lb_1:! ret #0! BPF bytecode
26
Deployment 27 iptables Mitigation Database
Mitigation database 28 ! $ gatekeeper dnsbpf list! --ip=1.2.3.4 *.example.com!
--ip=4.3.2.1 www.test.de *.www.test.de! --ip=4.3.4.4 *.cloudflare.com --except=www.** --except=ns1.**! --ip=2.3.1.4 www.onedomain.com,wwww.seconddomain.com! --ip=1.2.3.0/24 test.com! ! $ gatekeeper dnsbpf add -- --ip=4.3.2.1 *.newattack.com!
Detection 29 Attack Detection
Sflow 30 Sflow Central Aggregation
What is an "attack"? 31
"Attack" is large 32 Large attacks Small attacks Packets per
second
33 Attacks Mitigation "Attack" can be mitigated Attack Detection Mitigation
Database Attack Description = Mitigation 33 iptables Sflow
34 ! Mpps Descr! 3.878 --ip=141.245.59.191/32! 2.878 --ip=141.245.59.192/32! 1.878 --ip=141.245.59.193/32!
1.878 --ip=141.245.59.194/32! 1.878 --ip=141.245.59.195/32! 1.878 --ip=141.245.59.196/32! 1.878 --ip=141.245.59.197/32! 1.878 --ip=141.245.59.198/32! 1.878 --ip=141.245.59.199/32! ...! ! Mpps Descr! 35.878 --ip=141.245.59.0/24! vs "Attacks" shall be aggregated
35 An attack-finding algorithm
Top N / Heavy hitters • Fixed memory size; Algorithm:
Space Saving • https://github.com/cloudflare/golibs 36 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1
Multiple dimensions 37 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24
Multiple dimensions 38 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Multiple dimensions 39 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 reporting threshold: 1M
Attack report 40 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80! 12.2 --ip=1.2.3.4! 2.4 --ip=42.1.2.4! 12.2 --ip=1.2.3.0/24! 2.4 --ip=42.1.2.0/24!
Multiple dimensions 41 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 0.1M 1.2.3.4 0M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 0.1M 1.2.3.0/24 0M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Attack report 42 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80!
Scales well 43
Reactive automation 44 Reactive Automation
Connecting the pieces 45 sflow iptables Attack Detection Mitigation Database
?
46 ! --ip=1.2.3.4 example.com! ! --ip=1.2.3.4 example.com --qps=100! Reactive Rule
47 ! --ip=1.2.3.4 example.com --qps=500! ! example.com = FREE |
PAID! Reactive Rule ! --ip=1.2.3.4 example.com!
48 ! --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500! Reactive Rule !
example.com subdomains:! (www, ns1, ns2)! ! --ip=1.2.3.4 example.com! ! example.com = FREE | PAID!
49 Input Steam extra stream extra stream Output Stream Reactive
Rule
Chain of transformations 50 ! def dns_mitigation(attack, plan, subdomains):! domain
= attack['domain']! ! qps = 100! if plan[domain] == 'business':! qps = 500! ! mitigation =! attack['description'] + \! ' --qps=%s' % qps + \! ' --except=%s'.join(subdomains[domain])! ! return mitigation!
Fully composable 51
Putting it all together 52
Putting it all together 53 Mitigation Database sflow iptables Attack
Detection Reactive Automation 53
Gatebot: frequency 54 Gatebot actions per day 3 months
Gatebot: volume 55 1 week
Summary 56
The fight goes on 57 Malicious Attacker Internet Provider Origin
Server CloudFlare Server trust & safety team w orking w ith operators public outreach im proving our infrastructure
! ! • https://blog.cloudflare.com • https://github.com/cloudflare 58 marek@cloudflare.com @majek04 Thanks!
and good luck! @cfgatebot