Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building an automated DDoS mitigation pipeline
Search
majek04
January 23, 2016
Programming
5
1.3k
Building an automated DDoS mitigation pipeline
majek04
January 23, 2016
Tweet
Share
More Decks by majek04
See All by majek04
BPF programmable socket lookup
majek04
0
610
Linux at Cloudflare
majek04
3
7.9k
DDoS Landscape
majek04
0
390
Inside Cloudbleed
majek04
3
2.8k
Golang sucks
majek04
21
52k
Gatelogic - Somewhat functional reactive framework in Python
majek04
1
4.9k
How Cloudflare deals with largest DDoS attacks?
majek04
2
3.3k
Why we chose Service Worker API
majek04
0
2.6k
IP Spoofing - DEFCON
majek04
1
930
Other Decks in Programming
See All in Programming
バイブコーディング超えてバイブデプロイ〜CloudflareMCPで実現する、未来のアプリケーションデリバリー〜
azukiazusa1
2
690
Hack Claude Code with Claude Code
choplin
8
2.8k
AIともっと楽するE2Eテスト
myohei
9
3.1k
MDN Web Docs に日本語翻訳でコントリビュートしたくなる
ohmori_yusuke
1
130
What's new in AppKit on macOS 26
1024jp
0
170
構造化・自動化・ガードレール - Vibe Coding実践記 -
tonegawa07
0
140
Caude codeで爆速開発
codelynx
0
100
Git Sync を超える!OSS で実現する CDK Pull 型デプロイ / Deploying CDK with PipeCD in Pull-style
tkikuc
4
450
MCPを使ってイベントソーシングのAIコーディングを効率化する / Streamlining Event Sourcing AI Coding with MCP
tomohisa
0
180
「テストは愚直&&網羅的に書くほどよい」という誤解 / Test Smarter, Not Harder
munetoshi
1
220
レベル1の開発生産性向上に取り組む − 日々の作業の効率化・自動化を通じた改善活動
kesoji
1
350
Claude Code派?Gemini CLI派? みんなで比較LT会!_20250716
junholee
1
690
Featured
See All Featured
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.6k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
It's Worth the Effort
3n
185
28k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
850
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
108
19k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
530
Gamification - CAS2011
davidbonilla
81
5.4k
Building Applications with DynamoDB
mza
95
6.5k
Transcript
Building an automated DDoS Mitigation Pipeline Marek Majkowski
2 "Help Build a Better Internet"
Content neutral 3
DDoS is a threat 4
5 Malicious Attacker Internet Provider Origin Server CloudFlare Server trust
& safety team w orking w ith operators public outreach Big effort im proving our infrastructure
6 Automated DDoS Mitigations Malicious Attacker Internet Provider Origin Server
CloudFlare Server autom ating m itigations
7 attack volume CloudFlare network capacity >
BGP Nullroute and move on 8 ! route 1.2.3.4/32 {!
discard;! community [ 13335:666 13335:668 13335:36006 ];! }!
attack volume CloudFlare network capacity < 9
10 BGP Nullrouting Router firewall Server firewall Application Less damage
Reducing damage
11 BGP Nullrouting IP Router firewall IP, port, packet length
Server firewall all above + stateless DPI parameters Application all above + application logic More precision Reducing damage
12 Operator Precision Speed
13
14 Automation Precision Speed
15 Gatebot Precision Speed Automatic attack handling
Attack Detection Automatic attack handling 16 Mitigation Reactive Automation
The attack 17
High volume packet floods 18 Packets per second
DNS packet flood 19 ! $ tcpdump -ni eth2 inbound
and port 53 -c 100! ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476% [1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2333 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
1 in 10k packets is "real" 20
Finding attack parameters 21 ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476%
[1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2336 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
Mitigation 22 Mitigation Operator
Where to DROP? 23 Application iptables Router
Traffic matching with BPF 24 ! iptables -A INPUT \!
--dst 1.2.3.4 \! -p udp --dport 53 \! -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0" \! -j DROP!
25 ! ldx 4*([14]&0xf)! ld #34! add x! tax! lb_0:!
ldb [x + 0]! add x! add #1! tax! ld [x + 0]! jneq #0x07657861, lb_1! ld [x + 4]! jneq #0x6d706c65, lb_1! ld [x + 8]! jneq #0x03636f6d, lb_1! ldb [x + 12]! jneq #0x00, lb_1! ret #1! lb_1:! ret #0! BPF bytecode
26
Deployment 27 iptables Mitigation Database
Mitigation database 28 ! $ gatekeeper dnsbpf list! --ip=1.2.3.4 *.example.com!
--ip=4.3.2.1 www.test.de *.www.test.de! --ip=4.3.4.4 *.cloudflare.com --except=www.** --except=ns1.**! --ip=2.3.1.4 www.onedomain.com,wwww.seconddomain.com! --ip=1.2.3.0/24 test.com! ! $ gatekeeper dnsbpf add -- --ip=4.3.2.1 *.newattack.com!
Detection 29 Attack Detection
Sflow 30 Sflow Central Aggregation
What is an "attack"? 31
"Attack" is large 32 Large attacks Small attacks Packets per
second
33 Attacks Mitigation "Attack" can be mitigated Attack Detection Mitigation
Database Attack Description = Mitigation 33 iptables Sflow
34 ! Mpps Descr! 3.878 --ip=141.245.59.191/32! 2.878 --ip=141.245.59.192/32! 1.878 --ip=141.245.59.193/32!
1.878 --ip=141.245.59.194/32! 1.878 --ip=141.245.59.195/32! 1.878 --ip=141.245.59.196/32! 1.878 --ip=141.245.59.197/32! 1.878 --ip=141.245.59.198/32! 1.878 --ip=141.245.59.199/32! ...! ! Mpps Descr! 35.878 --ip=141.245.59.0/24! vs "Attacks" shall be aggregated
35 An attack-finding algorithm
Top N / Heavy hitters • Fixed memory size; Algorithm:
Space Saving • https://github.com/cloudflare/golibs 36 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1
Multiple dimensions 37 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24
Multiple dimensions 38 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Multiple dimensions 39 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 reporting threshold: 1M
Attack report 40 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80! 12.2 --ip=1.2.3.4! 2.4 --ip=42.1.2.4! 12.2 --ip=1.2.3.0/24! 2.4 --ip=42.1.2.0/24!
Multiple dimensions 41 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 0.1M 1.2.3.4 0M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 0.1M 1.2.3.0/24 0M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Attack report 42 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80!
Scales well 43
Reactive automation 44 Reactive Automation
Connecting the pieces 45 sflow iptables Attack Detection Mitigation Database
?
46 ! --ip=1.2.3.4 example.com! ! --ip=1.2.3.4 example.com --qps=100! Reactive Rule
47 ! --ip=1.2.3.4 example.com --qps=500! ! example.com = FREE |
PAID! Reactive Rule ! --ip=1.2.3.4 example.com!
48 ! --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500! Reactive Rule !
example.com subdomains:! (www, ns1, ns2)! ! --ip=1.2.3.4 example.com! ! example.com = FREE | PAID!
49 Input Steam extra stream extra stream Output Stream Reactive
Rule
Chain of transformations 50 ! def dns_mitigation(attack, plan, subdomains):! domain
= attack['domain']! ! qps = 100! if plan[domain] == 'business':! qps = 500! ! mitigation =! attack['description'] + \! ' --qps=%s' % qps + \! ' --except=%s'.join(subdomains[domain])! ! return mitigation!
Fully composable 51
Putting it all together 52
Putting it all together 53 Mitigation Database sflow iptables Attack
Detection Reactive Automation 53
Gatebot: frequency 54 Gatebot actions per day 3 months
Gatebot: volume 55 1 week
Summary 56
The fight goes on 57 Malicious Attacker Internet Provider Origin
Server CloudFlare Server trust & safety team w orking w ith operators public outreach im proving our infrastructure
! ! • https://blog.cloudflare.com • https://github.com/cloudflare 58 marek@cloudflare.com @majek04 Thanks!
and good luck! @cfgatebot