Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building an automated DDoS mitigation pipeline
Search
majek04
January 23, 2016
Programming
5
1.3k
Building an automated DDoS mitigation pipeline
majek04
January 23, 2016
Tweet
Share
More Decks by majek04
See All by majek04
BPF programmable socket lookup
majek04
0
610
Linux at Cloudflare
majek04
3
7.9k
DDoS Landscape
majek04
0
390
Inside Cloudbleed
majek04
3
2.8k
Golang sucks
majek04
21
52k
Gatelogic - Somewhat functional reactive framework in Python
majek04
1
4.9k
How Cloudflare deals with largest DDoS attacks?
majek04
2
3.3k
Why we chose Service Worker API
majek04
0
2.6k
IP Spoofing - DEFCON
majek04
1
930
Other Decks in Programming
See All in Programming
QA x AIエコシステム段階構築作戦
osu
0
260
なぜあなたのオブザーバビリティ導入は頓挫するのか
ryota_hnk
5
580
JetBrainsのAI機能の紹介 #jjug
yusuke
0
200
Flutterと Vibe Coding で個人開発!
hyshu
1
240
技術的負債で信頼性が限界だったWordPress運用をShifterで完全復活させた話
rvirus0817
0
1.2k
あまり知られていない MCP 仕様たち / MCP specifications that aren’t widely known
ktr_0731
0
240
LLMOpsのパフォーマンスを支える技術と現場で実践した改善
po3rin
1
140
オホーツクでコミュニティを立ち上げた理由―地方出身プログラマの挑戦 / TechRAMEN 2025 Conference
lemonade_37
2
460
Webinar: AI-Powered Development: Transformiere deinen Workflow mit Coding Tools und MCP Servern
danielsogl
0
110
あのころの iPod を どうにか再生させたい
orumin
2
2.4k
GUI操作LLMの最新動向: UI-TARSと関連論文紹介
kfujikawa
0
760
大規模FlutterプロジェクトのCI実行時間を約8割削減した話
teamlab
PRO
0
460
Featured
See All Featured
Why Our Code Smells
bkeepers
PRO
337
57k
Code Review Best Practice
trishagee
69
19k
A designer walks into a library…
pauljervisheath
207
24k
The Language of Interfaces
destraynor
158
25k
Measuring & Analyzing Core Web Vitals
bluesmoon
8
550
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
Designing for Performance
lara
610
69k
Embracing the Ebb and Flow
colly
86
4.8k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Transcript
Building an automated DDoS Mitigation Pipeline Marek Majkowski
2 "Help Build a Better Internet"
Content neutral 3
DDoS is a threat 4
5 Malicious Attacker Internet Provider Origin Server CloudFlare Server trust
& safety team w orking w ith operators public outreach Big effort im proving our infrastructure
6 Automated DDoS Mitigations Malicious Attacker Internet Provider Origin Server
CloudFlare Server autom ating m itigations
7 attack volume CloudFlare network capacity >
BGP Nullroute and move on 8 ! route 1.2.3.4/32 {!
discard;! community [ 13335:666 13335:668 13335:36006 ];! }!
attack volume CloudFlare network capacity < 9
10 BGP Nullrouting Router firewall Server firewall Application Less damage
Reducing damage
11 BGP Nullrouting IP Router firewall IP, port, packet length
Server firewall all above + stateless DPI parameters Application all above + application logic More precision Reducing damage
12 Operator Precision Speed
13
14 Automation Precision Speed
15 Gatebot Precision Speed Automatic attack handling
Attack Detection Automatic attack handling 16 Mitigation Reactive Automation
The attack 17
High volume packet floods 18 Packets per second
DNS packet flood 19 ! $ tcpdump -ni eth2 inbound
and port 53 -c 100! ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476% [1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2333 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
1 in 10k packets is "real" 20
Finding attack parameters 21 ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476%
[1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2336 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
Mitigation 22 Mitigation Operator
Where to DROP? 23 Application iptables Router
Traffic matching with BPF 24 ! iptables -A INPUT \!
--dst 1.2.3.4 \! -p udp --dport 53 \! -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0" \! -j DROP!
25 ! ldx 4*([14]&0xf)! ld #34! add x! tax! lb_0:!
ldb [x + 0]! add x! add #1! tax! ld [x + 0]! jneq #0x07657861, lb_1! ld [x + 4]! jneq #0x6d706c65, lb_1! ld [x + 8]! jneq #0x03636f6d, lb_1! ldb [x + 12]! jneq #0x00, lb_1! ret #1! lb_1:! ret #0! BPF bytecode
26
Deployment 27 iptables Mitigation Database
Mitigation database 28 ! $ gatekeeper dnsbpf list! --ip=1.2.3.4 *.example.com!
--ip=4.3.2.1 www.test.de *.www.test.de! --ip=4.3.4.4 *.cloudflare.com --except=www.** --except=ns1.**! --ip=2.3.1.4 www.onedomain.com,wwww.seconddomain.com! --ip=1.2.3.0/24 test.com! ! $ gatekeeper dnsbpf add -- --ip=4.3.2.1 *.newattack.com!
Detection 29 Attack Detection
Sflow 30 Sflow Central Aggregation
What is an "attack"? 31
"Attack" is large 32 Large attacks Small attacks Packets per
second
33 Attacks Mitigation "Attack" can be mitigated Attack Detection Mitigation
Database Attack Description = Mitigation 33 iptables Sflow
34 ! Mpps Descr! 3.878 --ip=141.245.59.191/32! 2.878 --ip=141.245.59.192/32! 1.878 --ip=141.245.59.193/32!
1.878 --ip=141.245.59.194/32! 1.878 --ip=141.245.59.195/32! 1.878 --ip=141.245.59.196/32! 1.878 --ip=141.245.59.197/32! 1.878 --ip=141.245.59.198/32! 1.878 --ip=141.245.59.199/32! ...! ! Mpps Descr! 35.878 --ip=141.245.59.0/24! vs "Attacks" shall be aggregated
35 An attack-finding algorithm
Top N / Heavy hitters • Fixed memory size; Algorithm:
Space Saving • https://github.com/cloudflare/golibs 36 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1
Multiple dimensions 37 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24
Multiple dimensions 38 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Multiple dimensions 39 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 reporting threshold: 1M
Attack report 40 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80! 12.2 --ip=1.2.3.4! 2.4 --ip=42.1.2.4! 12.2 --ip=1.2.3.0/24! 2.4 --ip=42.1.2.0/24!
Multiple dimensions 41 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 0.1M 1.2.3.4 0M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 0.1M 1.2.3.0/24 0M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Attack report 42 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80!
Scales well 43
Reactive automation 44 Reactive Automation
Connecting the pieces 45 sflow iptables Attack Detection Mitigation Database
?
46 ! --ip=1.2.3.4 example.com! ! --ip=1.2.3.4 example.com --qps=100! Reactive Rule
47 ! --ip=1.2.3.4 example.com --qps=500! ! example.com = FREE |
PAID! Reactive Rule ! --ip=1.2.3.4 example.com!
48 ! --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500! Reactive Rule !
example.com subdomains:! (www, ns1, ns2)! ! --ip=1.2.3.4 example.com! ! example.com = FREE | PAID!
49 Input Steam extra stream extra stream Output Stream Reactive
Rule
Chain of transformations 50 ! def dns_mitigation(attack, plan, subdomains):! domain
= attack['domain']! ! qps = 100! if plan[domain] == 'business':! qps = 500! ! mitigation =! attack['description'] + \! ' --qps=%s' % qps + \! ' --except=%s'.join(subdomains[domain])! ! return mitigation!
Fully composable 51
Putting it all together 52
Putting it all together 53 Mitigation Database sflow iptables Attack
Detection Reactive Automation 53
Gatebot: frequency 54 Gatebot actions per day 3 months
Gatebot: volume 55 1 week
Summary 56
The fight goes on 57 Malicious Attacker Internet Provider Origin
Server CloudFlare Server trust & safety team w orking w ith operators public outreach im proving our infrastructure
! ! • https://blog.cloudflare.com • https://github.com/cloudflare 58 marek@cloudflare.com @majek04 Thanks!
and good luck! @cfgatebot