Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building an automated DDoS mitigation pipeline
Search
majek04
January 23, 2016
Programming
5
1.3k
Building an automated DDoS mitigation pipeline
majek04
January 23, 2016
Tweet
Share
More Decks by majek04
See All by majek04
BPF programmable socket lookup
majek04
0
510
Linux at Cloudflare
majek04
3
6.6k
DDoS Landscape
majek04
0
340
Inside Cloudbleed
majek04
3
2.4k
Golang sucks
majek04
21
51k
Gatelogic - Somewhat functional reactive framework in Python
majek04
1
4.3k
How Cloudflare deals with largest DDoS attacks?
majek04
2
2.8k
Why we chose Service Worker API
majek04
0
2.2k
IP Spoofing - DEFCON
majek04
1
720
Other Decks in Programming
See All in Programming
デフォルトにして至高、RubyMineの大好きな所
ruzia
0
160
サイコロで理解する統計的仮説検定の考え方
tatamiya
4
830
try! Swift Tokyo 2024のLT枠に採択されたプロポーザルを出すときに考えていたこと
ski
0
350
Netty Chicago Java User Group 2024-04-17
sullis
0
170
MicrosoftのPlatform Engineeringガイドを読んで実際になにかやってみた
ymd65536
1
230
検証も兼ねて個人開発でHonoとかと向き合った話
hanetsuki
0
170
単体テストを書かない技術 #phpcon_odawara
o0h
PRO
26
8.1k
スキーマ駆動開発による品質とスピードの両立 - 私達は何故、スキーマを書くのか
kentaroutakeda
0
160
0→1と1→10の狭間で Javaという技術選定を振り返る/Reflecting on the Decision to Choose Java Between Scaling from 0 to 1 and 1 to 10
jaguar_imo
2
380
Komplexe Oberflächen mit SVG und der Web Animation API
joergneumann
0
670
使ってみよう Azure AI Document Intelligence
kosmosebi
2
260
Ruby GitHub Packages
bkuhlmann
0
630
Featured
See All Featured
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
243
20k
Optimizing for Happiness
mojombo
370
69k
Code Review Best Practice
trishagee
54
15k
10 Git Anti Patterns You Should be Aware of
lemiorhan
647
58k
How to train your dragon (web standard)
notwaldorf
72
5.1k
Mobile First: as difficult as doing things right
swwweet
216
8.6k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
The MySQL Ecosystem @ GitHub 2015
samlambert
242
12k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3k
No one is an island. Learnings from fostering a developers community.
thoeni
15
2.1k
Testing 201, or: Great Expectations
jmmastey
27
6.3k
Embracing the Ebb and Flow
colly
79
4.1k
Transcript
Building an automated DDoS Mitigation Pipeline Marek Majkowski
2 "Help Build a Better Internet"
Content neutral 3
DDoS is a threat 4
5 Malicious Attacker Internet Provider Origin Server CloudFlare Server trust
& safety team w orking w ith operators public outreach Big effort im proving our infrastructure
6 Automated DDoS Mitigations Malicious Attacker Internet Provider Origin Server
CloudFlare Server autom ating m itigations
7 attack volume CloudFlare network capacity >
BGP Nullroute and move on 8 ! route 1.2.3.4/32 {!
discard;! community [ 13335:666 13335:668 13335:36006 ];! }!
attack volume CloudFlare network capacity < 9
10 BGP Nullrouting Router firewall Server firewall Application Less damage
Reducing damage
11 BGP Nullrouting IP Router firewall IP, port, packet length
Server firewall all above + stateless DPI parameters Application all above + application logic More precision Reducing damage
12 Operator Precision Speed
13
14 Automation Precision Speed
15 Gatebot Precision Speed Automatic attack handling
Attack Detection Automatic attack handling 16 Mitigation Reactive Automation
The attack 17
High volume packet floods 18 Packets per second
DNS packet flood 19 ! $ tcpdump -ni eth2 inbound
and port 53 -c 100! ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476% [1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2333 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
1 in 10k packets is "real" 20
Finding attack parameters 21 ! IP 202.194.181.95.15443 > 1.2.3.4:53: 63476%
[1au] A? example.com. (50)! IP 221.12.236.115.6570 > 1.2.3.4:53: 11406% [1au] A? example.com. (50)! IP 203.94.134.43.18473 > 1.2.3.4:53: 8559% [1au] A? example.com. (50)! IP 203.196.66.75.32573 > 1.2.3.4:53: 47971% [1au] A? example.com. (50)! IP 124.240.198.136.2336 > 1.2.3.4:53: 61152% [1au] A? example.com. (50)! IP 218.247.70.185.11679 > 1.2.3.4:53: 16360% [1au] A? example.com. (50)! IP 202.109.218.98.27549 > 1.2.3.4:53: 17829% [1au] A? example.com. (50)! IP 203.148.240.82.21825 > 1.2.3.4:53: 22590% [1au] A? example.com. (50)! IP 211.167.108.67.25782 > 1.2.3.4:53: 17663% [1au] A? example.com. (50)! IP 203.209.60.18.20221 > 1.2.3.4:53: 38257% [1au] A? example.com. (50)! IP 203.81.181.168.12749 > 1.2.3.4:53: 53492% [1au] A? example.com. (50)!
Mitigation 22 Mitigation Operator
Where to DROP? 23 Application iptables Router
Traffic matching with BPF 24 ! iptables -A INPUT \!
--dst 1.2.3.4 \! -p udp --dport 53 \! -m bpf --bytecode "14,0 0 0 20,177 0 0 0,12 0 0 0,7 0 0 0,64 0 0 0,21 0 7 124090465,64 0 0 4,21 0 5 1836084325,64 0 0 8,21 0 3 56848237,80 0 0 12,21 0 1 0,6 0 0 1,6 0 0 0" \! -j DROP!
25 ! ldx 4*([14]&0xf)! ld #34! add x! tax! lb_0:!
ldb [x + 0]! add x! add #1! tax! ld [x + 0]! jneq #0x07657861, lb_1! ld [x + 4]! jneq #0x6d706c65, lb_1! ld [x + 8]! jneq #0x03636f6d, lb_1! ldb [x + 12]! jneq #0x00, lb_1! ret #1! lb_1:! ret #0! BPF bytecode
26
Deployment 27 iptables Mitigation Database
Mitigation database 28 ! $ gatekeeper dnsbpf list! --ip=1.2.3.4 *.example.com!
--ip=4.3.2.1 www.test.de *.www.test.de! --ip=4.3.4.4 *.cloudflare.com --except=www.** --except=ns1.**! --ip=2.3.1.4 www.onedomain.com,wwww.seconddomain.com! --ip=1.2.3.0/24 test.com! ! $ gatekeeper dnsbpf add -- --ip=4.3.2.1 *.newattack.com!
Detection 29 Attack Detection
Sflow 30 Sflow Central Aggregation
What is an "attack"? 31
"Attack" is large 32 Large attacks Small attacks Packets per
second
33 Attacks Mitigation "Attack" can be mitigated Attack Detection Mitigation
Database Attack Description = Mitigation 33 iptables Sflow
34 ! Mpps Descr! 3.878 --ip=141.245.59.191/32! 2.878 --ip=141.245.59.192/32! 1.878 --ip=141.245.59.193/32!
1.878 --ip=141.245.59.194/32! 1.878 --ip=141.245.59.195/32! 1.878 --ip=141.245.59.196/32! 1.878 --ip=141.245.59.197/32! 1.878 --ip=141.245.59.198/32! 1.878 --ip=141.245.59.199/32! ...! ! Mpps Descr! 35.878 --ip=141.245.59.0/24! vs "Attacks" shall be aggregated
35 An attack-finding algorithm
Top N / Heavy hitters • Fixed memory size; Algorithm:
Space Saving • https://github.com/cloudflare/golibs 36 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1
Multiple dimensions 37 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24
Multiple dimensions 38 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Multiple dimensions 39 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 12.2M 1.2.3.4 2.4M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 12.2M 1.2.3.0/24 2.4M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 reporting threshold: 1M
Attack report 40 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80! 12.2 --ip=1.2.3.4! 2.4 --ip=42.1.2.4! 12.2 --ip=1.2.3.0/24! 2.4 --ip=42.1.2.0/24!
Multiple dimensions 41 pps IP:port 12.2M 1.2.3.4:53 2.4M 42.1.2.4:80 0.01M
2.4.3.1:80 0.01M 192.168.1.1:443 pps IP 0.1M 1.2.3.4 0M 42.1.2.4 0.01M 2.4.3.1 0.01M 192.168.1.1 pps subnet 0.1M 1.2.3.0/24 0M 42.1.2.0/24 0.01M 2.4.3.0/24 0.01M 192.168.1.0/24 incoming sample: 42.1.2.4:80
Attack report 42 ! Mpps Descr! 12.2 --ip=1.2.3.4 --port=53! 2.4
--ip=42.1.2.4 --port=80!
Scales well 43
Reactive automation 44 Reactive Automation
Connecting the pieces 45 sflow iptables Attack Detection Mitigation Database
?
46 ! --ip=1.2.3.4 example.com! ! --ip=1.2.3.4 example.com --qps=100! Reactive Rule
47 ! --ip=1.2.3.4 example.com --qps=500! ! example.com = FREE |
PAID! Reactive Rule ! --ip=1.2.3.4 example.com!
48 ! --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500! Reactive Rule !
example.com subdomains:! (www, ns1, ns2)! ! --ip=1.2.3.4 example.com! ! example.com = FREE | PAID!
49 Input Steam extra stream extra stream Output Stream Reactive
Rule
Chain of transformations 50 ! def dns_mitigation(attack, plan, subdomains):! domain
= attack['domain']! ! qps = 100! if plan[domain] == 'business':! qps = 500! ! mitigation =! attack['description'] + \! ' --qps=%s' % qps + \! ' --except=%s'.join(subdomains[domain])! ! return mitigation!
Fully composable 51
Putting it all together 52
Putting it all together 53 Mitigation Database sflow iptables Attack
Detection Reactive Automation 53
Gatebot: frequency 54 Gatebot actions per day 3 months
Gatebot: volume 55 1 week
Summary 56
The fight goes on 57 Malicious Attacker Internet Provider Origin
Server CloudFlare Server trust & safety team w orking w ith operators public outreach im proving our infrastructure
! ! • https://blog.cloudflare.com • https://github.com/cloudflare 58 marek@cloudflare.com @majek04 Thanks!
and good luck! @cfgatebot