Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Gatelogic - Somewhat functional reactive framework in Python
majek04
November 27, 2017
Programming
1
2.9k
Gatelogic - Somewhat functional reactive framework in Python
majek04
November 27, 2017
Tweet
Share
More Decks by majek04
See All by majek04
majek04
0
320
majek04
3
2.3k
majek04
0
240
majek04
3
1.4k
majek04
21
47k
majek04
2
1.7k
majek04
0
1.2k
majek04
1
530
majek04
0
220
Other Decks in Programming
See All in Programming
mathetake
7
2.2k
timeseriesfr
0
140
chatii
2
300
seike460
8
2.2k
wafuwafu13
1
170
devinjeon
2
890
karosuwindam
0
100
hamakou108
4
530
shiba6v
0
210
bkuhlmann
2
330
minodriven
37
29k
ktgrstsh
1
240
Featured
See All Featured
maltzj
502
36k
lynnandtonic
273
16k
paulrobertlloyd
73
1.5k
lara
172
9.8k
addyosmani
312
21k
philhawksworth
192
17k
dotmariusz
94
5.5k
notwaldorf
21
2.1k
cherdarchuk
73
270k
sachag
267
17k
lara
17
2.9k
jakevdp
776
200k
Transcript
Gatelogic FRP framework Marek Majkowski @majek04 ...200 lines of Python
I still regret...
Who we are? 2
Reverse proxy 3 Eyeball Reverse proxy Origin server • Optimizations
• Caching • DDoS protection • Security
Attacked 4
Signal 5 pretty analytics signal Operator switch switch switch
6 Signal
7
8 iptables rules mitigation servers command line Operator Mitigation
9 Mitigation
10 pretty analytics command line iptables mitigation signal Operator servers
switch switch switch
Copy-pasta 11
"Business logic" 12 iptables rules mitigation signal servers switch switch
switch ?
12 months later... 13
14 --ip=1.2.3.4 example.com --ip=1.2.3.4 example.com --qps=100 Business logic
15 --ip=1.2.3.4 example.com --qps=500 example.com = FREE | PAID Business
logic --ip=1.2.3.4 example.com
16 --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500 Business logic example.com subdomains:
(www, ns1, ns2) --ip=1.2.3.4 example.com example.com = FREE | PAID
17 Input Steam extra stream extra stream Output Stream Reactive
Rule
18 Reactive rule def dns_mitigation(attack, plan, subdomains, toggles): domain =
attack['domain'] if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation
Subscriptions 19 def dns_mitigation(attack, plan, subdomains, toggles): domain = attack['domain']
if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation
Business logic • Hard problem! • Multiple DB lookups •
Wait for operator confirmation • Critical path 20
Functional reactive programming 21
22
23 models - Excel
models - Materialized data 24 input output function 00:01h 23:59h
x
models - Signals 25
Pure FRP is useless • Weird language - (ELM anyone?)
• Fixed signal flow • Strictly no side-effects 26
Dirty FRP is awesome 27 • Weird language • Python
• Fixed signal flow • Attacks come and go, but patterns fixed • Strictly no side-effects • Dynamic "subscriptions", but idempotent
Prior art - Trellis 28
29
Gatelogic! 30 https://github.com/cloudflare/gatelogic
Gatelogic • Input - ReadableHub • update(full_data) • Processing -
ComputableHub • maintain(key, function) • unmaintain • Subscriptions - QueryHub • update(full_data) 31
32 { '00001': {ip:'1.2.3.4', port: 80, domain: 'bar.com', '00002': {ip:'1.2.3.5',
port: 80, domain: 'foo.com', ... } Input data - a dict
33 ReadableHub update { 'attack1': 'example.com', ... } update()
34 ComputableHub def on_hook(_, kind, k, row): if kind ==
'add': mitigations.maintain(k, action, row) if kind == 'delete': mitigations.unmaintain(k) subscribe(readable, on_hook) def action(row): return None 34 ReadableHub
35 OutputHub ComputableHub ?
36 OutputHub ComputableHub ReadableHub QueryHub ? database
37 OutputHub ComputableHub ReadableHub QueryHub X X X X X
X Materialized
38 def action(row, plan_hub, subdomain_hub, toggle_hub): domain = row.value if
not domain: return None if toggle_hub.get('all_mitigations_disabled').value != 'True': return None qps = 100 if plan_hub.get(domain).value in ('business', 'b'): qps = 500 sd = (subdomain_hub.get(domain).value or '').split(' ') mitigation = \ domain + \ ' --qps=%s ' % qps + \ ' '.join('--except=%s' % s for s in sd) return mitigation
It works! • Solid foundation! • Composable! • Scalable •
Maintainable • But: • no event loop • lacks higher-order abstractions 39
40
204 loc 41 marek@ubuntu-saucy:~/cloudflare/gatelogic/gatelogic$ cloc *py 3 text files. 3
unique files. 0 files ignored. http://cloc.sourceforge.net v 1.60 T=0.01 s (240.8 files/s, 23197.8 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 3 61 24 204 ------------------------------------------------------------------------------- SUM: 3 61 24 204 -------------------------------------------------------------------------------
Thanks! • FRP is grea • http://www.flapjax-lang.org/ • https://www.youtube.com/watch?v=mEvo6TVAf64 •
https://www.youtube.com/watch?v=Agu6jipKfYw 42 https://github.com/cloudflare/gatelogic marek@cloudflare.com @majek04