Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
Gatelogic - Somewhat functional reactive framework in Python
majek04
November 27, 2017
Programming
1
2.9k
Gatelogic - Somewhat functional reactive framework in Python
majek04
November 27, 2017
Tweet
Share
More Decks by majek04
See All by majek04
majek04
0
310
majek04
3
2.2k
majek04
0
240
majek04
3
1.3k
majek04
21
46k
majek04
2
1.7k
majek04
0
1.2k
majek04
1
500
majek04
0
210
Other Decks in Programming
See All in Programming
standfm
0
170
dulltz
0
520
yaamaa
0
430
hanakla
2
3k
thatjeffsmith
0
430
akatsukinewgrad
0
200
siketyan
1
110
pdone
0
210
dictoss
0
170
chichou
1
840
masayaaoyama
4
540
shigeruoda
0
480
Featured
See All Featured
tenderlove
52
3.4k
paulrobertlloyd
71
3.6k
rocio
155
11k
dougneiner
119
7.8k
lauravandoore
437
28k
roundedbygravity
84
7.8k
jlugia
216
16k
schacon
145
6.6k
samanthasiow
56
6.3k
michaelherold
225
8.5k
shpigford
368
42k
lara
590
61k
Transcript
Gatelogic FRP framework Marek Majkowski @majek04 ...200 lines of Python
I still regret...
Who we are? 2
Reverse proxy 3 Eyeball Reverse proxy Origin server • Optimizations
• Caching • DDoS protection • Security
Attacked 4
Signal 5 pretty analytics signal Operator switch switch switch
6 Signal
7
8 iptables rules mitigation servers command line Operator Mitigation
9 Mitigation
10 pretty analytics command line iptables mitigation signal Operator servers
switch switch switch
Copy-pasta 11
"Business logic" 12 iptables rules mitigation signal servers switch switch
switch ?
12 months later... 13
14 --ip=1.2.3.4 example.com --ip=1.2.3.4 example.com --qps=100 Business logic
15 --ip=1.2.3.4 example.com --qps=500 example.com = FREE | PAID Business
logic --ip=1.2.3.4 example.com
16 --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500 Business logic example.com subdomains:
(www, ns1, ns2) --ip=1.2.3.4 example.com example.com = FREE | PAID
17 Input Steam extra stream extra stream Output Stream Reactive
Rule
18 Reactive rule def dns_mitigation(attack, plan, subdomains, toggles): domain =
attack['domain'] if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation
Subscriptions 19 def dns_mitigation(attack, plan, subdomains, toggles): domain = attack['domain']
if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation
Business logic • Hard problem! • Multiple DB lookups •
Wait for operator confirmation • Critical path 20
Functional reactive programming 21
22
23 models - Excel
models - Materialized data 24 input output function 00:01h 23:59h
x
models - Signals 25
Pure FRP is useless • Weird language - (ELM anyone?)
• Fixed signal flow • Strictly no side-effects 26
Dirty FRP is awesome 27 • Weird language • Python
• Fixed signal flow • Attacks come and go, but patterns fixed • Strictly no side-effects • Dynamic "subscriptions", but idempotent
Prior art - Trellis 28
29
Gatelogic! 30 https://github.com/cloudflare/gatelogic
Gatelogic • Input - ReadableHub • update(full_data) • Processing -
ComputableHub • maintain(key, function) • unmaintain • Subscriptions - QueryHub • update(full_data) 31
32 { '00001': {ip:'1.2.3.4', port: 80, domain: 'bar.com', '00002': {ip:'1.2.3.5',
port: 80, domain: 'foo.com', ... } Input data - a dict
33 ReadableHub update { 'attack1': 'example.com', ... } update()
34 ComputableHub def on_hook(_, kind, k, row): if kind ==
'add': mitigations.maintain(k, action, row) if kind == 'delete': mitigations.unmaintain(k) subscribe(readable, on_hook) def action(row): return None 34 ReadableHub
35 OutputHub ComputableHub ?
36 OutputHub ComputableHub ReadableHub QueryHub ? database
37 OutputHub ComputableHub ReadableHub QueryHub X X X X X
X Materialized
38 def action(row, plan_hub, subdomain_hub, toggle_hub): domain = row.value if
not domain: return None if toggle_hub.get('all_mitigations_disabled').value != 'True': return None qps = 100 if plan_hub.get(domain).value in ('business', 'b'): qps = 500 sd = (subdomain_hub.get(domain).value or '').split(' ') mitigation = \ domain + \ ' --qps=%s ' % qps + \ ' '.join('--except=%s' % s for s in sd) return mitigation
It works! • Solid foundation! • Composable! • Scalable •
Maintainable • But: • no event loop • lacks higher-order abstractions 39
40
204 loc 41 marek@ubuntu-saucy:~/cloudflare/gatelogic/gatelogic$ cloc *py 3 text files. 3
unique files. 0 files ignored. http://cloc.sourceforge.net v 1.60 T=0.01 s (240.8 files/s, 23197.8 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 3 61 24 204 ------------------------------------------------------------------------------- SUM: 3 61 24 204 -------------------------------------------------------------------------------
Thanks! • FRP is grea • http://www.flapjax-lang.org/ • https://www.youtube.com/watch?v=mEvo6TVAf64 •
https://www.youtube.com/watch?v=Agu6jipKfYw 42 https://github.com/cloudflare/gatelogic marek@cloudflare.com @majek04