Tweet
Tweet

Slide 1

Slide 1 text

Gatelogic FRP framework Marek Majkowski @majek04 ...200 lines of Python I still regret...

Slide 2

Slide 2 text

Who we are? 2

Slide 3

Slide 3 text

Reverse proxy 3 Eyeball Reverse proxy Origin server • Optimizations • Caching • DDoS protection • Security

Slide 4

Slide 4 text

Attacked 4

Slide 5

Slide 5 text

Signal 5 pretty analytics signal Operator switch switch switch

Slide 6

Slide 6 text

6 Signal

Slide 7

Slide 7 text

7

Slide 8

Slide 8 text

8 iptables rules mitigation servers command line Operator Mitigation

Slide 9

Slide 9 text

9 Mitigation

Slide 10

Slide 10 text

10 pretty analytics command line iptables mitigation signal Operator servers switch switch switch

Slide 11

Slide 11 text

Copy-pasta 11

Slide 12

Slide 12 text

"Business logic" 12 iptables rules mitigation signal servers switch switch switch ?

Slide 13

Slide 13 text

12 months later... 13

Slide 14

Slide 14 text

14 --ip=1.2.3.4 example.com --ip=1.2.3.4 example.com --qps=100 Business logic

Slide 15

Slide 15 text

15 --ip=1.2.3.4 example.com --qps=500 example.com = FREE | PAID Business logic --ip=1.2.3.4 example.com

Slide 16

Slide 16 text

16 --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500 Business logic example.com subdomains: (www, ns1, ns2) --ip=1.2.3.4 example.com example.com = FREE | PAID

Slide 17

Slide 17 text

17 Input Steam extra stream extra stream Output Stream Reactive Rule

Slide 18

Slide 18 text

18 Reactive rule def dns_mitigation(attack, plan, subdomains, toggles): domain = attack['domain'] if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation

Slide 19

Slide 19 text

Subscriptions 19 def dns_mitigation(attack, plan, subdomains, toggles): domain = attack['domain'] if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation

Slide 20

Slide 20 text

Business logic • Hard problem! • Multiple DB lookups • Wait for operator confirmation • Critical path 20

Slide 21

Slide 21 text

Functional reactive programming 21

Slide 22

Slide 22 text

22

Slide 23

Slide 23 text

23 models - Excel

Slide 24

Slide 24 text

models - Materialized data 24 input output function 00:01h 23:59h x

Slide 25

Slide 25 text

models - Signals 25

Slide 26

Slide 26 text

Pure FRP is useless • Weird language - (ELM anyone?) • Fixed signal flow • Strictly no side-effects 26

Slide 27

Slide 27 text

Dirty FRP is awesome 27 • Weird language • Python • Fixed signal flow • Attacks come and go, but patterns fixed • Strictly no side-effects • Dynamic "subscriptions", but idempotent

Slide 28

Slide 28 text

Prior art - Trellis 28

Slide 29

Slide 29 text

29

Slide 30

Slide 30 text

Gatelogic! 30 https://github.com/cloudflare/gatelogic

Slide 31

Slide 31 text

Gatelogic • Input - ReadableHub • update(full_data) • Processing - ComputableHub • maintain(key, function) • unmaintain • Subscriptions - QueryHub • update(full_data) 31

Slide 32

Slide 32 text

32 { '00001': {ip:'1.2.3.4', port: 80, domain: 'bar.com', '00002': {ip:'1.2.3.5', port: 80, domain: 'foo.com', ... } Input data - a dict

Slide 33

Slide 33 text

33 ReadableHub update { 'attack1': 'example.com', ... } update()

Slide 34

Slide 34 text

34 ComputableHub def on_hook(_, kind, k, row): if kind == 'add': mitigations.maintain(k, action, row) if kind == 'delete': mitigations.unmaintain(k) subscribe(readable, on_hook) def action(row): return None 34 ReadableHub

Slide 35

Slide 35 text

35 OutputHub ComputableHub ?

Slide 36

Slide 36 text

36 OutputHub ComputableHub ReadableHub QueryHub ? database

Slide 37

Slide 37 text

37 OutputHub ComputableHub ReadableHub QueryHub X X X X X X Materialized

Slide 38

Slide 38 text

38 def action(row, plan_hub, subdomain_hub, toggle_hub): domain = row.value if not domain: return None if toggle_hub.get('all_mitigations_disabled').value != 'True': return None qps = 100 if plan_hub.get(domain).value in ('business', 'b'): qps = 500 sd = (subdomain_hub.get(domain).value or '').split(' ') mitigation = \ domain + \ ' --qps=%s ' % qps + \ ' '.join('--except=%s' % s for s in sd) return mitigation

Slide 39

Slide 39 text

It works! • Solid foundation! • Composable! • Scalable • Maintainable • But: • no event loop • lacks higher-order abstractions 39

Slide 40

Slide 40 text

40

Slide 41

Slide 41 text

204 loc 41 marek@ubuntu-saucy:~/cloudflare/gatelogic/gatelogic$ cloc *py 3 text files. 3 unique files. 0 files ignored. http://cloc.sourceforge.net v 1.60 T=0.01 s (240.8 files/s, 23197.8 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 3 61 24 204 ------------------------------------------------------------------------------- SUM: 3 61 24 204 -------------------------------------------------------------------------------

Slide 42

Slide 42 text

Thanks! • FRP is grea • http://www.flapjax-lang.org/ • https://www.youtube.com/watch?v=mEvo6TVAf64 • https://www.youtube.com/watch?v=Agu6jipKfYw 42 https://github.com/cloudflare/gatelogic marek@cloudflare.com @majek04