Gatelogic - Somewhat functional reactive framework in Python

D4e1d473a995ef37b3e03e9e6006c3e3?s=47 majek04
November 27, 2017

Gatelogic - Somewhat functional reactive framework in Python

D4e1d473a995ef37b3e03e9e6006c3e3?s=128

majek04

November 27, 2017
Tweet

Transcript

  1. Gatelogic FRP framework Marek Majkowski @majek04 ...200 lines of Python

    I still regret...
  2. Who we are? 2

  3. Reverse proxy 3 Eyeball Reverse proxy Origin server • Optimizations

    • Caching • DDoS protection • Security
  4. Attacked 4

  5. Signal 5 pretty analytics signal Operator switch switch switch

  6. 6 Signal

  7. 7

  8. 8 iptables rules mitigation servers command line Operator Mitigation

  9. 9 Mitigation

  10. 10 pretty analytics command line iptables mitigation signal Operator servers

    switch switch switch
  11. Copy-pasta 11

  12. "Business logic" 12 iptables rules mitigation signal servers switch switch

    switch ?
  13. 12 months later... 13

  14. 14 --ip=1.2.3.4 example.com --ip=1.2.3.4 example.com --qps=100 Business logic

  15. 15 --ip=1.2.3.4 example.com --qps=500 example.com = FREE | PAID Business

    logic --ip=1.2.3.4 example.com
  16. 16 --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500 Business logic example.com subdomains:

    (www, ns1, ns2) --ip=1.2.3.4 example.com example.com = FREE | PAID
  17. 17 Input Steam extra stream extra stream Output Stream Reactive

    Rule
  18. 18 Reactive rule def dns_mitigation(attack, plan, subdomains, toggles): domain =

    attack['domain'] if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation
  19. Subscriptions 19 def dns_mitigation(attack, plan, subdomains, toggles): domain = attack['domain']

    if toggles['all_mitigations_disabled']: return qps = 100 if plan[domain] == 'business': qps = 500 mitigation = attack['description'] + \ ' --qps=%s' % qps + \ ' --except=%s'.join(subdomains[domain]) return mitigation
  20. Business logic • Hard problem! • Multiple DB lookups •

    Wait for operator confirmation • Critical path 20
  21. Functional reactive programming 21

  22. 22

  23. 23 models - Excel

  24. models - Materialized data 24 input output function 00:01h 23:59h

    x
  25. models - Signals 25

  26. Pure FRP is useless • Weird language - (ELM anyone?)

    • Fixed signal flow • Strictly no side-effects 26
  27. Dirty FRP is awesome 27 • Weird language • Python

    • Fixed signal flow • Attacks come and go, but patterns fixed • Strictly no side-effects • Dynamic "subscriptions", but idempotent
  28. Prior art - Trellis 28

  29. 29

  30. Gatelogic! 30 https://github.com/cloudflare/gatelogic

  31. Gatelogic • Input - ReadableHub • update(full_data) • Processing -

    ComputableHub • maintain(key, function) • unmaintain • Subscriptions - QueryHub • update(full_data) 31
  32. 32 { '00001': {ip:'1.2.3.4', port: 80, domain: 'bar.com', '00002': {ip:'1.2.3.5',

    port: 80, domain: 'foo.com', ... } Input data - a dict
  33. 33 ReadableHub update { 'attack1': 'example.com', ... } update()

  34. 34 ComputableHub def on_hook(_, kind, k, row): if kind ==

    'add': mitigations.maintain(k, action, row) if kind == 'delete': mitigations.unmaintain(k) subscribe(readable, on_hook) def action(row): return None 34 ReadableHub
  35. 35 OutputHub ComputableHub ?

  36. 36 OutputHub ComputableHub ReadableHub QueryHub ? database

  37. 37 OutputHub ComputableHub ReadableHub QueryHub X X X X X

    X Materialized
  38. 38 def action(row, plan_hub, subdomain_hub, toggle_hub): domain = row.value if

    not domain: return None if toggle_hub.get('all_mitigations_disabled').value != 'True': return None qps = 100 if plan_hub.get(domain).value in ('business', 'b'): qps = 500 sd = (subdomain_hub.get(domain).value or '').split(' ') mitigation = \ domain + \ ' --qps=%s ' % qps + \ ' '.join('--except=%s' % s for s in sd) return mitigation
  39. It works! • Solid foundation! • Composable! • Scalable •

    Maintainable • But: • no event loop • lacks higher-order abstractions 39
  40. 40

  41. 204 loc 41 marek@ubuntu-saucy:~/cloudflare/gatelogic/gatelogic$ cloc *py 3 text files. 3

    unique files. 0 files ignored. http://cloc.sourceforge.net v 1.60 T=0.01 s (240.8 files/s, 23197.8 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Python 3 61 24 204 ------------------------------------------------------------------------------- SUM: 3 61 24 204 -------------------------------------------------------------------------------
  42. Thanks! • FRP is grea • http://www.flapjax-lang.org/ • https://www.youtube.com/watch?v=mEvo6TVAf64 •

    https://www.youtube.com/watch?v=Agu6jipKfYw 42 https://github.com/cloudflare/gatelogic marek@cloudflare.com @majek04