Gatelogic - Somewhat functional reactive framework in Python

Transcript

1. Gatelogic FRP framework
   Marek Majkowski @majek04
   ...200 lines of Python I still regret...
   

   View Slide

2. Who we are?
   2
   

   View Slide

3. Reverse proxy
   3
   Eyeball Reverse proxy Origin server
   • Optimizations
   • Caching
   • DDoS protection
   • Security
   

   View Slide

4. Attacked
   4
   

   View Slide

5. Signal
   5
   pretty analytics
   signal
   Operator
   switch
   switch
   switch
   

   View Slide

6. 6
   Signal
   

   View Slide

7. View Slide

8. 8
   iptables rules
   mitigation
   servers
   command line
   Operator
   Mitigation
   

   View Slide

9. 9
   Mitigation
   

   View Slide

10. 10
    pretty analytics
    command line
    iptables
    mitigation
    signal
    Operator
    servers
    switch
    switch
    switch
    

    View Slide

11. Copy-pasta
    11
    

    View Slide

12. "Business logic"
    12
    iptables rules
    mitigation
    signal
    servers
    switch
    switch
    switch
    ?
    

    View Slide

13. 12 months later...
    13
    

    View Slide

14. 14
    --ip=1.2.3.4 example.com
    --ip=1.2.3.4 example.com --qps=100
    Business logic
    

    View Slide

15. 15
    --ip=1.2.3.4 example.com --qps=500
    example.com = FREE | PAID
    Business logic
    --ip=1.2.3.4 example.com
    

    View Slide

16. 16
    --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500
    Business logic
    example.com subdomains:
    (www, ns1, ns2)
    --ip=1.2.3.4 example.com
    example.com = FREE | PAID
    

    View Slide

17. 17
    Input Steam
    extra stream
    extra stream
    Output Stream
    Reactive
    Rule
    

    View Slide

18. 18
    Reactive rule
    def dns_mitigation(attack, plan, subdomains, toggles):
    domain = attack['domain']
    if toggles['all_mitigations_disabled']: return
    qps = 100
    if plan[domain] == 'business':
    qps = 500
    mitigation =
    attack['description'] + \
    ' --qps=%s' % qps + \
    ' --except=%s'.join(subdomains[domain])
    return mitigation
    

    View Slide

19. Subscriptions
    19
    def dns_mitigation(attack, plan, subdomains, toggles):
    domain = attack['domain']
    if toggles['all_mitigations_disabled']: return
    qps = 100
    if plan[domain] == 'business':
    qps = 500
    mitigation =
    attack['description'] + \
    ' --qps=%s' % qps + \
    ' --except=%s'.join(subdomains[domain])
    return mitigation
    

    View Slide

20. Business logic
    • Hard problem!
    • Multiple DB lookups
    • Wait for operator confirmation
    • Critical path
    20
    

    View Slide

21. Functional reactive programming
    21
    

    View Slide

22. View Slide

23. 23
    models - Excel
    

    View Slide

24. models - Materialized data
    24
    input
    output
    function
    00:01h
    23:59h
    x
    

    View Slide

25. models - Signals
    25
    

    View Slide

26. Pure FRP is useless
    • Weird language - (ELM anyone?)
    • Fixed signal flow
    • Strictly no side-effects
    26
    

    View Slide

27. Dirty FRP is awesome
    27
    • Weird language
    • Python
    • Fixed signal flow
    • Attacks come and go, but patterns fixed
    • Strictly no side-effects
    • Dynamic "subscriptions", but idempotent
    

    View Slide

28. Prior art - Trellis
    28
    

    View Slide

29. View Slide

30. Gatelogic!
    30
    https://github.com/cloudflare/gatelogic
    

    View Slide

31. Gatelogic
    • Input - ReadableHub
    • update(full_data)
    • Processing - ComputableHub
    • maintain(key, function)
    • unmaintain
    • Subscriptions - QueryHub
    • update(full_data)
    31
    

    View Slide

32. 32
    {
    '00001': {ip:'1.2.3.4', port: 80, domain: 'bar.com',
    '00002': {ip:'1.2.3.5', port: 80, domain: 'foo.com',
    ...
    }
    Input data - a dict
    

    View Slide

33. 33
    ReadableHub
    update
    {
    'attack1': 'example.com',
    ...
    }
    update()
    

    View Slide

34. 34
    ComputableHub
    def on_hook(_, kind, k, row):
    if kind == 'add':
    mitigations.maintain(k, action, row)
    if kind == 'delete':
    mitigations.unmaintain(k)
    subscribe(readable, on_hook)
    def action(row):
    return None
    34
    ReadableHub
    

    View Slide

35. 35
    OutputHub
    ComputableHub
    ?
    

    View Slide

36. 36
    OutputHub
    ComputableHub
    ReadableHub
    QueryHub
    ?
    database
    

    View Slide

37. 37
    OutputHub
    ComputableHub
    ReadableHub
    QueryHub
    X X X X
    X X
    Materialized
    

    View Slide

38. 38
    def action(row, plan_hub, subdomain_hub, toggle_hub):
    domain = row.value
    if not domain:
    return None
    if toggle_hub.get('all_mitigations_disabled').value != 'True':
    return None
    qps = 100
    if plan_hub.get(domain).value in ('business', 'b'):
    qps = 500
    sd = (subdomain_hub.get(domain).value or '').split(' ')
    mitigation = \
    domain + \
    ' --qps=%s ' % qps + \
    ' '.join('--except=%s' % s for s in sd)
    return mitigation
    

    View Slide

39. It works!
    • Solid foundation!
    • Composable!
    • Scalable
    • Maintainable
    • But:
    • no event loop
    • lacks higher-order abstractions
    39
    

    View Slide

40. View Slide

41. 204 loc
    41
    marek@ubuntu-saucy:~/cloudflare/gatelogic/gatelogic$ cloc *py
    3 text files.
    3 unique files.
    0 files ignored.
    http://cloc.sourceforge.net v 1.60 T=0.01 s (240.8 files/s, 23197.8 lines/s)
    -------------------------------------------------------------------------------
    Language files blank comment code
    -------------------------------------------------------------------------------
    Python 3 61 24 204
    -------------------------------------------------------------------------------
    SUM: 3 61 24 204
    -------------------------------------------------------------------------------
    

    View Slide

42. Thanks!
    • FRP is grea
    • http://www.flapjax-lang.org/
    • https://www.youtube.com/watch?v=mEvo6TVAf64
    • https://www.youtube.com/watch?v=Agu6jipKfYw
    42
    https://github.com/cloudflare/gatelogic
    marek@cloudflare.com @majek04
    

    View Slide