Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Gatelogic - Somewhat functional reactive framework in Python

majek04
November 27, 2017

Gatelogic - Somewhat functional reactive framework in Python

majek04

November 27, 2017
Tweet

More Decks by majek04

Other Decks in Programming

Transcript

  1. Gatelogic FRP framework
    Marek Majkowski @majek04
    ...200 lines of Python I still regret...

    View full-size slide

  2. Who we are?
    2

    View full-size slide

  3. Reverse proxy
    3
    Eyeball Reverse proxy Origin server
    • Optimizations
    • Caching
    • DDoS protection
    • Security

    View full-size slide

  4. Signal
    5
    pretty analytics
    signal
    Operator
    switch
    switch
    switch

    View full-size slide

  5. 8
    iptables rules
    mitigation
    servers
    command line
    Operator
    Mitigation

    View full-size slide

  6. 10
    pretty analytics
    command line
    iptables
    mitigation
    signal
    Operator
    servers
    switch
    switch
    switch

    View full-size slide

  7. Copy-pasta
    11

    View full-size slide

  8. "Business logic"
    12
    iptables rules
    mitigation
    signal
    servers
    switch
    switch
    switch
    ?

    View full-size slide

  9. 12 months later...
    13

    View full-size slide

  10. 14
    --ip=1.2.3.4 example.com
    --ip=1.2.3.4 example.com --qps=100
    Business logic

    View full-size slide

  11. 15
    --ip=1.2.3.4 example.com --qps=500
    example.com = FREE | PAID
    Business logic
    --ip=1.2.3.4 example.com

    View full-size slide

  12. 16
    --ip=1.2.3.4 example.com --except www,n1,ns2 --qps=500
    Business logic
    example.com subdomains:
    (www, ns1, ns2)
    --ip=1.2.3.4 example.com
    example.com = FREE | PAID

    View full-size slide

  13. 17
    Input Steam
    extra stream
    extra stream
    Output Stream
    Reactive
    Rule

    View full-size slide

  14. 18
    Reactive rule
    def dns_mitigation(attack, plan, subdomains, toggles):
    domain = attack['domain']
    if toggles['all_mitigations_disabled']: return
    qps = 100
    if plan[domain] == 'business':
    qps = 500
    mitigation =
    attack['description'] + \
    ' --qps=%s' % qps + \
    ' --except=%s'.join(subdomains[domain])
    return mitigation

    View full-size slide

  15. Subscriptions
    19
    def dns_mitigation(attack, plan, subdomains, toggles):
    domain = attack['domain']
    if toggles['all_mitigations_disabled']: return
    qps = 100
    if plan[domain] == 'business':
    qps = 500
    mitigation =
    attack['description'] + \
    ' --qps=%s' % qps + \
    ' --except=%s'.join(subdomains[domain])
    return mitigation

    View full-size slide

  16. Business logic
    • Hard problem!
    • Multiple DB lookups
    • Wait for operator confirmation
    • Critical path
    20

    View full-size slide

  17. Functional reactive programming
    21

    View full-size slide

  18. 23
    models - Excel

    View full-size slide

  19. models - Materialized data
    24
    input
    output
    function
    00:01h
    23:59h
    x

    View full-size slide

  20. models - Signals
    25

    View full-size slide

  21. Pure FRP is useless
    • Weird language - (ELM anyone?)
    • Fixed signal flow
    • Strictly no side-effects
    26

    View full-size slide

  22. Dirty FRP is awesome
    27
    • Weird language
    • Python
    • Fixed signal flow
    • Attacks come and go, but patterns fixed
    • Strictly no side-effects
    • Dynamic "subscriptions", but idempotent

    View full-size slide

  23. Prior art - Trellis
    28

    View full-size slide

  24. Gatelogic!
    30
    https://github.com/cloudflare/gatelogic

    View full-size slide

  25. Gatelogic
    • Input - ReadableHub
    • update(full_data)
    • Processing - ComputableHub
    • maintain(key, function)
    • unmaintain
    • Subscriptions - QueryHub
    • update(full_data)
    31

    View full-size slide

  26. 32
    {
    '00001': {ip:'1.2.3.4', port: 80, domain: 'bar.com',
    '00002': {ip:'1.2.3.5', port: 80, domain: 'foo.com',
    ...
    }
    Input data - a dict

    View full-size slide

  27. 33
    ReadableHub
    update
    {
    'attack1': 'example.com',
    ...
    }
    update()

    View full-size slide

  28. 34
    ComputableHub
    def on_hook(_, kind, k, row):
    if kind == 'add':
    mitigations.maintain(k, action, row)
    if kind == 'delete':
    mitigations.unmaintain(k)
    subscribe(readable, on_hook)
    def action(row):
    return None
    34
    ReadableHub

    View full-size slide

  29. 35
    OutputHub
    ComputableHub
    ?

    View full-size slide

  30. 36
    OutputHub
    ComputableHub
    ReadableHub
    QueryHub
    ?
    database

    View full-size slide

  31. 37
    OutputHub
    ComputableHub
    ReadableHub
    QueryHub
    X X X X
    X X
    Materialized

    View full-size slide

  32. 38
    def action(row, plan_hub, subdomain_hub, toggle_hub):
    domain = row.value
    if not domain:
    return None
    if toggle_hub.get('all_mitigations_disabled').value != 'True':
    return None
    qps = 100
    if plan_hub.get(domain).value in ('business', 'b'):
    qps = 500
    sd = (subdomain_hub.get(domain).value or '').split(' ')
    mitigation = \
    domain + \
    ' --qps=%s ' % qps + \
    ' '.join('--except=%s' % s for s in sd)
    return mitigation

    View full-size slide

  33. It works!
    • Solid foundation!
    • Composable!
    • Scalable
    • Maintainable
    • But:
    • no event loop
    • lacks higher-order abstractions
    39

    View full-size slide

  34. 204 loc
    41
    marek@ubuntu-saucy:~/cloudflare/gatelogic/gatelogic$ cloc *py
    3 text files.
    3 unique files.
    0 files ignored.
    http://cloc.sourceforge.net v 1.60 T=0.01 s (240.8 files/s, 23197.8 lines/s)
    -------------------------------------------------------------------------------
    Language files blank comment code
    -------------------------------------------------------------------------------
    Python 3 61 24 204
    -------------------------------------------------------------------------------
    SUM: 3 61 24 204
    -------------------------------------------------------------------------------

    View full-size slide

  35. Thanks!
    • FRP is grea
    • http://www.flapjax-lang.org/
    • https://www.youtube.com/watch?v=mEvo6TVAf64
    • https://www.youtube.com/watch?v=Agu6jipKfYw
    42
    https://github.com/cloudflare/gatelogic
    marek@cloudflare.com @majek04

    View full-size slide