Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Streaming Algorithms

majek04
March 15, 2016

Streaming Algorithms

majek04

March 15, 2016
Tweet

More Decks by majek04

Other Decks in Technology

Transcript

  1. Automatic DoS mitigation
    (streaming algorithms)
    Marek Majkowski
    [email protected]flare.com @majek04

    View Slide

  2. 2
    Who we are

    View Slide

  3. 3
    Large network

    View Slide

  4. 4
    Content neutral

    View Slide

  5. 5
    DoS is a problem
    DoS events per day

    View Slide

  6. 6
    X
    example.com
    Defending from DoS is hard

    View Slide

  7. • L3 - spoofed IP packets
    • source IP addresses are fake
    • very large
    • this is what you hear in news
    • L7 - fully established TCP connections
    • IP reputation is effective
    7
    Two DoS types

    View Slide

  8. 8
    L3 Volume per server
    Packets per second

    View Slide

  9. 9
    Automatic attack handling
    Attack
    Detection
    Mitigation
    database
    Reactive
    Automation
    sflow
    iptables

    View Slide

  10. 10
    Automatic attack detection
    Attack
    Detection
    sflow

    View Slide

  11. • infinite data stream on input
    • approximate
    11
    Streaming algorithms
    Streaming
    algorithm
    Data stream Results

    View Slide

  12. • sflow packets samples as input
    • detected attacks on output
    12
    Attack detection is streaming!
    Streaming
    algorithms
    Packet samples Attacks

    View Slide

  13. • EWMA - Exponentially weighted moving average
    • Counting rates of packets
    • Space saving
    • Known as Top-N or Heavy Hitters
    • Simplified hierarchical heavy hitters
    • Hyper log log
    • Cardinality estimation - Counting unique things
    13
    Streaming algorithms

    View Slide

  14. 14
    The problem: PPS
    !
    Mpps Descr!
    3.878 --ip=141.245.59.191/32!
    2.878 --ip=141.245.59.192/32!
    1.878 --ip=141.245.59.193/32!
    1.878 --ip=141.245.59.194/32!
    1.878 --ip=141.245.59.195/32!
    1.878 --ip=141.245.59.196/32!
    1.878 --ip=141.245.59.197/32!
    1.878 --ip=141.245.59.198/32!
    1.878 --ip=141.245.59.199/32!
    ...!

    View Slide

  15. 15
    Naive approach
    pps IP
    12.2M 1.2.3.4
    2.4M 42.1.2.4
    0.01M 2.4.3.1
    0.01M 192.168.1.1

    View Slide

  16. 16
    There is no such thing as pps

    View Slide

  17. 17
    Naive: Moving average
    1.0s 1.1s 1.3s 1.8s 1.99s 2.1s 2.4s 2.41s
    t=2.50s
    Precisely 5 samples

    View Slide

  18. 18
    Not-smoothed values
    1.0s 1.1s 1.3s 1.8s 1.99s 2.1s 2.4s 2.41s
    100
    3
    50
    5
    2
    5
    10
    raw pps=

    View Slide

  19. 19
    Not-smoothed values

    View Slide

  20. 20
    Linux load average - charge

    View Slide

  21. 21
    Linux load average - discharge

    View Slide

  22. 22
    Better: EWMA
    old load difference
    dampening
    factor
    measurement
    frequency
    half-life time

    View Slide

  23. 23

    View Slide

  24. 24

    View Slide

  25. • Smoothed average
    • The same maths as Linux "load average"
    • Charges slow (half-life)
    • Discharges quickly
    • Can be also used to count rates of packets
    25
    EWMA - summary

    View Slide

  26. 26
    The problem: PPS
    !
    Mpps Descr!
    3.878 --ip=141.245.59.191/32!
    2.878 --ip=141.245.59.192/32!
    1.878 --ip=141.245.59.193/32!
    1.878 --ip=141.245.59.194/32!
    1.878 --ip=141.245.59.195/32!
    1.878 --ip=141.245.59.196/32!
    1.878 --ip=141.245.59.197/32!
    1.878 --ip=141.245.59.198/32!
    1.878 --ip=141.245.59.199/32!
    ...!

    View Slide

  27. 27
    The problem: Memory
    pps IP
    12.2M 1.2.3.4
    2.4M 42.1.2.4
    0.01M 2.4.3.1
    0.01M 192.168.1.1
    ...

    View Slide

  28. • aka: heavy hitters
    • A fixed-memory data structure
    • That can "count" top-N items
    • think: top url's, top customer IP's, etc
    • Count-Min sketch, Space Saving
    28
    Top-N problem

    View Slide

  29. 29
    Space saving
    error count key

    View Slide

  30. 30
    Space saving
    error count key
    0 1 Alice
    Alice

    View Slide

  31. 31
    Space saving
    error count key
    0 2 Alice
    Alice

    View Slide

  32. 32
    Space saving
    error count key
    0 2 Alice
    0 1 Ben
    Ben

    View Slide

  33. 33
    Space saving
    error count key
    0 2 Alice
    0 1 Ben
    0 1 Charlie
    Charlie

    View Slide

  34. 34
    Space saving
    error count key
    0 2 Alice
    0 1 Ben
    0 1 Charlie
    Eric?

    View Slide

  35. 35
    Space saving
    error count key
    0 2 Alice
    0 1 Ben
    0 1 Charlie
    Eric?

    View Slide

  36. 36
    Space saving
    error count key
    0 2 Alice
    1 0 Eric
    0 1 Charlie
    +
    Eric

    View Slide

  37. 37
    Space saving
    error count key
    0 2 Alice
    1 1 Eric
    0 1 Charlie
    Eric

    View Slide

  38. 38
    Space saving
    error count key
    0 2 Alice
    1 1 Eric
    0 1 Charlie
    2
    Counter?
    1 .. 2
    1

    View Slide

  39. 39

    View Slide

  40. What about rates?
    40
    • It's hard
    • was: GetAll()
    • now: GetAll(time.Time)
    • No longer O(1)
    • Instead O(log n)

    View Slide

  41. 41

    View Slide

  42. • Top-N / Heavy-hitter algorithm
    • Fixed memory size
    • Strong error guarantees
    42
    Summary - Space saving

    View Slide

  43. 43
    Aggregating attacks
    !
    Mpps Descr!
    3.878 --ip=141.245.59.191/32!
    2.878 --ip=141.245.59.192/32!
    1.878 --ip=141.245.59.193/32!
    1.878 --ip=141.245.59.194/32!
    1.878 --ip=141.245.59.195/32!
    1.878 --ip=141.245.59.196/32!
    1.878 --ip=141.245.59.197/32!
    1.878 --ip=141.245.59.198/32!
    1.878 --ip=141.245.59.199/32!
    ...!
    !
    Mpps Descr!
    35.878 --ip=141.245.59.0/24!
    vs

    View Slide

  44. 44
    Hierarchical Heavy Hitters

    View Slide

  45. 45
    Simplified HHH

    View Slide

  46. 46
    Multiple dimensions
    pps IP:port
    12.2M 1.2.3.4:53
    2.4M 42.1.2.4:80
    0.01M 2.4.3.1:80
    0.01M 192.168.1.1:443
    pps IP
    12.2M 1.2.3.4
    2.4M 42.1.2.4
    0.01M 2.4.3.1
    0.01M 192.168.1.1
    pps subnet
    12.2M 1.2.3.0/24
    2.4M 42.1.2.0/24
    0.01M 2.4.3.0/24
    0.01M 192.168.1.0/24

    View Slide

  47. 47
    Multiple dimensions
    pps IP:port
    12.2M 1.2.3.4:53
    2.4M 42.1.2.4:80
    0.01M 2.4.3.1:80
    0.01M 192.168.1.1:443
    pps IP
    12.2M 1.2.3.4
    2.4M 42.1.2.4
    0.01M 2.4.3.1
    0.01M 192.168.1.1
    pps subnet
    12.2M 1.2.3.0/24
    2.4M 42.1.2.0/24
    0.01M 2.4.3.0/24
    0.01M 192.168.1.0/24
    incoming sample: 42.1.2.4:80

    View Slide

  48. 48
    Multiple dimensions
    pps IP:port
    12.2M 1.2.3.4:53
    2.4M 42.1.2.4:80
    0.01M 2.4.3.1:80
    0.01M 192.168.1.1:443
    pps IP
    12.2M 1.2.3.4
    2.4M 42.1.2.4
    0.01M 2.4.3.1
    0.01M 192.168.1.1
    pps subnet
    12.2M 1.2.3.0/24
    2.4M 42.1.2.0/24
    0.01M 2.4.3.0/24
    0.01M 192.168.1.0/24
    reporting threshold: 1M

    View Slide

  49. 49
    Attack report
    !
    Mpps Descr!
    12.2 --ip=1.2.3.4 --port=53!
    2.4 --ip=42.1.2.4 --port=80!
    12.2 --ip=1.2.3.4!
    2.4 --ip=42.1.2.4!
    12.2 --ip=1.2.3.0/24!
    2.4 --ip=42.1.2.0/24!

    View Slide

  50. 50
    Multiple dimensions
    pps IP:port
    12.2M 1.2.3.4:53
    2.4M 42.1.2.4:80
    0.01M 2.4.3.1:80
    0.01M 192.168.1.1:443
    pps IP
    0.1M 1.2.3.4
    0M 42.1.2.4
    0.01M 2.4.3.1
    0.01M 192.168.1.1
    pps subnet
    0.1M 1.2.3.0/24
    0M 42.1.2.0/24
    0.01M 2.4.3.0/24
    0.01M 192.168.1.0/24
    incoming sample: 42.1.2.4:80

    View Slide

  51. 51
    Attack report
    !
    Mpps Descr!
    12.2 --ip=1.2.3.4 --port=53!
    2.4 --ip=42.1.2.4 --port=80!

    View Slide

  52. 52
    Scales well

    View Slide

  53. • Approximate
    • High error in pps
    • Works well in practice
    • Scales well
    • Fast and simple to implement
    53
    Summary - SHHH

    View Slide

  54. 54
    Spoofed Source IP?
    !
    Mpps Description!
    23.833 --target=173.245.59.2 --agent=WAW --iface=659 Est= 57364!
    23.067 --target=173.245.58.1 --agent=WAW --iface=659 Est= 56995!
    7.139 --target=173.245.58.1 --agent=DUS --iface=893 Est= 11493!
    6.366 --target=173.245.59.2 --agent=DUS --iface=893 Est= 11240!
    2.590 --target=173.245.58.1 --agent=SIN --iface=657 Est=219987!
    2.557 --target=173.245.59.2 --agent=SIN --iface=657 Est=220380!
    1.045 --target=173.245.58.1 --agent=MAN --iface=756 Est= 207!
    1.039 --target=173.245.59.2 --agent=MAN --iface=756 Est= 200!

    View Slide

  55. 55
    Hyper log log
    "Alice"
    22 unique items!
    HLL

    View Slide

  56. 56
    Hyper log log
    OR 44 unique items
    ( ) =
    HLL#1 HLL#2

    View Slide

  57. 57

    View Slide

  58. 58
    What about rates?
    HLL #1 HLL #2 HLL #3 HLL #4

    View Slide

  59. 59
    Hard drives

    View Slide

  60. • Attack detection is a streaming problem
    • Streaming algorithms are awesome
    • Applicable to many more problems
    60
    Summary
    Thanks!
    [email protected]flare.com

    View Slide