Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Norikra defends web service

Norikra defends web service

Norikra meetup #2

FUJIWARA Shunichiro

June 03, 2015
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. Norikra defends web
    service
    Norikra meetup #2
    2015.06.03 @fujiwara

    View full-size slide

  2. Fujiwara Shunichiro (@fujiwara)
    KAYAC Inc.

    View full-size slide

  3. Norikra in a web service
    A community web service.
    • Realtime log analysis
    • HTTP status code, response time
    • worker logs
    • Switched from fluent-plugin-(datacounter|numeric-monitor)
    • Peak 10,000~ msgs/sec

    View full-size slide

  4. Norikra in a web service

    View full-size slide

  5. Norikra in a web service
    EC2 : r3.xlarge (4core 30GB mem)
    norikra start \
    -Xmx23012m \
    --small \
    --dump-stat-interval=300 \
    --stats=${DATA_DIR}/norikra.stats.json \
    --pidfile=${DATA_DIR}/norikra.pid \
    --gc-log=${DATA_DIR}/gc.log \
    -XX:+UseGCLogFileRotation \
    -XX:NumberOfGCLogFiles=10 \
    -XX:GCLogFileSize=1048576 \

    View full-size slide

  6. Spammers has come
    Smartphone apps only at first.
    Web version (for browsers) released!!
    But, it gives many benefits for ATTACKERS...
    • Easy to analyze API protocols
    • Easy to create MANY accounts (email, twitter...)
    → Spam floods!!!

    View full-size slide

  7. Norikra detects spam POSTs
    SELECT 'spam-detect' as tag,
    count(*) as calls,
    min(time) as time_min,
    max(time) as time_max,
    N as threshold,
    host
    FROM accesslog.win:time_batch(1 min)
    WHERE uri LIKE '/api/%' and method = 'POST'
    GROUP BY host
    HAVING count(*) >= N

    View full-size slide

  8. Norikra detects spam account creation
    SELECT 'spam-detect' as tag,
    count(*) as calls,
    min(time) as time_min,
    max(time) as time_max,
    N as threshold,
    host
    FROM accesslog.win:time_batch(1 min)
    WHERE uri LIKE '/api/signup/twitter%'
    GROUP BY host
    HAVING count(*) >= N

    View full-size slide

  9. fluent-plugin-spam-reactor
    In-house plugin

    type norikra
    norikra log-analyzer.service.consul:26571

    method sweep
    tag field tag
    tag_prefix norikra
    interval 3s



    type spam_reactor
    whitelist ["127.0.0.1", "192.168.0.0/16"]

    View full-size slide

  10. fluent-plugin-spam-reactor
    • Norikra detects spammers IP address from access log
    • Fluentd fetches Norikra outputs as records

    View full-size slide

  11. fluent-plugin-spam-reactor
    • Calc : count / (time_max - time_min) >= threshold
    • Set the IP address to memcached (exclude whitelist)
    memd.set("banned-host:#{host}", 1, expires, false)
    memd.set("banned-host-count:#{host}", count, 86400, false)
    • Emit a report message
    router.emit("spam-report", time, {"message" => message})
    WebApp BLOCKS accesses from IP address in memcached!

    View full-size slide

  12. Merits of using memcached
    1. Flexible expires.
    • spam-reactor extends expires by exponential backoff.
    2. Accessibility between middlewares.
    • WebApp (Perl, Go)
    • Fluentd (Ruby)
    • Nginx (module, lua)

    View full-size slide

  13. Conclusion
    • Norikra can detect spam accesses.
    • Easy to customize logic by queries
    • Fluentd custom plugins fit to your applications.

    View full-size slide