Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Norikra defends web service

Norikra defends web service

Norikra meetup #2

Ca6281fff64797dc419b78f51f25c0a5?s=128

FUJIWARA Shunichiro
PRO

June 03, 2015
Tweet

Transcript

  1. Norikra defends web service Norikra meetup #2 2015.06.03 @fujiwara

  2. Fujiwara Shunichiro (@fujiwara) KAYAC Inc.

  3. Norikra in a web service A community web service. •

    Realtime log analysis • HTTP status code, response time • worker logs • Switched from fluent-plugin-(datacounter|numeric-monitor) • Peak 10,000~ msgs/sec
  4. Norikra in a web service

  5. Norikra in a web service EC2 : r3.xlarge (4core 30GB

    mem) norikra start \ -Xmx23012m \ --small \ --dump-stat-interval=300 \ --stats=${DATA_DIR}/norikra.stats.json \ --pidfile=${DATA_DIR}/norikra.pid \ --gc-log=${DATA_DIR}/gc.log \ -XX:+UseGCLogFileRotation \ -XX:NumberOfGCLogFiles=10 \ -XX:GCLogFileSize=1048576 \
  6. Spammers has come Smartphone apps only at first. Web version

    (for browsers) released!! But, it gives many benefits for ATTACKERS... • Easy to analyze API protocols • Easy to create MANY accounts (email, twitter...) → Spam floods!!!
  7. Norikra detects spam POSTs SELECT 'spam-detect' as tag, count(*) as

    calls, min(time) as time_min, max(time) as time_max, N as threshold, host FROM accesslog.win:time_batch(1 min) WHERE uri LIKE '/api/%' and method = 'POST' GROUP BY host HAVING count(*) >= N
  8. Norikra detects spam account creation SELECT 'spam-detect' as tag, count(*)

    as calls, min(time) as time_min, max(time) as time_max, N as threshold, host FROM accesslog.win:time_batch(1 min) WHERE uri LIKE '/api/signup/twitter%' GROUP BY host HAVING count(*) >= N
  9. fluent-plugin-spam-reactor In-house plugin <source> type norikra norikra log-analyzer.service.consul:26571 <fetch> method

    sweep tag field tag tag_prefix norikra interval 3s </fetch> </source> <match norikra.spam-detect.**> type spam_reactor whitelist ["127.0.0.1", "192.168.0.0/16"] </match>
  10. fluent-plugin-spam-reactor • Norikra detects spammers IP address from access log

    • Fluentd fetches Norikra outputs as records
  11. fluent-plugin-spam-reactor • Calc : count / (time_max - time_min) >=

    threshold • Set the IP address to memcached (exclude whitelist) memd.set("banned-host:#{host}", 1, expires, false) memd.set("banned-host-count:#{host}", count, 86400, false) • Emit a report message router.emit("spam-report", time, {"message" => message}) WebApp BLOCKS accesses from IP address in memcached!
  12. Merits of using memcached 1. Flexible expires. • spam-reactor extends

    expires by exponential backoff. 2. Accessibility between middlewares. • WebApp (Perl, Go) • Fluentd (Ruby) • Nginx (module, lua)
  13. Conclusion • Norikra can detect spam accesses. • Easy to

    customize logic by queries • Fluentd custom plugins fit to your applications.