Norikra defends web service

Transcript

1. Norikra defends web service Norikra meetup #2 2015.06.03 @fujiwara

2. Fujiwara Shunichiro (@fujiwara) KAYAC Inc.

3. Norikra in a web service A community web service. •

   Realtime log analysis • HTTP status code, response time • worker logs • Switched from fluent-plugin-(datacounter|numeric-monitor) • Peak 10,000~ msgs/sec

4. Norikra in a web service

5. Norikra in a web service EC2 : r3.xlarge (4core 30GB

   mem) norikra start \ -Xmx23012m \ --small \ --dump-stat-interval=300 \ --stats=${DATA_DIR}/norikra.stats.json \ --pidfile=${DATA_DIR}/norikra.pid \ --gc-log=${DATA_DIR}/gc.log \ -XX:+UseGCLogFileRotation \ -XX:NumberOfGCLogFiles=10 \ -XX:GCLogFileSize=1048576 \

6. Spammers has come Smartphone apps only at first. Web version

   (for browsers) released!! But, it gives many benefits for ATTACKERS... • Easy to analyze API protocols • Easy to create MANY accounts (email, twitter...) → Spam floods!!!

7. Norikra detects spam POSTs SELECT 'spam-detect' as tag, count(*) as

   calls, min(time) as time_min, max(time) as time_max, N as threshold, host FROM accesslog.win:time_batch(1 min) WHERE uri LIKE '/api/%' and method = 'POST' GROUP BY host HAVING count(*) >= N

8. Norikra detects spam account creation SELECT 'spam-detect' as tag, count(*)

   as calls, min(time) as time_min, max(time) as time_max, N as threshold, host FROM accesslog.win:time_batch(1 min) WHERE uri LIKE '/api/signup/twitter%' GROUP BY host HAVING count(*) >= N

9. fluent-plugin-spam-reactor In-house plugin <source> type norikra norikra log-analyzer.service.consul:26571 <fetch> method

   sweep tag field tag tag_prefix norikra interval 3s </fetch> </source> <match norikra.spam-detect.**> type spam_reactor whitelist ["127.0.0.1", "192.168.0.0/16"] </match>

10. fluent-plugin-spam-reactor • Norikra detects spammers IP address from access log

    • Fluentd fetches Norikra outputs as records

11. fluent-plugin-spam-reactor • Calc : count / (time_max - time_min) >=

    threshold • Set the IP address to memcached (exclude whitelist) memd.set("banned-host:#{host}", 1, expires, false) memd.set("banned-host-count:#{host}", count, 86400, false) • Emit a report message router.emit("spam-report", time, {"message" => message}) WebApp BLOCKS accesses from IP address in memcached!

12. Merits of using memcached 1. Flexible expires. • spam-reactor extends

    expires by exponential backoff. 2. Accessibility between middlewares. • WebApp (Perl, Go) • Fluentd (Ruby) • Nginx (module, lua)

13. Conclusion • Norikra can detect spam accesses. • Easy to

    customize logic by queries • Fluentd custom plugins fit to your applications.