Norikra defends web service

Norikra meetup #2

Transcript

1. Norikra defends web service Norikra meetup #2 2015.06.03 @fujiwara
2. Fujiwara Shunichiro (@fujiwara) KAYAC Inc.
3. Norikra in a web service A community web service. • Realtime log analysis • HTTP status code, response time • worker logs • Switched from fluent-plugin-(datacounter|numeric-monitor) • Peak 10,000~ msgs/sec
4. Norikra in a web service
5. Norikra in a web service EC2 : r3.xlarge (4core 30GB mem) norikra start \ -Xmx23012m \ --small \ --dump-stat-interval=300 \ --stats=${DATA_DIR}/norikra.stats.json \ --pidfile=${DATA_DIR}/norikra.pid \ --gc-log=${DATA_DIR}/gc.log \ -XX:+UseGCLogFileRotation \ -XX:NumberOfGCLogFiles=10 \ -XX:GCLogFileSize=1048576 \
6. Spammers has come Smartphone apps only at first. Web version (for browsers) released!! But, it gives many benefits for ATTACKERS... • Easy to analyze API protocols • Easy to create MANY accounts (email, twitter...) → Spam floods!!!
7. Norikra detects spam POSTs SELECT 'spam-detect' as tag, count(*) as calls, min(time) as time_min, max(time) as time_max, N as threshold, host FROM accesslog.win:time_batch(1 min) WHERE uri LIKE '/api/%' and method = 'POST' GROUP BY host HAVING count(*) >= N
8. Norikra detects spam account creation SELECT 'spam-detect' as tag, count(*) as calls, min(time) as time_min, max(time) as time_max, N as threshold, host FROM accesslog.win:time_batch(1 min) WHERE uri LIKE '/api/signup/twitter%' GROUP BY host HAVING count(*) >= N
9. fluent-plugin-spam-reactor In-house plugin <source> type norikra norikra log-analyzer.service.consul:26571 <fetch> method sweep tag field tag tag_prefix norikra interval 3s </fetch> </source> <match norikra.spam-detect.**> type spam_reactor whitelist ["127.0.0.1", "192.168.0.0/16"] </match>
10. fluent-plugin-spam-reactor • Norikra detects spammers IP address from access log • Fluentd fetches Norikra outputs as records
11. fluent-plugin-spam-reactor • Calc : count / (time_max - time_min) >= threshold • Set the IP address to memcached (exclude whitelist) memd.set("banned-host:#{host}", 1, expires, false) memd.set("banned-host-count:#{host}", count, 86400, false) • Emit a report message router.emit("spam-report", time, {"message" => message}) WebApp BLOCKS accesses from IP address in memcached!
12. Merits of using memcached 1. Flexible expires. • spam-reactor extends expires by exponential backoff. 2. Accessibility between middlewares. • WebApp (Perl, Go) • Fluentd (Ruby) • Nginx (module, lua)
13. Conclusion • Norikra can detect spam accesses. • Easy to customize logic by queries • Fluentd custom plugins fit to your applications.