Slide 1
Slide 1 text
Norikra defends web
service
Norikra meetup #2
2015.06.03 @fujiwara
Slide 2
Slide 2 text
Fujiwara Shunichiro (@fujiwara)
KAYAC Inc.
Slide 3
Slide 3 text
Norikra in a web service
A community web service.
• Realtime log analysis
• HTTP status code, response time
• worker logs
• Switched from fluent-plugin-(datacounter|numeric-monitor)
• Peak 10,000~ msgs/sec
Slide 4
Slide 4 text
Norikra in a web service
Slide 5
Slide 5 text
Norikra in a web service
EC2 : r3.xlarge (4core 30GB mem)
norikra start \
-Xmx23012m \
--small \
--dump-stat-interval=300 \
--stats=${DATA_DIR}/norikra.stats.json \
--pidfile=${DATA_DIR}/norikra.pid \
--gc-log=${DATA_DIR}/gc.log \
-XX:+UseGCLogFileRotation \
-XX:NumberOfGCLogFiles=10 \
-XX:GCLogFileSize=1048576 \
Slide 6
Slide 6 text
Spammers has come
Smartphone apps only at first.
Web version (for browsers) released!!
But, it gives many benefits for ATTACKERS...
• Easy to analyze API protocols
• Easy to create MANY accounts (email, twitter...)
→ Spam floods!!!
Slide 7
Slide 7 text
Norikra detects spam POSTs
SELECT 'spam-detect' as tag,
count(*) as calls,
min(time) as time_min,
max(time) as time_max,
N as threshold,
host
FROM accesslog.win:time_batch(1 min)
WHERE uri LIKE '/api/%' and method = 'POST'
GROUP BY host
HAVING count(*) >= N
Slide 8
Slide 8 text
Norikra detects spam account creation
SELECT 'spam-detect' as tag,
count(*) as calls,
min(time) as time_min,
max(time) as time_max,
N as threshold,
host
FROM accesslog.win:time_batch(1 min)
WHERE uri LIKE '/api/signup/twitter%'
GROUP BY host
HAVING count(*) >= N
Slide 9
Slide 9 text
fluent-plugin-spam-reactor
In-house plugin
type norikra
norikra log-analyzer.service.consul:26571
method sweep
tag field tag
tag_prefix norikra
interval 3s
type spam_reactor
whitelist ["127.0.0.1", "192.168.0.0/16"]
Slide 10
Slide 10 text
fluent-plugin-spam-reactor
• Norikra detects spammers IP address from access log
• Fluentd fetches Norikra outputs as records
Slide 11
Slide 11 text
fluent-plugin-spam-reactor
• Calc : count / (time_max - time_min) >= threshold
• Set the IP address to memcached (exclude whitelist)
memd.set("banned-host:#{host}", 1, expires, false)
memd.set("banned-host-count:#{host}", count, 86400, false)
• Emit a report message
router.emit("spam-report", time, {"message" => message})
WebApp BLOCKS accesses from IP address in memcached!
Slide 12
Slide 12 text
Merits of using memcached
1. Flexible expires.
• spam-reactor extends expires by exponential backoff.
2. Accessibility between middlewares.
• WebApp (Perl, Go)
• Fluentd (Ruby)
• Nginx (module, lua)
Slide 13
Slide 13 text
Conclusion
• Norikra can detect spam accesses.
• Easy to customize logic by queries
• Fluentd custom plugins fit to your applications.